2023-10-17T17:54:02.906 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T17:54:02.907 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137 --name yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/orig.config.yaml 2023-10-17T17:54:02.943 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T17:54:03.015 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137 branch: wip-yuri-testing-2023-10-16-1247-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7430137' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-10-16-1247-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-10-16-1247-reef sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 993 sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-10-16-1247-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKx21743cUlmo6TKUbvR5PAR3YJyl6RDmMH8t+R6LplmFaEgnLfqs3jO/mdESrHA1/7zybwQDizFBDdcbLTm/18= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxePA3TlvYqb+4IRWm+eI1CkcB6eXCdh5zeTPXhaTJWuWW7972TipdWZLJP4OU5UiJN2/V8uTym1MqHpDcG0e4= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_21:55:28 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-17T17:54:03.015 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa; will attempt to use it 2023-10-17T17:54:03.016 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks 2023-10-17T17:54:03.016 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-17T17:54:03.018 INFO:teuthology.task.internal:Checking packages... 2023-10-17T17:54:03.040 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cff47d96488123dae42a0c6258323ad6422fe0a6' 2023-10-17T17:54:03.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-17T17:54:03.041 INFO:teuthology.packaging:ref: None 2023-10-17T17:54:03.041 INFO:teuthology.packaging:tag: None 2023-10-17T17:54:03.041 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-16-1247-reef 2023-10-17T17:54:03.042 INFO:teuthology.packaging:sha1: cff47d96488123dae42a0c6258323ad6422fe0a6 2023-10-17T17:54:03.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-16-1247-reef 2023-10-17T17:54:03.282 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-708.gcff47d96 2023-10-17T17:54:03.283 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-17T17:54:03.289 INFO:teuthology.task.internal:no buildpackages task found 2023-10-17T17:54:03.289 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-17T17:54:03.338 INFO:teuthology.task.internal:Saving configuration 2023-10-17T17:54:03.355 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-17T17:54:03.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T17:54:03.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 17:47:27.961529', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKx21743cUlmo6TKUbvR5PAR3YJyl6RDmMH8t+R6LplmFaEgnLfqs3jO/mdESrHA1/7zybwQDizFBDdcbLTm/18='} 2023-10-17T17:54:03.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 17:47:27.959582', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxePA3TlvYqb+4IRWm+eI1CkcB6eXCdh5zeTPXhaTJWuWW7972TipdWZLJP4OU5UiJN2/V8uTym1MqHpDcG0e4='} 2023-10-17T17:54:03.411 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-17T17:54:03.416 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-10-17T17:54:03.416 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-10-17T17:54:03.417 INFO:teuthology.run_tasks:Running task console_log... 2023-10-17T17:54:03.502 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fef02ca0a60>, signals=[15]) 2023-10-17T17:54:03.503 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-17T17:54:03.509 INFO:teuthology.task.internal:Opening connections... 2023-10-17T17:54:03.509 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T17:54:03.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:54:03.578 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T17:54:03.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:54:03.648 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-17T17:54:03.655 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-10-17T17:54:03.678 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-10-17T17:54:03.678 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-10-17T17:54:03.739 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2023-10-17T17:54:03.739 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8" 2023-10-17T17:54:03.740 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2023-10-17T17:54:03.740 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2023-10-17T17:54:03.740 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8" 2023-10-17T17:54:03.740 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2023-10-17T17:54:03.740 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-17T17:54:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-10-17T17:54:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-17T17:54:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2023-10-17T17:54:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-17T17:54:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-17T17:54:03.742 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-17T17:54:03.742 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-10-17T17:54:03.760 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2023-10-17T17:54:03.781 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2023-10-17T17:54:03.782 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-10-17T17:54:03.843 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2023-10-17T17:54:03.844 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8" 2023-10-17T17:54:03.844 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2023-10-17T17:54:03.844 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2023-10-17T17:54:03.844 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8" 2023-10-17T17:54:03.845 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2023-10-17T17:54:03.845 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-17T17:54:03.845 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-10-17T17:54:03.845 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-17T17:54:03.845 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2023-10-17T17:54:03.846 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-17T17:54:03.846 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-17T17:54:03.846 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-17T17:54:03.846 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-10-17T17:54:03.868 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-17T17:54:03.877 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-17T17:54:03.895 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-17T17:54:03.896 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-10-17T17:54:03.899 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2023-10-17T17:54:03.918 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-17T17:54:03.924 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-17T17:54:03.924 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-10-17T17:54:03.960 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2023-10-17T17:54:03.990 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-17T17:54:04.051 INFO:teuthology.run_tasks:Running task kernel... 2023-10-17T17:54:04.066 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-17T17:54:04.067 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-17T17:54:04.067 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-17T17:54:04.068 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-17T17:54:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T17:54:04.088 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-10-17T17:54:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T17:54:04.090 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-10-17T17:54:04.147 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-489.el8.x86_64 2023-10-17T17:54:04.148 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-489.el8.x86_64 2023-10-17T17:54:04.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2023-10-17T17:54:04.152 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-489.el8.x86_64 2023-10-17T17:54:04.152 INFO:teuthology.task.kernel:Running kernel on smithi049: 4.18.0-489.el8.x86_64 2023-10-17T17:54:04.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-10-17T17:54:10.066 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 9.7 MB/s | 33 MB 00:03 2023-10-17T17:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 9.2 MB/s | 33 MB 00:03 2023-10-17T17:54:18.440 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 50 MB 00:02 2023-10-17T17:54:19.485 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 50 MB 00:02 2023-10-17T17:54:25.740 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 113 kB/s | 18 kB 00:00 2023-10-17T17:54:26.073 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 44 kB/s | 6.9 kB 00:00 2023-10-17T17:54:26.409 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-10-17T17:54:27.210 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 122 kB/s | 18 kB 00:00 2023-10-17T17:54:27.677 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2023-10-17T17:54:27.995 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-10-17T17:54:28.692 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2023-10-17T17:54:29.282 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-17T17:54:32.534 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 376 kB/s | 24 kB 00:00 2023-10-17T17:54:33.313 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 467 kB/s | 24 kB 00:00 2023-10-17T17:54:36.058 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-17T17:54:36.058 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-17T17:54:36.311 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T17:54:36.311 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T17:54:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-10-17T17:54:36.312 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T17:54:36.312 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T17:54:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-17T17:54:36.312 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T17:54:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-17T17:54:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-17T17:54:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:54:36.313 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T17:54:36.313 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T17:54:36.314 INFO:teuthology.orchestra.run.smithi071.stdout:Install 3 Packages 2023-10-17T17:54:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:54:36.314 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 89 M 2023-10-17T17:54:36.315 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 96 M 2023-10-17T17:54:36.315 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T17:54:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-17T17:54:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-17T17:54:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T17:54:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T17:54:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-10-17T17:54:37.269 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T17:54:37.269 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T17:54:37.269 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-17T17:54:37.269 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T17:54:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-17T17:54:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-17T17:54:37.270 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:54:37.270 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T17:54:37.271 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T17:54:37.271 INFO:teuthology.orchestra.run.smithi049.stdout:Install 3 Packages 2023-10-17T17:54:37.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:54:37.272 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 89 M 2023-10-17T17:54:37.272 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 96 M 2023-10-17T17:54:37.272 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T17:54:37.785 INFO:teuthology.orchestra.run.smithi071.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 7.8 MB/s | 10 MB 00:01 2023-10-17T17:54:39.043 INFO:teuthology.orchestra.run.smithi049.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 6.1 MB/s | 10 MB 00:01 2023-10-17T17:54:39.168 INFO:teuthology.orchestra.run.smithi071.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2023-10-17T17:54:39.694 INFO:teuthology.orchestra.run.smithi071.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2023-10-17T17:54:39.694 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T17:54:39.694 INFO:teuthology.orchestra.run.smithi071.stdout:Total 26 MB/s | 89 MB 00:03 2023-10-17T17:54:40.366 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T17:54:40.422 INFO:teuthology.orchestra.run.smithi049.stdout:(2/3): kernel-core-4.18.0-517.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2023-10-17T17:54:40.773 INFO:teuthology.orchestra.run.smithi049.stdout:(3/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2023-10-17T17:54:40.773 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T17:54:40.774 INFO:teuthology.orchestra.run.smithi049.stdout:Total 25 MB/s | 89 MB 00:03 2023-10-17T17:54:40.782 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T17:54:40.783 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T17:54:41.471 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T17:54:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T17:54:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T17:54:42.010 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T17:54:42.030 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T17:54:43.224 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T17:54:43.235 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T17:54:44.806 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T17:54:45.358 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:54:46.146 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T17:54:46.884 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:54:48.903 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:54:49.307 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:54:50.631 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:54:51.017 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:54:52.684 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:54:52.936 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:54:54.647 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:54:54.926 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:14.501 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:14.524 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:14.853 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:14.854 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:55:14.854 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:55:15.587 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:15.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:55:15.588 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T17:55:15.588 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-17T17:55:15.588 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-17T17:55:15.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:55:15.589 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T17:55:15.728 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T17:55:15.943 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-17T17:55:15.944 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-17T17:55:15.944 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-17T17:55:15.945 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-17T17:55:15.945 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-17T17:55:15.945 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-10-17T17:55:15.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2023-10-17T17:55:15.946 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T17:55:16.154 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-17T17:55:16.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-17T17:55:16.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2023-10-17T17:55:16.188 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2023-10-17T17:55:16.188 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2023-10-17T17:55:16.188 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2023-10-17T17:55:16.189 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 129.el8 2023-10-17T17:55:16.189 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2023-10-17T17:55:16.189 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-17T17:55:16.189 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2023-10-17T17:55:16.189 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9265586 2023-10-17T17:55:16.190 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2023-10-17T17:55:16.190 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-17T17:55:16.190 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-17T17:55:16.190 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-17T17:55:16.190 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2023-10-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : CentOS BuildSystem 2023-10-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2023-10-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2023-10-17T17:55:16.192 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2023-10-17T17:55:16.192 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:55:16.192 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-17T17:55:16.192 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-17T17:55:16.192 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-17T17:55:16.193 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2023-10-17T17:55:16.193 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T17:55:16.193 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2023-10-17T17:55:16.194 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-17T17:55:16.194 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-517.el8.x86_64 2023-10-17T17:55:16.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-17T17:55:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:18.048 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2023-10-17T17:55:18.069 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:18.723 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:18.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-17T17:55:18.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-17T17:55:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-17T17:55:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:55:19.564 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T17:55:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-17T17:55:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-17T17:55:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:55:19.565 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T17:55:19.719 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T17:55:19.945 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-17T17:55:19.945 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-17T17:55:19.946 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-17T17:55:19.946 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-17T17:55:19.946 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-17T17:55:19.946 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-10-17T17:55:19.947 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2023-10-17T17:55:19.947 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-10-17T17:55:20.166 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-17T17:55:20.166 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-17T17:55:20.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2023-10-17T17:55:20.200 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2023-10-17T17:55:20.201 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2023-10-17T17:55:20.201 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.02 2023-10-17T17:55:20.201 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 129.el8 2023-10-17T17:55:20.201 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2023-10-17T17:55:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-17T17:55:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:Group : System Environment/Base 2023-10-17T17:55:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 9265586 2023-10-17T17:55:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2023-10-17T17:55:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-17T17:55:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-17T17:55:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-17T17:55:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-17T17:55:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:Relocations : (not relocatable) 2023-10-17T17:55:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : CentOS BuildSystem 2023-10-17T17:55:20.204 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2023-10-17T17:55:20.204 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-17T17:55:20.204 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2023-10-17T17:55:20.204 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2023-10-17T17:55:20.204 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:55:20.205 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-17T17:55:20.205 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-17T17:55:20.205 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-17T17:55:20.205 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2023-10-17T17:55:20.205 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T17:55:20.206 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2023-10-17T17:55:20.207 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-17T17:55:20.207 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 4.18.0-517.el8.x86_64 2023-10-17T17:55:20.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-17T17:55:20.234 INFO:teuthology.orchestra.run.smithi071.stderr:done 2023-10-17T17:55:20.236 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-10-17T17:55:20.267 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.C4GpFtLAYt 2023-10-17T17:55:20.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.C4GpFtLAYt 2023-10-17T17:55:20.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.C4GpFtLAYt 2023-10-17T17:55:20.488 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.C4GpFtLAYt is 6KB 2023-10-17T17:55:20.535 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.C4GpFtLAYt 2023-10-17T17:55:20.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2023-10-17T17:55:20.626 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-17T17:55:20.626 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-17T17:55:20.627 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-17T17:55:20.627 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-17T17:55:20.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-17T17:55:20.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2023-10-17T17:55:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2023-10-17T17:55:23.983 INFO:teuthology.orchestra.run.smithi049.stderr:done 2023-10-17T17:55:23.985 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-17T17:55:24.020 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.D8LynyupyG 2023-10-17T17:55:24.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D8LynyupyG 2023-10-17T17:55:24.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.D8LynyupyG 2023-10-17T17:55:24.351 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.D8LynyupyG is 6KB 2023-10-17T17:55:24.399 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.D8LynyupyG 2023-10-17T17:55:24.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2023-10-17T17:55:24.491 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-17T17:55:24.491 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-17T17:55:24.492 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-17T17:55:24.492 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-17T17:55:24.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-17T17:55:24.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2023-10-17T17:55:50.814 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-10-17T17:55:50.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-10-17T17:55:50.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:55:54.666 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-10-17T17:55:54.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:55:54.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:56:29.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T17:56:38.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:56:38.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:56:41.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T17:56:50.821 DEBUG:teuthology.orchestra.remote:timed out 2023-10-17T17:56:53.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:56:53.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:56:56.314 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-10-17T17:56:59.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-10-17T17:56:59.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:57:00.254 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-10-17T17:57:00.478 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-10-17T17:57:00.478 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-17T17:57:00.479 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-10-17T17:57:00.495 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-517.el8.x86_64 2023-10-17T17:57:00.496 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-17T17:57:00.496 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-17T17:57:00.496 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-17T17:57:01.498 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-10-17T17:57:01.498 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-17T17:57:01.561 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2023-10-17T17:57:01.577 DEBUG:teuthology.parallel:result is None 2023-10-17T17:57:11.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:57:11.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:58:11.319 DEBUG:teuthology.orchestra.remote:timed out 2023-10-17T17:58:29.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:58:29.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T17:58:29.665 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-10-17T17:58:29.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T17:58:29.950 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-17T17:58:29.950 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-10-17T17:58:29.970 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-517.el8.x86_64 2023-10-17T17:58:29.971 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-17T17:58:29.971 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-17T17:58:29.971 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-17T17:58:30.972 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-10-17T17:58:30.972 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-17T17:58:31.032 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2023-10-17T17:58:31.044 DEBUG:teuthology.parallel:result is None 2023-10-17T17:58:31.044 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-17T17:58:31.052 INFO:teuthology.task.internal:Creating test directory... 2023-10-17T17:58:31.053 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-17T17:58:31.055 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-17T17:58:31.079 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-17T17:58:31.095 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-17T17:58:31.128 INFO:teuthology.task.internal:Creating archive directory... 2023-10-17T17:58:31.129 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-17T17:58:31.131 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-17T17:58:31.172 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-17T17:58:31.180 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-17T17:58:31.181 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-17T17:58:31.209 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-17T17:58:31.244 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T17:58:31.255 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T17:58:31.264 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T17:58:31.276 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-17T17:58:31.277 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-17T17:58:31.288 INFO:teuthology.task.internal:Configuring sudo... 2023-10-17T17:58:31.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-17T17:58:31.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-17T17:58:31.329 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-17T17:58:31.339 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-17T17:58:31.339 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-17T17:58:31.373 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-17T17:58:31.392 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T17:58:31.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T17:58:31.538 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T17:58:31.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T17:58:31.628 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T17:58:31.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-17T17:58:31.696 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T17:58:31.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-17T17:58:31.791 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T17:58:31.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-17T17:58:31.921 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T17:58:31.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-17T17:58:31.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-10-17T17:58:31.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2023-10-17T17:58:32.045 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T17:58:32.097 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T17:58:32.516 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-17T17:58:32.524 INFO:teuthology.task.internal:Starting timer... 2023-10-17T17:58:32.524 INFO:teuthology.run_tasks:Running task pcp... 2023-10-17T17:58:32.556 INFO:teuthology.run_tasks:Running task selinux... 2023-10-17T17:58:32.593 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-17T17:58:32.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-10-17T17:58:32.664 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-10-17T17:58:32.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2023-10-17T17:58:32.745 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2023-10-17T17:58:32.746 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-17T17:58:32.747 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-10-17T17:58:32.775 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-10-17T17:58:32.775 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2023-10-17T17:58:32.823 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2023-10-17T17:58:32.823 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2023-10-17T17:58:32.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T17:58:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T17:58:32.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T17:58:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T17:58:32.880 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-17T17:58:32.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-10-17T17:58:32.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2023-10-17T17:58:32.956 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-17T17:58:32.967 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-17T17:58:32.972 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-17T17:58:33.089 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-17T17:58:33.106 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-17T17:58:33.108 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-17T18:05:58.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2023-10-17T18:05:58.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T18:05:58.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T18:05:58.910 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-10-17T18:05:58.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-17T18:05:58.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-10-17T18:05:58.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-17T18:05:59.048 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-10-17T18:05:59.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-10-17T18:05:59.119 INFO:teuthology.run_tasks:Running task clock... 2023-10-17T18:05:59.132 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-17T18:05:59.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-17T18:05:59.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T18:05:59.136 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-17T18:05:59.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T18:05:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-17T18:05:59.181 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-17T18:05:59.192 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-17T18:05:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-17T18:05:59.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-10-17T18:05:59.254 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-10-17T18:05:59.269 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-17T18:05:59.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2023-10-17T18:05:59.283 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-17T18:05:59.290 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2023-10-17T18:05:59.304 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-17T18:05:59.317 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-17T18:05:59.348 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-10-17T18:05:59.353 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T18:05:59.353 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-17T18:05:59.353 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.353 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.354 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.354 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.396 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-10-17T18:05:59.400 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T18:05:59.401 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-10-17T18:05:59.401 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.401 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.401 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.401 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T18:05:59.402 INFO:teuthology.run_tasks:Running task install... 2023-10-17T18:05:59.411 DEBUG:teuthology.task.install:project ceph 2023-10-17T18:05:59.411 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'}} 2023-10-17T18:05:59.411 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'} 2023-10-17T18:05:59.412 INFO:teuthology.task.install:Using flavor: default 2023-10-17T18:05:59.417 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-17T18:05:59.417 INFO:teuthology.task.install:extra packages: [] 2023-10-17T18:05:59.418 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'tag': None, 'wait_for_package': False} 2023-10-17T18:05:59.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cff47d96488123dae42a0c6258323ad6422fe0a6 2023-10-17T18:05:59.421 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'tag': None, 'wait_for_package': False} 2023-10-17T18:05:59.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cff47d96488123dae42a0c6258323ad6422fe0a6 2023-10-17T18:05:59.563 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/ 2023-10-17T18:05:59.563 INFO:teuthology.task.install.rpm:Package version is 18.2.0-708.gcff47d96 2023-10-17T18:05:59.580 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/ 2023-10-17T18:05:59.580 INFO:teuthology.task.install.rpm:Package version is 18.2.0-708.gcff47d96 2023-10-17T18:05:59.683 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-17T18:05:59.684 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:05:59.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-17T18:05:59.713 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-17T18:05:59.713 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cff47d96488123dae42a0c6258323ad6422fe0a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-17T18:05:59.737 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-16-1247-reef/cff47d96488123dae42a0c6258323ad6422fe0a6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-17T18:05:59.737 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:05:59.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-17T18:05:59.767 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-17T18:05:59.767 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cff47d96488123dae42a0c6258323ad6422fe0a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-17T18:05:59.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-17T18:05:59.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-17T18:05:59.858 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-17T18:05:59.915 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-17T18:05:59.924 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2023-10-17T18:05:59.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2023-10-17T18:05:59.982 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-10-17T18:05:59.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-10-17T18:06:00.201 INFO:teuthology.orchestra.run.smithi071.stdout:76 files removed 2023-10-17T18:06:00.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2023-10-17T18:06:00.270 INFO:teuthology.orchestra.run.smithi049.stdout:76 files removed 2023-10-17T18:06:00.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2023-10-17T18:06:01.068 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 183 kB/s | 80 kB 00:00 2023-10-17T18:06:01.149 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 191 kB/s | 80 kB 00:00 2023-10-17T18:06:01.369 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 36 kB/s | 9.4 kB 00:00 2023-10-17T18:06:01.483 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-17T18:06:01.678 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 5.3 kB/s | 1.5 kB 00:00 2023-10-17T18:06:01.775 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2023-10-17T18:06:03.417 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 21 MB/s | 33 MB 00:01 2023-10-17T18:06:03.511 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 19 MB/s | 33 MB 00:01 2023-10-17T18:06:11.770 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 50 MB 00:02 2023-10-17T18:06:13.251 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 50 MB 00:03 2023-10-17T18:06:19.095 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 80 kB/s | 18 kB 00:00 2023-10-17T18:06:19.445 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 39 kB/s | 6.9 kB 00:00 2023-10-17T18:06:20.522 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - PowerTools 7.0 MB/s | 6.5 MB 00:00 2023-10-17T18:06:21.192 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2023-10-17T18:06:21.543 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 43 kB/s | 6.9 kB 00:00 2023-10-17T18:06:22.048 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-10-17T18:06:22.478 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - PowerTools 8.4 MB/s | 6.5 MB 00:00 2023-10-17T18:06:23.990 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 16 MB 00:01 2023-10-17T18:06:24.013 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-10-17T18:06:24.971 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-10-17T18:06:27.890 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 229 kB/s | 24 kB 00:00 2023-10-17T18:06:29.063 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 427 kB/s | 24 kB 00:00 2023-10-17T18:06:32.323 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:06:32.326 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:06:32.326 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:06:32.326 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:06:32.327 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:06:32.327 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:18.2.0-708.gcff47d96.el8 ceph 13 M 2023-10-17T18:06:32.327 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-10-17T18:06:32.327 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.7 M 2023-10-17T18:06:32.327 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.6 M 2023-10-17T18:06:32.328 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:06:32.328 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:18.2.0-708.gcff47d96.el8 ceph 5.4 M 2023-10-17T18:06:32.328 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:18.2.0-708.gcff47d96.el8 ceph 22 M 2023-10-17T18:06:32.328 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:18.2.0-708.gcff47d96.el8 ceph 24 k 2023-10-17T18:06:32.328 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-17T18:06:32.329 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 770 k 2023-10-17T18:06:32.329 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-17T18:06:32.329 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-17T18:06:32.329 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 474 k 2023-10-17T18:06:32.330 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-17T18:06:32.330 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 6.5 M 2023-10-17T18:06:32.330 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-17T18:06:32.330 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-17T18:06:32.330 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 45 k 2023-10-17T18:06:32.331 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:18.2.0-708.gcff47d96.el8 ceph 123 k 2023-10-17T18:06:32.331 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:18.2.0-708.gcff47d96.el8 ceph 190 k 2023-10-17T18:06:32.331 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-17T18:06:32.331 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:18.2.0-708.gcff47d96.el8 ceph 371 k 2023-10-17T18:06:32.331 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 365 k 2023-10-17T18:06:32.332 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:18.2.0-708.gcff47d96.el8 ceph 107 k 2023-10-17T18:06:32.332 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-17T18:06:32.332 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:06:32.332 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:06:32.332 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:06:32.333 INFO:teuthology.orchestra.run.smithi071.stdout:Install 21 Packages 2023-10-17T18:06:32.333 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2023-10-17T18:06:32.333 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:06:32.334 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 59 M 2023-10-17T18:06:32.334 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:06:33.767 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:06:33.770 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:06:33.770 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:06:33.770 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:06:33.771 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:06:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:18.2.0-708.gcff47d96.el8 ceph 13 M 2023-10-17T18:06:33.771 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-10-17T18:06:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.7 M 2023-10-17T18:06:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.6 M 2023-10-17T18:06:33.772 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:06:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:18.2.0-708.gcff47d96.el8 ceph 5.4 M 2023-10-17T18:06:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:18.2.0-708.gcff47d96.el8 ceph 22 M 2023-10-17T18:06:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:18.2.0-708.gcff47d96.el8 ceph 24 k 2023-10-17T18:06:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-17T18:06:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 770 k 2023-10-17T18:06:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-17T18:06:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-17T18:06:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 474 k 2023-10-17T18:06:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-17T18:06:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:18.2.0-708.gcff47d96.el8 ceph 6.5 M 2023-10-17T18:06:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-17T18:06:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-17T18:06:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 45 k 2023-10-17T18:06:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:18.2.0-708.gcff47d96.el8 ceph 123 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:18.2.0-708.gcff47d96.el8 ceph 190 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:18.2.0-708.gcff47d96.el8 ceph 371 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 365 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:18.2.0-708.gcff47d96.el8 ceph 107 k 2023-10-17T18:06:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-17T18:06:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:06:33.776 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:06:33.776 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:06:33.777 INFO:teuthology.orchestra.run.smithi049.stdout:Install 21 Packages 2023-10-17T18:06:33.777 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2023-10-17T18:06:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:06:33.778 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 59 M 2023-10-17T18:06:33.778 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:06:35.083 INFO:teuthology.orchestra.run.smithi049.stdout:(1/23): ceph-base-18.2.0-708.gcff47d96.el8.x86_ 5.4 MB/s | 5.4 MB 00:01 2023-10-17T18:06:40.141 INFO:teuthology.orchestra.run.smithi049.stdout:(2/23): ceph-selinux-18.2.0-708.gcff47d96.el8.x 4.8 kB/s | 24 kB 00:05 2023-10-17T18:06:40.683 INFO:teuthology.orchestra.run.smithi049.stdout:(3/23): libcephfs2-18.2.0-708.gcff47d96.el8.x86 1.4 MB/s | 770 kB 00:00 2023-10-17T18:06:40.917 INFO:teuthology.orchestra.run.smithi049.stdout:(4/23): libradosstriper1-18.2.0-708.gcff47d96.e 2.0 MB/s | 474 kB 00:00 2023-10-17T18:06:41.300 INFO:teuthology.orchestra.run.smithi049.stdout:(5/23): ceph-radosgw-18.2.0-708.gcff47d96.el8.x 1.8 MB/s | 13 MB 00:07 2023-10-17T18:06:41.417 INFO:teuthology.orchestra.run.smithi049.stdout:(6/23): python3-ceph-argparse-18.2.0-708.gcff47 383 kB/s | 45 kB 00:00 2023-10-17T18:06:41.526 INFO:teuthology.orchestra.run.smithi049.stdout:(7/23): python3-ceph-common-18.2.0-708.gcff47d9 1.1 MB/s | 123 kB 00:00 2023-10-17T18:06:41.556 INFO:teuthology.orchestra.run.smithi071.stdout:(1/23): ceph-common-18.2.0-708.gcff47d96.el8.x8 2.5 MB/s | 22 MB 00:08 2023-10-17T18:06:41.623 INFO:teuthology.orchestra.run.smithi071.stdout:(2/23): ceph-selinux-18.2.0-708.gcff47d96.el8.x 367 kB/s | 24 kB 00:00 2023-10-17T18:06:41.693 INFO:teuthology.orchestra.run.smithi049.stdout:(8/23): python3-cephfs-18.2.0-708.gcff47d96.el8 1.1 MB/s | 190 kB 00:00 2023-10-17T18:06:41.843 INFO:teuthology.orchestra.run.smithi049.stdout:(9/23): python3-rados-18.2.0-708.gcff47d96.el8. 2.4 MB/s | 371 kB 00:00 2023-10-17T18:06:41.848 INFO:teuthology.orchestra.run.smithi071.stdout:(3/23): libcephfs2-18.2.0-708.gcff47d96.el8.x86 3.3 MB/s | 770 kB 00:00 2023-10-17T18:06:41.907 INFO:teuthology.orchestra.run.smithi071.stdout:(4/23): libradosstriper1-18.2.0-708.gcff47d96.e 8.0 MB/s | 474 kB 00:00 2023-10-17T18:06:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:(10/23): librgw2-18.2.0-708.gcff47d96.el8.x86_6 6.5 MB/s | 6.5 MB 00:01 2023-10-17T18:06:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:(11/23): python3-rbd-18.2.0-708.gcff47d96.el8.x 3.3 MB/s | 365 kB 00:00 2023-10-17T18:06:42.074 INFO:teuthology.orchestra.run.smithi071.stdout:(5/23): ceph-radosgw-18.2.0-708.gcff47d96.el8.x 1.4 MB/s | 13 MB 00:09 2023-10-17T18:06:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:(12/23): python3-rgw-18.2.0-708.gcff47d96.el8.x 711 kB/s | 107 kB 00:00 2023-10-17T18:06:42.132 INFO:teuthology.orchestra.run.smithi071.stdout:(6/23): python3-ceph-argparse-18.2.0-708.gcff47 770 kB/s | 45 kB 00:00 2023-10-17T18:06:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:(7/23): python3-ceph-common-18.2.0-708.gcff47d9 2.1 MB/s | 123 kB 00:00 2023-10-17T18:06:42.258 INFO:teuthology.orchestra.run.smithi071.stdout:(8/23): python3-cephfs-18.2.0-708.gcff47d96.el8 2.8 MB/s | 190 kB 00:00 2023-10-17T18:06:42.286 INFO:teuthology.orchestra.run.smithi049.stdout:(13/23): ceph-common-18.2.0-708.gcff47d96.el8.x 2.6 MB/s | 22 MB 00:08 2023-10-17T18:06:42.350 INFO:teuthology.orchestra.run.smithi071.stdout:(9/23): python3-rados-18.2.0-708.gcff47d96.el8. 4.0 MB/s | 371 kB 00:00 2023-10-17T18:06:42.428 INFO:teuthology.orchestra.run.smithi049.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 744 kB/s | 354 kB 00:00 2023-10-17T18:06:42.483 INFO:teuthology.orchestra.run.smithi071.stdout:(10/23): librgw2-18.2.0-708.gcff47d96.el8.x86_6 11 MB/s | 6.5 MB 00:00 2023-10-17T18:06:42.517 INFO:teuthology.orchestra.run.smithi071.stdout:(11/23): python3-rbd-18.2.0-708.gcff47d96.el8.x 2.1 MB/s | 365 kB 00:00 2023-10-17T18:06:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 182 kB/s | 44 kB 00:00 2023-10-17T18:06:42.542 INFO:teuthology.orchestra.run.smithi071.stdout:(12/23): python3-rgw-18.2.0-708.gcff47d96.el8.x 1.8 MB/s | 107 kB 00:00 2023-10-17T18:06:42.570 INFO:teuthology.orchestra.run.smithi049.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 527 kB/s | 259 kB 00:00 2023-10-17T18:06:42.764 INFO:teuthology.orchestra.run.smithi049.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 303 kB/s | 59 kB 00:00 2023-10-17T18:06:42.854 INFO:teuthology.orchestra.run.smithi049.stdout:(18/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 110 kB/s | 47 kB 00:00 2023-10-17T18:06:42.887 INFO:teuthology.orchestra.run.smithi049.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 610 kB/s | 75 kB 00:00 2023-10-17T18:06:42.938 INFO:teuthology.orchestra.run.smithi049.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 747 kB/s | 306 kB 00:00 2023-10-17T18:06:42.984 INFO:teuthology.orchestra.run.smithi071.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 588 kB/s | 259 kB 00:00 2023-10-17T18:06:43.084 INFO:teuthology.orchestra.run.smithi071.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 440 kB/s | 44 kB 00:00 2023-10-17T18:06:43.110 INFO:teuthology.orchestra.run.smithi071.stdout:(15/23): librdkafka-0.11.4-3.el8.x86_64.rpm 597 kB/s | 354 kB 00:00 2023-10-17T18:06:43.510 INFO:teuthology.orchestra.run.smithi071.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 765 kB/s | 306 kB 00:00 2023-10-17T18:06:43.535 INFO:teuthology.orchestra.run.smithi071.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 104 kB/s | 47 kB 00:00 2023-10-17T18:06:43.586 INFO:teuthology.orchestra.run.smithi071.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 782 kB/s | 59 kB 00:00 2023-10-17T18:06:43.744 INFO:teuthology.orchestra.run.smithi071.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 360 kB/s | 75 kB 00:00 2023-10-17T18:06:43.786 INFO:teuthology.orchestra.run.smithi071.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 8.7 MB/s | 1.7 MB 00:00 2023-10-17T18:06:44.128 INFO:teuthology.orchestra.run.smithi071.stdout:(21/23): librados2-18.2.0-708.gcff47d96.el8.x86 9.5 MB/s | 3.7 MB 00:00 2023-10-17T18:06:44.270 INFO:teuthology.orchestra.run.smithi071.stdout:(22/23): librbd1-18.2.0-708.gcff47d96.el8.x86_6 7.4 MB/s | 3.6 MB 00:00 2023-10-17T18:06:45.246 INFO:teuthology.orchestra.run.smithi049.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 742 kB/s | 1.7 MB 00:02 2023-10-17T18:06:45.630 INFO:teuthology.orchestra.run.smithi049.stdout:(22/23): librados2-18.2.0-708.gcff47d96.el8.x86 1.3 MB/s | 3.7 MB 00:02 2023-10-17T18:06:45.728 INFO:teuthology.orchestra.run.smithi071.stdout:(23/23): ceph-base-18.2.0-708.gcff47d96.el8.x86 430 kB/s | 5.4 MB 00:12 2023-10-17T18:06:45.731 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:06:45.731 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.4 MB/s | 59 MB 00:13 2023-10-17T18:06:45.773 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:06:45.819 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:06:45.820 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:06:46.183 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:06:46.186 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:06:46.755 INFO:teuthology.orchestra.run.smithi049.stdout:(23/23): librbd1-18.2.0-708.gcff47d96.el8.x86_6 955 kB/s | 3.6 MB 00:03 2023-10-17T18:06:46.757 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:06:46.758 INFO:teuthology.orchestra.run.smithi049.stdout:Total 4.5 MB/s | 59 MB 00:12 2023-10-17T18:06:46.802 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:06:46.852 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:06:46.852 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:06:47.224 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:06:47.228 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:06:47.467 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:06:47.674 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-17T18:06:47.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-17T18:06:48.100 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-17T18:06:48.495 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:06:48.506 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-17T18:06:48.684 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-17T18:06:48.691 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:06:48.787 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:06:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-17T18:06:49.260 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-17T18:06:49.274 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 5/25 2023-10-17T18:06:49.442 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:06:49.692 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-17T18:06:49.836 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:06:49.885 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:06:49.982 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:06:50.066 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:06:50.134 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:06:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 5/25 2023-10-17T18:06:50.356 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-17T18:06:50.385 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:06:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-17T18:06:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-17T18:06:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x 10/25 2023-10-17T18:06:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:06:50.986 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 11/25 2023-10-17T18:06:51.001 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:06:51.070 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:06:51.226 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 12/25 2023-10-17T18:06:51.287 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-17T18:06:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:06:51.428 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:06:51.461 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-17T18:06:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-17T18:06:51.669 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-17T18:06:51.708 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x 10/25 2023-10-17T18:06:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 11/25 2023-10-17T18:06:52.121 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 12/25 2023-10-17T18:06:52.251 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:06:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:06:52.405 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-17T18:06:52.513 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-17T18:06:52.558 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 16/25 2023-10-17T18:06:52.631 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 16/25 2023-10-17T18:06:52.825 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 17/25 2023-10-17T18:06:53.000 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-17T18:06:53.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86 19/25 2023-10-17T18:06:53.322 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-17T18:06:53.443 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 16/25 2023-10-17T18:06:53.517 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 16/25 2023-10-17T18:06:53.709 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 17/25 2023-10-17T18:06:54.193 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-17T18:06:54.343 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86 19/25 2023-10-17T18:06:55.737 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:55.966 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:56.583 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:56.846 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 21/25 2023-10-17T18:06:56.958 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 21/25 2023-10-17T18:06:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:57.148 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:06:57.218 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 20/25 2023-10-17T18:06:58.084 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 21/25 2023-10-17T18:06:58.195 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 21/25 2023-10-17T18:06:58.384 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:07:15.752 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2023-10-17T18:07:15.753 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2023-10-17T18:07:15.754 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2023-10-17T18:07:15.754 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2023-10-17T18:07:15.754 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-10-17T18:07:15.754 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:17.159 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 23/25 2023-10-17T18:07:17.179 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 23/25 2023-10-17T18:07:17.180 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-17T18:07:17.430 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-17T18:07:17.431 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:17.906 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:07:17.907 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-10-17T18:07:17.907 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-10-17T18:07:17.907 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-10-17T18:07:17.907 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /run 2023-10-17T18:07:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-10-17T18:07:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-10-17T18:07:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-10-17T18:07:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-10-17T18:07:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-10-17T18:07:17.909 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:18.151 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:18.151 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 1/25 2023-10-17T18:07:18.151 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2/25 2023-10-17T18:07:18.152 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 3/25 2023-10-17T18:07:18.152 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:07:18.152 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 5/25 2023-10-17T18:07:18.152 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:07:18.153 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:07:18.153 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x 8/25 2023-10-17T18:07:18.153 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86 9/25 2023-10-17T18:07:18.153 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 10/25 2023-10-17T18:07:18.153 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 11/25 2023-10-17T18:07:18.154 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 12/25 2023-10-17T18:07:18.154 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:07:18.154 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-17T18:07:18.154 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-17T18:07:18.155 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-17T18:07:18.155 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-17T18:07:18.155 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-17T18:07:18.155 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-17T18:07:18.155 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-17T18:07:18.156 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-17T18:07:18.156 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:07:18.156 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-17T18:07:18.156 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 24/25 2023-10-17T18:07:19.250 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:19.250 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:19.250 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-10-17T18:07:19.251 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.251 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.251 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:07:19.251 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.251 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-17T18:07:19.252 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-17T18:07:19.253 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.253 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-17T18:07:19.253 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.253 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-17T18:07:19.253 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-17T18:07:19.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-17T18:07:19.254 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.255 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.255 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:19.255 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-17T18:07:19.255 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:19.255 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:07:19.339 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 23/25 2023-10-17T18:07:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 23/25 2023-10-17T18:07:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-17T18:07:19.541 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-17T18:07:19.542 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:19.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2023-10-17T18:07:20.259 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:20.259 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 1/25 2023-10-17T18:07:20.259 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2/25 2023-10-17T18:07:20.259 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 3/25 2023-10-17T18:07:20.260 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 4/25 2023-10-17T18:07:20.260 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 5/25 2023-10-17T18:07:20.260 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 6/25 2023-10-17T18:07:20.260 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 7/25 2023-10-17T18:07:20.260 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x 8/25 2023-10-17T18:07:20.261 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86 9/25 2023-10-17T18:07:20.261 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 10/25 2023-10-17T18:07:20.261 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 11/25 2023-10-17T18:07:20.261 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 12/25 2023-10-17T18:07:20.261 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 13/25 2023-10-17T18:07:20.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-17T18:07:20.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-17T18:07:20.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-17T18:07:20.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-17T18:07:20.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-17T18:07:20.263 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-17T18:07:20.263 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-17T18:07:20.263 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-17T18:07:20.263 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:18.2.0-708.gcff47d96.el8.x86_64 22/25 2023-10-17T18:07:20.263 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-17T18:07:20.264 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 24/25 2023-10-17T18:07:20.408 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:07:21.385 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-17T18:07:21.385 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:21.386 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-10-17T18:07:21.386 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.386 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.386 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:07:21.386 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.387 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.387 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.387 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.387 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-17T18:07:21.388 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.388 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-17T18:07:21.388 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-17T18:07:21.388 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.388 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-17T18:07:21.389 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.389 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-17T18:07:21.389 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-17T18:07:21.389 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.389 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.390 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.390 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-17T18:07:21.390 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.390 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.390 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:21.391 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-17T18:07:21.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:21.391 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:07:21.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2023-10-17T18:07:22.005 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:07:22.006 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:22.006 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:07:22.007 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:22.007 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:07:22.007 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:18.2.0-708.gcff47d96.el8 ceph 47 M 2023-10-17T18:07:22.007 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:07:22.007 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-17T18:07:22.008 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:18.2.0-708.gcff47d96.el8 ceph 137 k 2023-10-17T18:07:22.008 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-17T18:07:22.008 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-17T18:07:22.008 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-17T18:07:22.008 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:22.009 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:07:22.009 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:22.009 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-10-17T18:07:22.009 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:22.010 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2023-10-17T18:07:22.010 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 211 M 2023-10-17T18:07:22.010 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:07:22.279 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 898 kB/s | 202 kB 00:00 2023-10-17T18:07:22.337 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2023-10-17T18:07:22.412 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): libcephsqlite-18.2.0-708.gcff47d96.el8.x 381 kB/s | 137 kB 00:00 2023-10-17T18:07:22.454 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-10-17T18:07:22.510 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:07:22.638 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 313 kB/s | 71 kB 00:00 2023-10-17T18:07:24.224 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:07:24.225 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:24.225 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:07:24.225 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:24.225 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:07:24.226 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:18.2.0-708.gcff47d96.el8 ceph 47 M 2023-10-17T18:07:24.226 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:07:24.226 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-17T18:07:24.226 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:18.2.0-708.gcff47d96.el8 ceph 137 k 2023-10-17T18:07:24.226 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-17T18:07:24.227 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-17T18:07:24.227 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-17T18:07:24.227 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:24.227 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:07:24.227 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:24.228 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-10-17T18:07:24.228 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:24.228 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 48 M 2023-10-17T18:07:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 211 M 2023-10-17T18:07:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:07:24.683 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 748 kB/s | 202 kB 00:00 2023-10-17T18:07:24.834 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-17T18:07:25.017 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): libcephsqlite-18.2.0-708.gcff47d96.el8.x 226 kB/s | 137 kB 00:00 2023-10-17T18:07:25.043 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2023-10-17T18:07:25.185 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 423 kB/s | 71 kB 00:00 2023-10-17T18:07:25.388 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-18.2.0-708.gcff47d96.el8.x86_6 14 MB/s | 47 MB 00:03 2023-10-17T18:07:25.389 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:07:25.390 INFO:teuthology.orchestra.run.smithi071.stdout:Total 14 MB/s | 48 MB 00:03 2023-10-17T18:07:25.432 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:07:25.453 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:07:25.454 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:07:25.742 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:07:25.745 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:07:26.204 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:07:26.445 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-17T18:07:26.669 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-17T18:07:26.817 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-17T18:07:26.901 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-17T18:07:27.112 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-17T18:07:27.242 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 5/6 2023-10-17T18:07:27.676 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): ceph-test-18.2.0-708.gcff47d96.el8.x86_6 14 MB/s | 47 MB 00:03 2023-10-17T18:07:27.678 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:07:27.679 INFO:teuthology.orchestra.run.smithi049.stdout:Total 14 MB/s | 48 MB 00:03 2023-10-17T18:07:27.723 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:07:27.746 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:07:27.747 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:07:28.057 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:07:28.060 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:07:28.538 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:07:28.742 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-17T18:07:28.966 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-17T18:07:29.104 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-17T18:07:29.184 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-17T18:07:29.333 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-17T18:07:29.504 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 5/6 2023-10-17T18:07:31.747 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 5/6 2023-10-17T18:07:31.973 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 6/6 2023-10-17T18:07:32.243 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 6/6 2023-10-17T18:07:32.243 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 1/6 2023-10-17T18:07:32.243 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 2/6 2023-10-17T18:07:32.243 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-17T18:07:32.243 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-17T18:07:32.244 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-17T18:07:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-17T18:07:32.925 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:32.925 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-7.el8.x86_64 2023-10-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-17T18:07:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-17T18:07:32.927 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:32.927 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:07:33.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2023-10-17T18:07:33.990 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:06 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:07:34.321 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 5/6 2023-10-17T18:07:34.560 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 6/6 2023-10-17T18:07:34.857 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 6/6 2023-10-17T18:07:34.857 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 1/6 2023-10-17T18:07:34.857 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 2/6 2023-10-17T18:07:34.857 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-17T18:07:34.858 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-17T18:07:34.858 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-17T18:07:35.580 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-17T18:07:35.580 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-7.el8.x86_64 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-17T18:07:35.581 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-17T18:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-17T18:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:07:35.601 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:07:35.604 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:35.604 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:07:35.605 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:35.605 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:07:35.605 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:18.2.0-708.gcff47d96.el8 ceph 6.4 k 2023-10-17T18:07:35.605 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:07:35.605 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:18.2.0-708.gcff47d96.el8 ceph 2.3 M 2023-10-17T18:07:35.606 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:18.2.0-708.gcff47d96.el8 ceph 1.5 M 2023-10-17T18:07:35.606 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 242 k 2023-10-17T18:07:35.606 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:18.2.0-708.gcff47d96.el8 ceph 4.6 M 2023-10-17T18:07:35.606 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 18 M 2023-10-17T18:07:35.607 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-17T18:07:35.607 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-17T18:07:35.607 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-17T18:07:35.607 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-17T18:07:35.607 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-17T18:07:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-17T18:07:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-17T18:07:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-17T18:07:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-17T18:07:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-17T18:07:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-17T18:07:35.610 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-17T18:07:35.610 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-17T18:07:35.610 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-17T18:07:35.610 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-17T18:07:35.611 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-17T18:07:35.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:35.612 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:07:35.612 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:35.612 INFO:teuthology.orchestra.run.smithi071.stdout:Install 32 Packages 2023-10-17T18:07:35.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:35.613 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2023-10-17T18:07:35.613 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 119 M 2023-10-17T18:07:35.614 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:07:35.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2023-10-17T18:07:36.022 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-18.2.0-708.gcff47d96.el8.x86_64.rp 48 kB/s | 6.4 kB 00:00 2023-10-17T18:07:36.447 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-mon-18.2.0-708.gcff47d96.el8.x86_6 11 MB/s | 4.6 MB 00:00 2023-10-17T18:07:36.682 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:07:37.214 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): ceph-mds-18.2.0-708.gcff47d96.el8.x86_6 1.7 MB/s | 2.3 MB 00:01 2023-10-17T18:07:37.373 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): ceph-mgr-modules-core-18.2.0-708.gcff47 1.5 MB/s | 242 kB 00:00 2023-10-17T18:07:37.681 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): ceph-osd-18.2.0-708.gcff47d96.el8.x86_6 14 MB/s | 18 MB 00:01 2023-10-17T18:07:37.706 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 471 kB/s | 157 kB 00:00 2023-10-17T18:07:37.815 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 955 kB/s | 103 kB 00:00 2023-10-17T18:07:37.857 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): ceph-mgr-18.2.0-708.gcff47d96.el8.x86_6 805 kB/s | 1.5 MB 00:01 2023-10-17T18:07:37.883 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 193 kB/s | 39 kB 00:00 2023-10-17T18:07:37.908 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 617 kB/s | 57 kB 00:00 2023-10-17T18:07:38.308 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2023-10-17T18:07:38.402 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:07:38.405 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:38.405 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:07:38.406 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:38.406 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:07:38.406 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:18.2.0-708.gcff47d96.el8 ceph 6.4 k 2023-10-17T18:07:38.406 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:07:38.406 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:18.2.0-708.gcff47d96.el8 ceph 2.3 M 2023-10-17T18:07:38.407 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:18.2.0-708.gcff47d96.el8 ceph 1.5 M 2023-10-17T18:07:38.407 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 242 k 2023-10-17T18:07:38.407 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:18.2.0-708.gcff47d96.el8 ceph 4.6 M 2023-10-17T18:07:38.407 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 18 M 2023-10-17T18:07:38.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-17T18:07:38.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-17T18:07:38.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-17T18:07:38.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-17T18:07:38.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-17T18:07:38.409 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-17T18:07:38.409 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-17T18:07:38.409 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-17T18:07:38.409 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-17T18:07:38.409 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-17T18:07:38.410 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-17T18:07:38.410 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-17T18:07:38.410 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-17T18:07:38.410 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-17T18:07:38.410 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-17T18:07:38.411 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-17T18:07:38.411 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-17T18:07:38.411 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-17T18:07:38.411 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-17T18:07:38.411 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-17T18:07:38.412 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-17T18:07:38.413 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:38.413 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:07:38.413 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:07:38.413 INFO:teuthology.orchestra.run.smithi049.stdout:Install 32 Packages 2023-10-17T18:07:38.413 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:38.414 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 30 M 2023-10-17T18:07:38.414 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 119 M 2023-10-17T18:07:38.415 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:07:38.491 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 408 kB/s | 249 kB 00:00 2023-10-17T18:07:38.600 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 927 kB/s | 641 kB 00:00 2023-10-17T18:07:38.767 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 238 kB/s | 109 kB 00:00 2023-10-17T18:07:38.942 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 97 kB/s | 44 kB 00:00 2023-10-17T18:07:38.969 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-18.2.0-708.gcff47d96.el8.x86_64.rp 51 kB/s | 6.4 kB 00:00 2023-10-17T18:07:39.111 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-mgr-18.2.0-708.gcff47d96.el8.x86_6 5.8 MB/s | 1.5 MB 00:00 2023-10-17T18:07:39.234 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 294 kB/s | 185 kB 00:00 2023-10-17T18:07:39.334 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-10-17T18:07:39.368 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 907 kB/s | 384 kB 00:00 2023-10-17T18:07:39.393 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 276 kB/s | 173 kB 00:00 2023-10-17T18:07:39.443 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2023-10-17T18:07:39.485 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 416 kB/s | 49 kB 00:00 2023-10-17T18:07:39.519 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2023-10-17T18:07:39.636 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 104 kB/s | 16 kB 00:00 2023-10-17T18:07:39.661 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2023-10-17T18:07:39.670 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): ceph-mon-18.2.0-708.gcff47d96.el8.x86_6 6.6 MB/s | 4.6 MB 00:00 2023-10-17T18:07:39.687 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-10-17T18:07:39.745 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): ceph-mgr-modules-core-18.2.0-708.gcff47 3.2 MB/s | 242 kB 00:00 2023-10-17T18:07:39.753 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2023-10-17T18:07:39.787 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-10-17T18:07:39.812 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2023-10-17T18:07:39.921 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.5 MB/s | 255 kB 00:00 2023-10-17T18:07:39.946 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 639 kB/s | 85 kB 00:00 2023-10-17T18:07:40.047 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 970 kB/s | 251 kB 00:00 2023-10-17T18:07:40.063 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 162 kB/s | 23 kB 00:00 2023-10-17T18:07:40.067 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:07:40.068 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.7 MB/s | 30 MB 00:04 2023-10-17T18:07:40.118 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): ceph-osd-18.2.0-708.gcff47d96.el8.x86_6 18 MB/s | 18 MB 00:01 2023-10-17T18:07:40.146 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 392 kB/s | 157 kB 00:00 2023-10-17T18:07:40.151 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:07:40.188 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:07:40.188 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:07:40.254 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 957 kB/s | 103 kB 00:00 2023-10-17T18:07:40.279 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 241 kB/s | 39 kB 00:00 2023-10-17T18:07:40.313 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 975 kB/s | 57 kB 00:00 2023-10-17T18:07:40.504 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:07:40.507 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:07:40.538 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.7 MB/s | 457 kB 00:00 2023-10-17T18:07:40.914 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 414 kB/s | 249 kB 00:00 2023-10-17T18:07:41.072 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 689 kB/s | 109 kB 00:00 2023-10-17T18:07:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 894 kB/s | 641 kB 00:00 2023-10-17T18:07:41.298 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): ceph-mds-18.2.0-708.gcff47d96.el8.x86_ 952 kB/s | 2.3 MB 00:02 2023-10-17T18:07:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 168 kB/s | 44 kB 00:00 2023-10-17T18:07:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-10-17T18:07:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-10-17T18:07:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 651 kB/s | 185 kB 00:00 2023-10-17T18:07:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 211 kB/s | 18 kB 00:00 2023-10-17T18:07:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 731 kB/s | 49 kB 00:00 2023-10-17T18:07:41.633 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 881 kB/s | 59 kB 00:00 2023-10-17T18:07:41.655 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:07:41.658 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2023-10-17T18:07:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2023-10-17T18:07:41.726 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 279 kB/s | 19 kB 00:00 2023-10-17T18:07:41.751 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-10-17T18:07:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-10-17T18:07:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2023-10-17T18:07:41.827 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-10-17T18:07:41.852 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.9 MB/s | 251 kB 00:00 2023-10-17T18:07:41.862 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-17T18:07:41.878 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2023-10-17T18:07:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-10-17T18:07:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-10-17T18:07:41.933 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:07:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.5 MB/s | 30 MB 00:03 2023-10-17T18:07:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:07:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:07:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:07:42.096 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-17T18:07:42.297 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-17T18:07:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:07:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:07:42.520 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-17T18:07:42.676 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-17T18:07:42.907 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-17T18:07:43.092 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-17T18:07:43.344 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-17T18:07:43.496 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:07:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-17T18:07:43.703 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-17T18:07:43.774 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-17T18:07:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-17T18:07:44.038 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-17T18:07:44.148 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-17T18:07:44.245 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-17T18:07:44.374 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-17T18:07:44.531 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-17T18:07:44.661 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-17T18:07:44.730 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-17T18:07:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-17T18:07:44.958 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-17T18:07:45.116 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-17T18:07:45.171 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-17T18:07:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-17T18:07:45.387 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-17T18:07:45.522 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-17T18:07:45.635 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-17T18:07:45.744 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-17T18:07:45.955 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-17T18:07:46.086 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-17T18:07:46.160 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-17T18:07:46.390 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-17T18:07:46.469 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-17T18:07:46.568 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-17T18:07:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-17T18:07:46.839 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-17T18:07:46.917 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-17T18:07:47.086 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-17T18:07:47.146 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-17T18:07:47.342 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-17T18:07:47.352 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-17T18:07:47.687 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-17T18:07:47.703 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-17T18:07:47.908 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-17T18:07:48.036 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-17T18:07:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-17T18:07:48.290 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-17T18:07:48.380 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.n 27/32 2023-10-17T18:07:48.543 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-17T18:07:48.585 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 28/32 2023-10-17T18:07:48.784 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-17T18:07:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-17T18:07:49.350 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-17T18:07:49.655 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-17T18:07:50.147 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.n 27/32 2023-10-17T18:07:50.164 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 28/32 2023-10-17T18:07:50.343 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 29/32 2023-10-17T18:07:50.350 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 28/32 2023-10-17T18:07:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 29/32 2023-10-17T18:07:51.673 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 30/32 2023-10-17T18:07:51.890 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 30/32 2023-10-17T18:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 28/32 2023-10-17T18:07:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 31/32 2023-10-17T18:07:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 31/32 2023-10-17T18:07:52.217 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 29/32 2023-10-17T18:07:52.228 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:18.2.0-708.gcff47d96.el8.x86_64 32/32 2023-10-17T18:07:52.618 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:18.2.0-708.gcff47d96.el8.x86_64 32/32 2023-10-17T18:07:52.618 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:18.2.0-708.gcff47d96.el8.x86_64 1/32 2023-10-17T18:07:52.619 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 2/32 2023-10-17T18:07:52.619 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 3/32 2023-10-17T18:07:52.620 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 4/32 2023-10-17T18:07:52.620 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 5/32 2023-10-17T18:07:52.620 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.n 6/32 2023-10-17T18:07:52.621 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-17T18:07:52.621 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-17T18:07:52.622 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-17T18:07:52.622 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-17T18:07:52.622 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-17T18:07:52.623 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-17T18:07:52.623 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-17T18:07:52.624 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-17T18:07:52.624 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-17T18:07:52.624 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-17T18:07:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-17T18:07:52.625 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-17T18:07:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-17T18:07:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-17T18:07:52.626 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-17T18:07:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-17T18:07:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-17T18:07:52.627 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-17T18:07:52.628 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-17T18:07:52.628 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-17T18:07:52.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-17T18:07:52.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-17T18:07:52.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-17T18:07:52.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-17T18:07:52.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-17T18:07:52.706 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 29/32 2023-10-17T18:07:53.689 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-17T18:07:53.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:53.689 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:07:53.689 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:53.690 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:53.690 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:53.690 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:07:53.690 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:53.690 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:53.691 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-17T18:07:53.691 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-17T18:07:53.691 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-17T18:07:53.691 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-17T18:07:53.691 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-17T18:07:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-17T18:07:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-17T18:07:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2023-10-17T18:07:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-17T18:07:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-17T18:07:53.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-17T18:07:53.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-17T18:07:53.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-17T18:07:53.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-17T18:07:53.694 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-17T18:07:53.695 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:53.696 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:07:53.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 30/32 2023-10-17T18:07:53.967 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 30/32 2023-10-17T18:07:53.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2023-10-17T18:07:54.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 31/32 2023-10-17T18:07:54.127 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 31/32 2023-10-17T18:07:54.250 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:18.2.0-708.gcff47d96.el8.x86_64 32/32 2023-10-17T18:07:54.676 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:18.2.0-708.gcff47d96.el8.x86_64 32/32 2023-10-17T18:07:54.677 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:18.2.0-708.gcff47d96.el8.x86_64 1/32 2023-10-17T18:07:54.677 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 2/32 2023-10-17T18:07:54.677 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 3/32 2023-10-17T18:07:54.677 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 4/32 2023-10-17T18:07:54.677 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 5/32 2023-10-17T18:07:54.678 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.n 6/32 2023-10-17T18:07:54.678 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-17T18:07:54.678 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-17T18:07:54.678 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-17T18:07:54.678 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-17T18:07:54.679 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-17T18:07:54.679 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-17T18:07:54.679 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-17T18:07:54.679 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-17T18:07:54.679 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-17T18:07:54.680 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-17T18:07:54.680 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-17T18:07:54.680 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-17T18:07:54.680 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-17T18:07:54.680 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-17T18:07:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-17T18:07:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-17T18:07:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-17T18:07:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-17T18:07:54.681 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-17T18:07:54.682 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-17T18:07:54.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-17T18:07:54.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-17T18:07:54.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-17T18:07:54.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-17T18:07:54.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-17T18:07:54.829 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:27 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:07:55.840 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-17T18:07:55.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:55.841 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:07:55.841 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:55.841 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:55.841 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:55.842 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:07:55.842 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:55.842 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:07:55.842 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-17T18:07:55.843 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-17T18:07:55.843 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-17T18:07:55.843 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-17T18:07:55.843 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-17T18:07:55.843 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-17T18:07:55.844 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-17T18:07:55.844 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-17T18:07:55.844 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-17T18:07:55.844 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-2.6-1.el8.noarch 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-17T18:07:55.845 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-17T18:07:55.846 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-17T18:07:55.846 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-17T18:07:55.846 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-17T18:07:55.846 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-17T18:07:55.846 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-17T18:07:55.847 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-17T18:07:55.847 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-17T18:07:55.847 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-17T18:07:55.847 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-17T18:07:55.847 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-17T18:07:55.848 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:07:55.848 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:07:56.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2023-10-17T18:07:56.243 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:07:56.436 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:07:56.437 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:07:56.437 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:07:56.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2023-10-17T18:07:56.941 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:27 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:07:57.401 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:07:58.454 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:07:58.663 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:07:58.664 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:07:58.664 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:07:58.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2023-10-17T18:07:58.998 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:07:58.999 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:58.999 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 167 k 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:07:59.000 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:07:59.001 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:07:59.001 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:07:59.001 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 167 k 2023-10-17T18:07:59.001 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 674 k 2023-10-17T18:07:59.001 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:07:59.342 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-18.2.0-708.gcff47d96.el8.noarch.rpm 488 kB/s | 167 kB 00:00 2023-10-17T18:07:59.343 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:07:59.343 INFO:teuthology.orchestra.run.smithi071.stdout:Total 487 kB/s | 167 kB 00:00 2023-10-17T18:07:59.344 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:07:59.351 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:07:59.351 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:07:59.353 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:07:59.356 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:07:59.398 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:07:59.675 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:07:59.891 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:00.045 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:00.269 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:00.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:00.877 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:00.877 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:00.877 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:00.877 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:00.877 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:01.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2023-10-17T18:08:01.399 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:01.400 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:01.400 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:01.400 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:01.400 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:01.400 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 167 k 2023-10-17T18:08:01.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:01.401 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:01.401 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:01.401 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:08:01.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:01.402 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 167 k 2023-10-17T18:08:01.402 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 674 k 2023-10-17T18:08:01.402 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:01.584 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-18.2.0-708.gcff47d96.el8.noarch.rpm 908 kB/s | 167 kB 00:00 2023-10-17T18:08:01.585 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:01.585 INFO:teuthology.orchestra.run.smithi049.stdout:Total 902 kB/s | 167 kB 00:00 2023-10-17T18:08:01.586 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:01.594 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:01.594 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:01.596 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:01.599 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:01.641 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:01.966 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:02.320 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:02.552 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:03.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:03.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:03.230 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:03.230 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:03.231 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:03.231 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:03.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2023-10-17T18:08:03.601 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:03.602 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:03.603 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-10-17T18:08:03.603 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:03.603 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:03.603 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-708.gcff47d96.el8 ceph 165 k 2023-10-17T18:08:03.603 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:03.604 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:03.604 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:03.604 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:08:03.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:03.604 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 165 k 2023-10-17T18:08:03.605 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 503 k 2023-10-17T18:08:03.605 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:03.921 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-18.2.0-708.gcff47d9 520 kB/s | 165 kB 00:00 2023-10-17T18:08:03.922 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:03.923 INFO:teuthology.orchestra.run.smithi071.stdout:Total 518 kB/s | 165 kB 00:00 2023-10-17T18:08:03.923 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:03.934 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:03.934 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:03.981 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:03.984 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:04.094 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:04.259 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:04.323 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:04.680 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:05.339 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:05.339 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:08:05.340 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:05.340 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:05.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2023-10-17T18:08:06.040 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:06.041 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:06.042 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T18:08:06.042 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:06.042 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:06.042 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-708.gcff47d96.el8 ceph 165 k 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:06.043 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 165 k 2023-10-17T18:08:06.044 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 503 k 2023-10-17T18:08:06.044 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-18.2.0-708.gcff47d9 915 kB/s | 165 kB 00:00 2023-10-17T18:08:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:Total 909 kB/s | 165 kB 00:00 2023-10-17T18:08:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:06.237 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:06.237 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:06.288 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:06.291 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:06.441 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:06.442 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:06.606 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:07.049 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:07.732 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.e 1/1 2023-10-17T18:08:07.733 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:07.733 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:07.733 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:08:07.733 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:07.733 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:07.864 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:08:07.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2023-10-17T18:08:08.060 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:08.061 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:08:08.061 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:08.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2023-10-17T18:08:08.866 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:09.020 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:10.392 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:08:10.601 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:10.602 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:10.603 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:08:10.603 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:10.603 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:10.604 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:10.604 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:10.604 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:10.604 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 1.8 M 2023-10-17T18:08:10.604 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:08:10.605 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 23 k 2023-10-17T18:08:10.605 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 15 k 2023-10-17T18:08:10.605 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-17T18:08:10.605 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-17T18:08:10.605 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-17T18:08:10.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:10.606 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:10.606 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:10.606 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-10-17T18:08:10.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:10.607 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.1 M 2023-10-17T18:08:10.607 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 8.5 M 2023-10-17T18:08:10.608 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:10.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2023-10-17T18:08:11.115 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-grafana-dashboards-18.2.0-708.gcff4 155 kB/s | 23 kB 00:00 2023-10-17T18:08:11.265 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-prometheus-alerts-18.2.0-708.gcff47 49 kB/s | 15 kB 00:00 2023-10-17T18:08:11.433 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2023-10-17T18:08:11.458 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 124 kB/s | 43 kB 00:00 2023-10-17T18:08:11.583 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): ceph-mgr-dashboard-18.2.0-708.gcff47d96. 2.9 MB/s | 1.8 MB 00:00 2023-10-17T18:08:11.608 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:11.609 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-10-17T18:08:11.610 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:11.610 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.1 MB/s | 2.1 MB 00:01 2023-10-17T18:08:11.620 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:11.629 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:11.629 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:11.704 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:11.707 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:11.992 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:12.147 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-17T18:08:12.391 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-17T18:08:12.618 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-17T18:08:12.767 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.no 4/6 2023-10-17T18:08:13.084 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.n 5/6 2023-10-17T18:08:13.254 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 6/6 2023-10-17T18:08:13.319 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:13.322 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:13.322 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:13.322 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 1.8 M 2023-10-17T18:08:13.323 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:08:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 23 k 2023-10-17T18:08:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 15 k 2023-10-17T18:08:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-17T18:08:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-17T18:08:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-17T18:08:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:13.324 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:13.324 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:13.325 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-10-17T18:08:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:13.325 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.1 M 2023-10-17T18:08:13.325 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 8.5 M 2023-10-17T18:08:13.325 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:13.539 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 6/6 2023-10-17T18:08:13.539 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.n 1/6 2023-10-17T18:08:13.539 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 2/6 2023-10-17T18:08:13.539 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.no 3/6 2023-10-17T18:08:13.540 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-17T18:08:13.540 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-17T18:08:13.675 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): ceph-prometheus-alerts-18.2.0-708.gcff47 106 kB/s | 15 kB 00:00 2023-10-17T18:08:13.759 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): ceph-grafana-dashboards-18.2.0-708.gcff4 105 kB/s | 23 kB 00:00 2023-10-17T18:08:14.009 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): ceph-mgr-dashboard-18.2.0-708.gcff47d96. 3.7 MB/s | 1.8 MB 00:00 2023-10-17T18:08:14.035 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 119 kB/s | 43 kB 00:00 2023-10-17T18:08:14.127 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 92 kB/s | 34 kB 00:00 2023-10-17T18:08:14.224 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-17T18:08:14.224 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:14.224 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:14.224 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:14.225 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:14.225 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:14.225 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-17T18:08:14.225 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-17T18:08:14.225 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-17T18:08:14.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:14.226 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:14.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-17T18:08:14.552 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 362 kB/s | 196 kB 00:00 2023-10-17T18:08:14.555 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:14.556 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.7 MB/s | 2.1 MB 00:01 2023-10-17T18:08:14.568 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:14.579 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:14.579 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:14.698 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:14.701 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:14.986 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:15.154 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-17T18:08:15.206 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:15.357 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-17T18:08:15.549 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-17T18:08:15.695 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.no 4/6 2023-10-17T18:08:16.191 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.n 5/6 2023-10-17T18:08:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 6/6 2023-10-17T18:08:16.718 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 6/6 2023-10-17T18:08:16.718 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.n 1/6 2023-10-17T18:08:16.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 2/6 2023-10-17T18:08:16.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.no 3/6 2023-10-17T18:08:16.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-17T18:08:16.719 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-17T18:08:16.813 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:16.815 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-10-17T18:08:16.815 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:16.815 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-10-17T18:08:16.815 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:16.815 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 7.4 M 2023-10-17T18:08:16.816 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:08:16.816 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-17T18:08:16.816 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-17T18:08:16.816 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-17T18:08:16.816 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-17T18:08:16.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-17T18:08:16.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-17T18:08:16.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-17T18:08:16.817 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:16.817 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:16.818 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================================ 2023-10-17T18:08:16.818 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2023-10-17T18:08:16.818 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:16.819 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 35 M 2023-10-17T18:08:16.819 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 210 M 2023-10-17T18:08:16.819 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:17.395 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-17T18:08:17.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:17.395 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:17.395 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:17.396 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:17.396 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:17.396 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-17T18:08:17.396 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-17T18:08:17.396 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-17T18:08:17.397 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:17.397 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:17.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-17T18:08:17.755 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2023-10-17T18:08:17.822 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2023-10-17T18:08:17.889 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.3 MB/s | 225 kB 00:00 2023-10-17T18:08:18.039 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): ceph-mgr-diskprediction-local-18.2.0-708 6.7 MB/s | 7.4 MB 00:01 2023-10-17T18:08:18.190 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 8.6 MB/s | 3.7 MB 00:00 2023-10-17T18:08:18.414 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:18.682 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 351 kB/s | 172 kB 00:00 2023-10-17T18:08:18.824 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 15 MB/s | 14 MB 00:00 2023-10-17T18:08:18.857 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 789 kB/s | 645 kB 00:00 2023-10-17T18:08:18.858 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:18.858 INFO:teuthology.orchestra.run.smithi071.stdout:Total 17 MB/s | 35 MB 00:02 2023-10-17T18:08:19.091 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:19.110 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:19.110 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:19.490 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:19.493 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:20.140 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:20.141 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:20.142 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-10-17T18:08:20.142 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:20.142 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-10-17T18:08:20.143 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:20.143 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 7.4 M 2023-10-17T18:08:20.143 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:08:20.143 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-17T18:08:20.143 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-17T18:08:20.144 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-17T18:08:20.144 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-17T18:08:20.144 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-17T18:08:20.144 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-17T18:08:20.144 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-17T18:08:20.145 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:20.145 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:20.145 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-10-17T18:08:20.145 INFO:teuthology.orchestra.run.smithi049.stdout:Install 8 Packages 2023-10-17T18:08:20.145 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:20.146 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 35 M 2023-10-17T18:08:20.146 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 210 M 2023-10-17T18:08:20.146 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:20.297 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-17T18:08:20.415 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-17T18:08:20.589 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-17T18:08:21.099 INFO:teuthology.orchestra.run.smithi049.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.9 MB 00:00 2023-10-17T18:08:21.175 INFO:teuthology.orchestra.run.smithi049.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.9 MB/s | 4.7 MB 00:00 2023-10-17T18:08:21.175 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-17T18:08:21.250 INFO:teuthology.orchestra.run.smithi049.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-10-17T18:08:21.388 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:21.977 INFO:teuthology.orchestra.run.smithi049.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.3 MB/s | 3.7 MB 00:00 2023-10-17T18:08:21.985 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:22.172 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-17T18:08:22.219 INFO:teuthology.orchestra.run.smithi049.stdout:(5/8): ceph-mgr-diskprediction-local-18.2.0-708 4.0 MB/s | 7.4 MB 00:01 2023-10-17T18:08:22.603 INFO:teuthology.orchestra.run.smithi049.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 10 MB/s | 14 MB 00:01 2023-10-17T18:08:22.695 INFO:teuthology.orchestra.run.smithi049.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 363 kB/s | 172 kB 00:00 2023-10-17T18:08:22.753 INFO:teuthology.orchestra.run.smithi049.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 832 kB/s | 645 kB 00:00 2023-10-17T18:08:22.754 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:22.754 INFO:teuthology.orchestra.run.smithi049.stdout:Total 14 MB/s | 35 MB 00:02 2023-10-17T18:08:22.842 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-17T18:08:22.998 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:23.017 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:23.017 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:23.183 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-17T18:08:23.410 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:23.413 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:23.463 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-17T18:08:24.003 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-17T18:08:24.308 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-17T18:08:24.436 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-17T18:08:25.056 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-17T18:08:25.219 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:25.240 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-17T18:08:25.868 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-17T18:08:26.513 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-17T18:08:26.698 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-17T18:08:26.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 8/8 2023-10-17T18:08:27.029 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 8/8 2023-10-17T18:08:27.029 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 1/8 2023-10-17T18:08:27.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-17T18:08:27.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:27.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-17T18:08:27.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-17T18:08:27.030 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-17T18:08:27.031 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-17T18:08:27.052 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-17T18:08:27.294 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-17T18:08:27.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-17T18:08:27.682 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:27.682 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:27.682 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:27.683 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-17T18:08:27.683 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-17T18:08:27.683 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-17T18:08:27.683 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-17T18:08:27.683 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-17T18:08:27.684 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-17T18:08:27.684 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-17T18:08:27.684 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:27.684 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:27.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2023-10-17T18:08:28.665 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:01 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:29.170 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-17T18:08:30.267 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:30.269 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:30.269 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:30.269 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:30.270 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:30.270 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 47 k 2023-10-17T18:08:30.270 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:08:30.270 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-17T18:08:30.270 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-17T18:08:30.271 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-17T18:08:30.272 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-17T18:08:30.272 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-17T18:08:30.272 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-17T18:08:30.272 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-17T18:08:30.272 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:30.273 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:30.273 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:30.273 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2023-10-17T18:08:30.273 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:30.274 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2023-10-17T18:08:30.274 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2023-10-17T18:08:30.274 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:30.433 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-17T18:08:30.643 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 8/8 2023-10-17T18:08:30.664 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 187 kB/s | 27 kB 00:00 2023-10-17T18:08:30.681 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 126 kB/s | 20 kB 00:00 2023-10-17T18:08:30.707 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): ceph-mgr-rook-18.2.0-708.gcff47d96.el8. 257 kB/s | 47 kB 00:00 2023-10-17T18:08:30.807 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 879 kB/s | 110 kB 00:00 2023-10-17T18:08:30.832 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 840 kB/s | 126 kB 00:00 2023-10-17T18:08:30.916 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 395 kB/s | 33 kB 00:00 2023-10-17T18:08:30.949 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 487 kB/s | 16 kB 00:00 2023-10-17T18:08:30.957 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 8/8 2023-10-17T18:08:30.957 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96 1/8 2023-10-17T18:08:30.958 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-17T18:08:30.958 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-17T18:08:30.958 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-17T18:08:30.958 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-17T18:08:30.958 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-17T18:08:30.959 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-17T18:08:31.016 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2023-10-17T18:08:31.146 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 9.0 MB/s | 1.2 MB 00:00 2023-10-17T18:08:31.169 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 118 kB/s | 43 kB 00:00 2023-10-17T18:08:31.192 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.4 MB/s | 64 kB 00:00 2023-10-17T18:08:31.226 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 299 kB/s | 155 kB 00:00 2023-10-17T18:08:31.276 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 576 kB/s | 61 kB 00:00 2023-10-17T18:08:31.277 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:31.277 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2023-10-17T18:08:31.329 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:31.344 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:31.344 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:31.428 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:31.431 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:31.648 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-17T18:08:31.648 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:31.648 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:31.648 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-17T18:08:31.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-17T18:08:31.650 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-17T18:08:31.650 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:31.650 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:31.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2023-10-17T18:08:31.946 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:32.188 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-17T18:08:32.402 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-17T18:08:32.596 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-17T18:08:32.673 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:03 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:32.843 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-17T18:08:33.033 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-17T18:08:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-17T18:08:33.626 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-17T18:08:33.857 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-17T18:08:34.390 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:34.392 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:34.392 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:34.393 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:34.393 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:34.393 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 47 k 2023-10-17T18:08:34.393 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:08:34.393 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-17T18:08:34.394 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-17T18:08:34.394 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-17T18:08:34.394 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-17T18:08:34.394 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-17T18:08:34.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-17T18:08:34.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-17T18:08:34.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-17T18:08:34.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-17T18:08:34.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-17T18:08:34.396 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-17T18:08:34.396 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-17T18:08:34.396 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:34.396 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:34.396 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:34.397 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2023-10-17T18:08:34.397 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:34.397 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 1.9 M 2023-10-17T18:08:34.398 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 25 M 2023-10-17T18:08:34.398 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:34.897 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-17T18:08:35.117 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 123 kB/s | 20 kB 00:00 2023-10-17T18:08:35.143 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 142 kB/s | 27 kB 00:00 2023-10-17T18:08:35.200 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-17T18:08:35.293 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 721 kB/s | 126 kB 00:00 2023-10-17T18:08:35.318 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): ceph-mgr-rook-18.2.0-708.gcff47d96.el8. 130 kB/s | 47 kB 00:00 2023-10-17T18:08:35.344 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 548 kB/s | 110 kB 00:00 2023-10-17T18:08:35.417 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-17T18:08:35.544 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 165 kB/s | 33 kB 00:00 2023-10-17T18:08:35.644 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2023-10-17T18:08:35.645 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-17T18:08:35.670 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-10-17T18:08:35.770 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 325 kB/s | 155 kB 00:00 2023-10-17T18:08:35.835 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 13/13 2023-10-17T18:08:35.879 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 348 kB/s | 81 kB 00:00 2023-10-17T18:08:35.962 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 735 kB/s | 61 kB 00:00 2023-10-17T18:08:36.012 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 265 kB/s | 64 kB 00:00 2023-10-17T18:08:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2023-10-17T18:08:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:36.106 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-10-17T18:08:36.148 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 13/13 2023-10-17T18:08:36.148 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 1/13 2023-10-17T18:08:36.148 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-17T18:08:36.148 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-17T18:08:36.149 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-17T18:08:36.149 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-17T18:08:36.149 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-17T18:08:36.149 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-17T18:08:36.149 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-17T18:08:36.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-17T18:08:36.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-17T18:08:36.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-17T18:08:36.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-17T18:08:36.168 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:36.770 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-17T18:08:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:36.861 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-17T18:08:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-17T18:08:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-17T18:08:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-17T18:08:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-17T18:08:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-17T18:08:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-17T18:08:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-17T18:08:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-17T18:08:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-17T18:08:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-17T18:08:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-17T18:08:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:36.864 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:36.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2023-10-17T18:08:37.039 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-17T18:08:37.244 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-17T18:08:37.446 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-17T18:08:37.690 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-17T18:08:37.847 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:10 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:37.874 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-17T18:08:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-17T18:08:38.419 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-17T18:08:38.631 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-17T18:08:39.446 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:39.447 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:39.448 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:39.448 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:39.448 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:39.448 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 132 k 2023-10-17T18:08:39.449 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:08:39.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-17T18:08:39.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-17T18:08:39.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-17T18:08:39.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-17T18:08:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:39.450 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:39.450 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:39.450 INFO:teuthology.orchestra.run.smithi071.stdout:Install 5 Packages 2023-10-17T18:08:39.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:39.451 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.0 M 2023-10-17T18:08:39.451 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2023-10-17T18:08:39.451 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:39.707 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-17T18:08:39.940 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-17T18:08:40.116 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-17T18:08:40.235 INFO:teuthology.orchestra.run.smithi071.stdout:(1/5): ceph-mgr-cephadm-18.2.0-708.gcff47d96.el 323 kB/s | 132 kB 00:00 2023-10-17T18:08:40.285 INFO:teuthology.orchestra.run.smithi071.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.1 MB/s | 538 kB 00:00 2023-10-17T18:08:40.302 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-17T18:08:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 13/13 2023-10-17T18:08:40.511 INFO:teuthology.orchestra.run.smithi071.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 274 kB/s | 60 kB 00:00 2023-10-17T18:08:40.636 INFO:teuthology.orchestra.run.smithi071.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-10-17T18:08:40.813 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 13/13 2023-10-17T18:08:40.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 1/13 2023-10-17T18:08:40.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-17T18:08:40.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-17T18:08:40.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-17T18:08:40.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-17T18:08:40.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-17T18:08:40.816 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-17T18:08:41.286 INFO:teuthology.orchestra.run.smithi071.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.3 MB/s | 4.8 MB 00:01 2023-10-17T18:08:41.288 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:41.289 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.2 MB/s | 6.0 MB 00:01 2023-10-17T18:08:41.342 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:41.354 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:41.354 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:41.465 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:41.468 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:41.584 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-17T18:08:41.585 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:41.585 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:41.585 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:41.585 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-17T18:08:41.586 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-17T18:08:41.586 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-17T18:08:41.586 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-17T18:08:41.586 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-17T18:08:41.586 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-17T18:08:41.587 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-17T18:08:41.587 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-17T18:08:41.587 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-17T18:08:41.587 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-17T18:08:41.587 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-17T18:08:41.588 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-17T18:08:41.588 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:41.588 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:41.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2023-10-17T18:08:41.808 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:42.115 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-17T18:08:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:13 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:43.086 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-17T18:08:43.429 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-17T18:08:43.674 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-17T18:08:43.900 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 5/5 2023-10-17T18:08:44.187 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 5/5 2023-10-17T18:08:44.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/5 2023-10-17T18:08:44.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-17T18:08:44.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-17T18:08:44.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-17T18:08:44.365 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:44.366 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:44.367 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:44.367 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:44.367 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:44.367 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 132 k 2023-10-17T18:08:44.368 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:08:44.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-17T18:08:44.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-17T18:08:44.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-17T18:08:44.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-17T18:08:44.369 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:44.369 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:44.369 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:44.369 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2023-10-17T18:08:44.369 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:44.370 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.0 M 2023-10-17T18:08:44.370 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2023-10-17T18:08:44.370 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:44.819 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-17T18:08:44.820 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:44.820 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:44.820 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:44.820 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-17T18:08:44.821 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-17T18:08:44.821 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-17T18:08:44.821 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-17T18:08:44.821 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:44.821 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:44.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2023-10-17T18:08:45.129 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): ceph-mgr-cephadm-18.2.0-708.gcff47d96.el 354 kB/s | 132 kB 00:00 2023-10-17T18:08:45.279 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2023-10-17T18:08:45.605 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 186 kB/s | 60 kB 00:00 2023-10-17T18:08:45.646 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 920 kB/s | 476 kB 00:00 2023-10-17T18:08:45.777 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:18 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:46.488 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 2.8 MB/s | 4.8 MB 00:01 2023-10-17T18:08:46.490 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:46.491 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.8 MB/s | 6.0 MB 00:02 2023-10-17T18:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:47.348 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-17T18:08:47.380 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:47.380 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:47.381 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:47.381 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:47.381 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:47.381 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 847 k 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:47.382 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2023-10-17T18:08:47.383 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:47.383 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 930 k 2023-10-17T18:08:47.384 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.0 M 2023-10-17T18:08:47.384 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:47.910 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): ceph-fuse-18.2.0-708.gcff47d96.el8.x86_6 2.7 MB/s | 847 kB 00:00 2023-10-17T18:08:48.018 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 202 kB/s | 83 kB 00:00 2023-10-17T18:08:48.019 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:48.019 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 930 kB 00:00 2023-10-17T18:08:48.029 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:48.051 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:48.051 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:48.135 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:48.138 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:48.308 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-17T18:08:48.597 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-17T18:08:48.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2/2 2023-10-17T18:08:48.846 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-17T18:08:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-17T18:08:49.217 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2/2 2023-10-17T18:08:49.217 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/2 2023-10-17T18:08:49.290 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 5/5 2023-10-17T18:08:49.588 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 5/5 2023-10-17T18:08:49.588 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 1/5 2023-10-17T18:08:49.588 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-17T18:08:49.589 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-17T18:08:49.589 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-17T18:08:49.898 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-17T18:08:49.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:49.898 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:49.898 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-17T18:08:49.899 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:49.899 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:50.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2023-10-17T18:08:50.262 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-17T18:08:50.262 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:50.262 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-17T18:08:50.263 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:50.264 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:50.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2023-10-17T18:08:51.003 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:24 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:51.261 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:22 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:52.598 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:52.599 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:52.599 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-10-17T18:08:52.600 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:52.600 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:52.600 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 261 k 2023-10-17T18:08:52.600 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:52.601 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:52.601 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:52.601 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:08:52.601 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:52.601 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 261 k 2023-10-17T18:08:52.602 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2023-10-17T18:08:52.602 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:52.830 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-18.2.0-708.gcff47d96.el8.noarch.rpm 1.1 MB/s | 261 kB 00:00 2023-10-17T18:08:52.831 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:52.831 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.1 MB/s | 261 kB 00:00 2023-10-17T18:08:52.831 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:52.841 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:52.841 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:52.872 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:52.875 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:52.979 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:52.980 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:52.980 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:52.980 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:52.980 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 847 k 2023-10-17T18:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-17T18:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-17T18:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:52.982 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:52.982 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2023-10-17T18:08:52.982 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 930 k 2023-10-17T18:08:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.0 M 2023-10-17T18:08:52.983 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:53.050 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:53.298 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:53.321 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-fuse-18.2.0-708.gcff47d96.el8.x86_6 3.0 MB/s | 847 kB 00:00 2023-10-17T18:08:53.462 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 201 kB/s | 83 kB 00:00 2023-10-17T18:08:53.463 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:53.463 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.9 MB/s | 930 kB 00:00 2023-10-17T18:08:53.474 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:53.497 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:53.497 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:53.588 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:53.592 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:53.708 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:53.775 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:54.026 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-17T18:08:54.198 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2/2 2023-10-17T18:08:54.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:08:54.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:54.395 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:54.395 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:08:54.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:54.395 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:54.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2023-10-17T18:08:54.585 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2/2 2023-10-17T18:08:54.585 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/2 2023-10-17T18:08:55.227 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-17T18:08:55.227 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:55.227 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:08:55.227 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-17T18:08:55.228 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:55.228 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:08:55.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2023-10-17T18:08:55.427 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:28 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:08:56.301 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:27 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:08:57.033 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:08:57.034 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:57.034 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-10-17T18:08:57.034 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:57.034 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:08:57.034 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:18.2.0-708.gcff47d96.el8 ceph 139 k 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:57.035 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 139 k 2023-10-17T18:08:57.036 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 513 k 2023-10-17T18:08:57.036 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:08:58.014 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:08:58.015 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:58.015 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-17T18:08:58.016 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:58.016 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:08:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:18.2.0-708.gcff47d96.el8 ceph-noarch 261 k 2023-10-17T18:08:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:58.016 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:08:58.017 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:08:58.017 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:08:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:08:58.017 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 261 k 2023-10-17T18:08:58.018 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.3 M 2023-10-17T18:08:58.018 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:08:58.043 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-18.2.0-708.gcff47d96.el8.x86_64. 138 kB/s | 139 kB 00:01 2023-10-17T18:08:58.043 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:58.044 INFO:teuthology.orchestra.run.smithi071.stdout:Total 138 kB/s | 139 kB 00:01 2023-10-17T18:08:58.044 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:08:58.050 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:08:58.051 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:08:58.106 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:08:58.109 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:08:58.257 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:08:58.513 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:08:58.549 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-volume-18.2.0-708.gcff47d96.el8.noarch.rpm 490 kB/s | 261 kB 00:00 2023-10-17T18:08:58.550 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:08:58.550 INFO:teuthology.orchestra.run.smithi049.stdout:Total 489 kB/s | 261 kB 00:00 2023-10-17T18:08:58.550 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:08:58.561 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:08:58.561 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:08:58.594 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:08:58.597 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:08:58.788 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:08:58.802 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:08:59.422 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:08:59.423 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:59.423 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:08:59.423 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:08:59.423 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:08:59.424 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:08:59.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2023-10-17T18:08:59.919 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:09:00.527 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:33 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:00.825 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:09:01.828 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 1/1 2023-10-17T18:09:01.828 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:01.828 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:01.828 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:18.2.0-708.gcff47d96.el8.noarch 2023-10-17T18:09:01.829 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:01.829 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:01.938 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:02.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2023-10-17T18:09:02.132 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:02.133 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:02.133 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:02.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2023-10-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:33 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:03.074 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:36 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:04.649 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:04.649 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:04.650 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T18:09:04.650 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:04.650 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:09:04.650 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:18.2.0-708.gcff47d96.el8 ceph 139 k 2023-10-17T18:09:04.651 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:04.651 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:09:04.651 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:04.651 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:09:04.651 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:04.652 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 139 k 2023-10-17T18:09:04.652 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 513 k 2023-10-17T18:09:04.652 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:09:04.667 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:04.668 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:04.668 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-10-17T18:09:04.668 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:04.669 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:09:04.669 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:18.2.0-708.gcff47d96.el8 ceph 30 k 2023-10-17T18:09:04.669 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:04.669 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:09:04.669 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:04.670 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:09:04.670 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:04.670 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 k 2023-10-17T18:09:04.671 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 137 k 2023-10-17T18:09:04.671 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:09:04.997 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-18.2.0-708.gcff47d96.el8.x86_64 93 kB/s | 30 kB 00:00 2023-10-17T18:09:04.998 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:04.998 INFO:teuthology.orchestra.run.smithi071.stdout:Total 92 kB/s | 30 kB 00:00 2023-10-17T18:09:04.999 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:09:05.002 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:09:05.002 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:09:05.018 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-18.2.0-708.gcff47d96.el8.x86_64. 379 kB/s | 139 kB 00:00 2023-10-17T18:09:05.018 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:05.018 INFO:teuthology.orchestra.run.smithi049.stdout:Total 377 kB/s | 139 kB 00:00 2023-10-17T18:09:05.019 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:09:05.024 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:09:05.025 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:09:05.025 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:09:05.028 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:09:05.077 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:09:05.080 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:09:05.123 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:09:05.196 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:09:05.311 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:05.454 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:05.570 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:05.756 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:06.166 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:06.166 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:06.166 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:09:06.166 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:06.166 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:06.167 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:06.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2023-10-17T18:09:06.409 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:06.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:06.410 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:06.410 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:06.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:06.410 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:06.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2023-10-17T18:09:07.135 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:40 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:07.525 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:38 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:08.547 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:08.744 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:08.745 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:08.746 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:08.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2023-10-17T18:09:09.045 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:09.253 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:09.254 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:09.255 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:09.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2023-10-17T18:09:09.689 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:42 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:10.245 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:41 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:11.091 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:11.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2023-10-17T18:09:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:11.983 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T18:09:11.984 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:11.984 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:09:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:18.2.0-708.gcff47d96.el8 ceph 30 k 2023-10-17T18:09:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:11.985 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:09:11.985 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:11.985 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:09:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:11.985 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 30 k 2023-10-17T18:09:11.986 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 137 k 2023-10-17T18:09:11.986 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:09:12.257 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:45 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:12.323 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-18.2.0-708.gcff47d96.el8.x86_64 90 kB/s | 30 kB 00:00 2023-10-17T18:09:12.324 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:12.324 INFO:teuthology.orchestra.run.smithi049.stdout:Total 89 kB/s | 30 kB 00:00 2023-10-17T18:09:12.324 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:09:12.328 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:09:12.328 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:09:12.357 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:09:12.360 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:09:12.418 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:09:12.577 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:12.865 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:13.502 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:13.502 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:13.503 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:13.503 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:13.503 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:13.503 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:13.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2023-10-17T18:09:13.640 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:13.835 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:13.837 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:13.837 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:14.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2023-10-17T18:09:14.527 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:45 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:14.956 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:47 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:16.050 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:16.260 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:16.262 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:16.262 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:16.341 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:16.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2023-10-17T18:09:16.534 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:16.536 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:16.536 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:16.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2023-10-17T18:09:17.255 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:48 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:17.488 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:50 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:18.757 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:18.904 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:18.968 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:18.969 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:18.969 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:19.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2023-10-17T18:09:19.096 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:19.098 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:19.098 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:19.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2023-10-17T18:09:19.959 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:50 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:20.044 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:53 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:21.453 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:21.475 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:21.646 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:21.647 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-10-17T18:09:21.647 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:21.691 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:21.692 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:21.692 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:21.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2023-10-17T18:09:21.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2023-10-17T18:09:22.597 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:55 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:22.689 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:53 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:24.181 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:24.182 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:24.182 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-10-17T18:09:24.183 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:24.183 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:09:24.183 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 94 k 2023-10-17T18:09:24.183 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:24.183 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:09:24.184 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:24.184 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:09:24.184 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:24.184 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 94 k 2023-10-17T18:09:24.184 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 280 k 2023-10-17T18:09:24.185 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:09:24.210 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:24.420 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:24.422 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:24.422 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:24.494 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-18.2.0-708.gcff47d96.el8.x86_64.rpm 303 kB/s | 94 kB 00:00 2023-10-17T18:09:24.495 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:24.495 INFO:teuthology.orchestra.run.smithi071.stdout:Total 302 kB/s | 94 kB 00:00 2023-10-17T18:09:24.496 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:09:24.506 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:09:24.506 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:09:24.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2023-10-17T18:09:24.555 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:09:24.558 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:09:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:09:24.874 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:25.146 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:25.421 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:56 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:25.798 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:25.799 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:25.799 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:09:25.799 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:25.799 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:25.799 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:26.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2023-10-17T18:09:26.866 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:59 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:26.952 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:27.161 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:27.162 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:27.162 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:27.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2023-10-17T18:09:28.154 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:59 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:28.458 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:28.459 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:28.459 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.4 M 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:09:28.460 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:28.461 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:09:28.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:28.461 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.4 M 2023-10-17T18:09:28.462 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 16 M 2023-10-17T18:09:28.462 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:09:28.879 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-18.2.0-708.gcff47d96.el8.x86_64.rpm 8.2 MB/s | 3.4 MB 00:00 2023-10-17T18:09:28.880 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:28.881 INFO:teuthology.orchestra.run.smithi071.stdout:Total 8.2 MB/s | 3.4 MB 00:00 2023-10-17T18:09:28.881 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:09:28.893 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:09:28.893 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:09:28.954 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:09:28.957 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:09:29.391 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:09:29.666 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:18.2.0-708.gcff47d96.el8.x86_64 is already installed. 2023-10-17T18:09:29.875 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:29.876 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-17T18:09:29.877 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:29.966 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:29.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2023-10-17T18:09:30.388 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:30.875 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:01 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:31.059 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:31.060 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:31.060 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:09:31.060 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:31.060 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:31.060 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:31.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2023-10-17T18:09:32.160 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:05 ago on Tue 17 Oct 2023 06:06:27 PM UTC. 2023-10-17T18:09:32.589 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:32.590 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-10-17T18:09:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:32.590 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:09:32.591 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:18.2.0-708.gcff47d96.el8 ceph 94 k 2023-10-17T18:09:32.591 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:09:32.591 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:09:32.592 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:32.592 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 94 k 2023-10-17T18:09:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 280 k 2023-10-17T18:09:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:09:32.851 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-18.2.0-708.gcff47d96.el8.x86_64.rpm 362 kB/s | 94 kB 00:00 2023-10-17T18:09:32.852 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:32.852 INFO:teuthology.orchestra.run.smithi049.stdout:Total 361 kB/s | 94 kB 00:00 2023-10-17T18:09:32.853 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:09:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:09:32.864 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:09:32.917 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:09:32.920 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:09:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:09:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:33.769 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-10-17T18:09:33.770 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:33.770 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-10-17T18:09:33.770 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:33.770 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-10-17T18:09:33.771 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 179 k 2023-10-17T18:09:33.771 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:33.771 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-10-17T18:09:33.771 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-10-17T18:09:33.771 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-10-17T18:09:33.772 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:33.772 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 179 k 2023-10-17T18:09:33.772 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 568 k 2023-10-17T18:09:33.772 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-10-17T18:09:33.793 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:33.998 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-18.2.0-708.gcff47d96.el8.x86_64.rpm 787 kB/s | 179 kB 00:00 2023-10-17T18:09:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:Total 783 kB/s | 179 kB 00:00 2023-10-17T18:09:34.000 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-10-17T18:09:34.011 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-10-17T18:09:34.011 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-10-17T18:09:34.060 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-10-17T18:09:34.063 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-10-17T18:09:34.186 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-10-17T18:09:34.397 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:34.453 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:34.453 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:34.454 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:34.454 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:34.454 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:34.454 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:34.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2023-10-17T18:09:34.684 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:35.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:35.343 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:35.343 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-10-17T18:09:35.343 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:35.343 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:35.343 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-10-17T18:09:35.558 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:06 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:35.596 DEBUG:teuthology.parallel:result is None 2023-10-17T18:09:37.323 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:37.324 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:37.324 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:18.2.0-708.gcff47d96.el8 ceph 3.4 M 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:09:37.325 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:37.326 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:09:37.326 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:37.326 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.4 M 2023-10-17T18:09:37.326 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 16 M 2023-10-17T18:09:37.326 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:09:38.505 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-18.2.0-708.gcff47d96.el8.x86_64.rpm 2.9 MB/s | 3.4 MB 00:01 2023-10-17T18:09:38.505 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:38.506 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.9 MB/s | 3.4 MB 00:01 2023-10-17T18:09:38.506 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:09:38.520 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:09:38.521 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:09:38.588 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:09:38.592 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:09:39.053 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:09:39.257 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:40.331 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:40.332 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:40.332 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:40.332 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:40.332 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:40.333 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:40.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2023-10-17T18:09:41.468 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:12 ago on Tue 17 Oct 2023 06:06:29 PM UTC. 2023-10-17T18:09:43.178 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-17T18:09:43.179 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:43.179 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-10-17T18:09:43.180 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:43.180 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-17T18:09:43.180 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:18.2.0-708.gcff47d96.el8 ceph 179 k 2023-10-17T18:09:43.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:43.180 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-17T18:09:43.181 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-10-17T18:09:43.181 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-10-17T18:09:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:43.181 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 179 k 2023-10-17T18:09:43.181 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 568 k 2023-10-17T18:09:43.182 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-17T18:09:43.482 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-18.2.0-708.gcff47d96.el8.x86_64.rpm 594 kB/s | 179 kB 00:00 2023-10-17T18:09:43.483 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-17T18:09:43.483 INFO:teuthology.orchestra.run.smithi049.stdout:Total 591 kB/s | 179 kB 00:00 2023-10-17T18:09:43.483 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-17T18:09:43.495 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-17T18:09:43.496 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-17T18:09:43.550 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-17T18:09:43.553 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-17T18:09:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-17T18:09:43.872 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:44.159 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:44.838 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 1/1 2023-10-17T18:09:44.838 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-17T18:09:44.838 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:18.2.0-708.gcff47d96.el8.x86_64 2023-10-17T18:09:44.838 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:44.839 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-17T18:09:45.022 DEBUG:teuthology.parallel:result is None 2023-10-17T18:09:45.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cff47d96488123dae42a0c6258323ad6422fe0a6 2023-10-17T18:09:45.167 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-17T18:09:45.191 INFO:teuthology.orchestra.run.smithi049.stdout:18.2.0-708.gcff47d96.el8 2023-10-17T18:09:45.192 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-708.gcff47d96.el8 2023-10-17T18:09:45.192 INFO:teuthology.task.install:The correct ceph version 18.2.0-708.gcff47d96 is installed. 2023-10-17T18:09:45.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cff47d96488123dae42a0c6258323ad6422fe0a6 2023-10-17T18:09:45.339 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-17T18:09:45.368 INFO:teuthology.orchestra.run.smithi071.stdout:18.2.0-708.gcff47d96.el8 2023-10-17T18:09:45.368 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-708.gcff47d96.el8 2023-10-17T18:09:45.369 INFO:teuthology.task.install:The correct ceph version 18.2.0-708.gcff47d96 is installed. 2023-10-17T18:09:45.369 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-17T18:09:45.370 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:45.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-17T18:09:45.399 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:45.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-17T18:09:45.441 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-17T18:09:45.442 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:45.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-10-17T18:09:45.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-17T18:09:45.538 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:45.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2023-10-17T18:09:45.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-17T18:09:45.640 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-17T18:09:45.640 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:45.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-17T18:09:45.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-17T18:09:45.736 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:45.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-17T18:09:45.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-17T18:09:45.836 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-17T18:09:45.837 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:45.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-10-17T18:09:45.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-17T18:09:45.933 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:45.934 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2023-10-17T18:09:45.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-17T18:09:46.029 INFO:teuthology.run_tasks:Running task ceph... 2023-10-17T18:09:46.090 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-17T18:09:46.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2023-10-17T18:09:46.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2023-10-17T18:09:46.120 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-17T18:09:46.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-17T18:09:46.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-17T18:09:46.189 INFO:tasks.ceph:Creating extra log directories... 2023-10-17T18:09:46.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-17T18:09:46.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-17T18:09:46.268 INFO:tasks.ceph:Creating ceph cluster c1... 2023-10-17T18:09:46.268 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-17T18:09:46.268 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7430137', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-16-1247-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 993, 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKx21743cUlmo6TKUbvR5PAR3YJyl6RDmMH8t+R6LplmFaEgnLfqs3jO/mdESrHA1/7zybwQDizFBDdcbLTm/18=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxePA3TlvYqb+4IRWm+eI1CkcB6eXCdh5zeTPXhaTJWuWW7972TipdWZLJP4OU5UiJN2/V8uTym1MqHpDcG0e4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_21:55:28', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-17T18:09:46.269 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-17T18:09:46.298 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-17T18:09:46.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T18:09:46.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T18:09:46.384 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:46.384 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2023-10-17T18:09:46.440 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-17T18:09:46.441 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2023-10-17T18:09:46.498 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-17T18:09:46.498 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:09:46.498 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 386912 Links: 1 2023-10-17T18:09:46.499 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:09:46.499 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:09:46.499 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 18:09:39.414351898 +0000 2023-10-17T18:09:46.499 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 18:05:22.160531332 +0000 2023-10-17T18:09:46.499 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 18:05:22.160531332 +0000 2023-10-17T18:09:46.500 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-17T18:09:46.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-17T18:09:46.564 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T18:09:46.564 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T18:09:46.565 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000110058 s, 4.7 MB/s 2023-10-17T18:09:46.566 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-17T18:09:46.627 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2023-10-17T18:09:46.685 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-17T18:09:46.686 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:09:46.686 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 386976 Links: 1 2023-10-17T18:09:46.686 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:09:46.686 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:09:46.686 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 18:09:39.415351871 +0000 2023-10-17T18:09:46.687 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 18:05:22.662517649 +0000 2023-10-17T18:09:46.687 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 18:05:22.662517649 +0000 2023-10-17T18:09:46.687 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-17T18:09:46.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-17T18:09:46.752 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T18:09:46.752 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T18:09:46.752 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.5572e-05 s, 6.0 MB/s 2023-10-17T18:09:46.753 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-17T18:09:46.812 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2023-10-17T18:09:46.870 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-17T18:09:46.870 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:09:46.870 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 391060 Links: 1 2023-10-17T18:09:46.871 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:09:46.871 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:09:46.871 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 18:09:39.415351871 +0000 2023-10-17T18:09:46.871 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 18:05:23.174503693 +0000 2023-10-17T18:09:46.872 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 18:05:23.174503693 +0000 2023-10-17T18:09:46.872 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-17T18:09:46.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-17T18:09:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T18:09:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T18:09:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.0001028 s, 5.0 MB/s 2023-10-17T18:09:46.937 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-17T18:09:46.997 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2023-10-17T18:09:47.055 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-17T18:09:47.055 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:09:47.055 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 401431 Links: 1 2023-10-17T18:09:47.056 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:09:47.056 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:09:47.056 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-17 18:09:39.415351871 +0000 2023-10-17T18:09:47.056 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-17 18:05:23.691489601 +0000 2023-10-17T18:09:47.057 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-17 18:05:23.691489601 +0000 2023-10-17T18:09:47.057 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-17T18:09:47.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-17T18:09:47.121 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-17T18:09:47.121 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-17T18:09:47.121 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.7311e-05 s, 5.9 MB/s 2023-10-17T18:09:47.122 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-17T18:09:47.182 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:09:47.182 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T18:09:47.183 INFO:tasks.ceph:Generating config... 2023-10-17T18:09:47.184 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-17T18:09:47.184 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-17T18:09:47.185 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-17T18:09:47.185 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-17T18:09:47.185 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:09:47.186 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-17T18:09:47.186 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-17T18:09:47.186 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-17T18:09:47.186 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-17T18:09:47.187 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-17T18:09:47.187 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-17T18:09:47.187 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-17T18:09:47.187 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-17T18:09:47.188 INFO:tasks.ceph:[client] setuser = ceph 2023-10-17T18:09:47.188 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-17T18:09:47.188 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-17T18:09:47.188 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-17T18:09:47.189 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-17T18:09:47.189 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-17T18:09:47.189 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-17T18:09:47.190 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-17T18:09:47.190 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-17T18:09:47.190 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-17T18:09:47.190 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-17T18:09:47.191 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-17T18:09:47.191 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-17T18:09:47.191 INFO:tasks.ceph:Setting up c1.mon.a... 2023-10-17T18:09:47.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-10-17T18:09:47.282 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/c1.keyring 2023-10-17T18:09:47.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-10-17T18:09:47.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-17T18:09:47.359 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.49')] 2023-10-17T18:09:47.361 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-17T18:09:47.362 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:47.362 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T18:09:47.418 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --print /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:47.501 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:09:47.501 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:09:47.502 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:47.502 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid ec7c8da4-7f75-454e-8466-c20b932240f2 2023-10-17T18:09:47.502 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = pacific 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:fsid ec7c8da4-7f75-454e-8466-c20b932240f2 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2023-10-17T18:09:47.502376+0000 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:created 2023-10-17T18:09:47.502376+0000 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 16 (pacific) 2023-10-17T18:09:47.503 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2023-10-17T18:09:47.504 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2023-10-17T18:09:47.504 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-10-17T18:09:47.505 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T18:09:47.563 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID ec7c8da4-7f75-454e-8466-c20b932240f2... 2023-10-17T18:09:47.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-17T18:09:47.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-17T18:09:47.649 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-10-17T18:09:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2023-10-17T18:09:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T18:09:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2023-10-17T18:09:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2023-10-17T18:09:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2023-10-17T18:09:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2023-10-17T18:09:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2023-10-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.658 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T18:09:47.658 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.658 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2023-10-17T18:09:47.658 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.658 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.49 2023-10-17T18:09:47.659 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = ec7c8da4-7f75-454e-8466-c20b932240f2 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2023-10-17T18:09:47.660 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.661 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2023-10-17T18:09:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2023-10-17T18:09:47.663 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2023-10-17T18:09:47.664 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout: rocksdb delete range threshold = 0 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:09:47.665 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2023-10-17T18:09:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T18:09:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T18:09:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:09:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:09:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2023-10-17T18:09:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2023-10-17T18:09:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:09:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T18:09:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T18:09:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T18:09:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2023-10-17T18:09:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2023-10-17T18:09:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2023-10-17T18:09:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:47.671 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2023-10-17T18:09:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw = 20 2023-10-17T18:09:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw sync = 20 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt require ssl = False 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt s3 kms backend = testing 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw curl low speed time = 300 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw data log num shards = 4 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw md log max shards = 4 2023-10-17T18:09:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync data inject err probability = 0.1 2023-10-17T18:09:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync log trim interval = 0 2023-10-17T18:09:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync meta inject err probability = 0.1 2023-10-17T18:09:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync obj etag verify = True 2023-10-17T18:09:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: setgroup = ceph 2023-10-17T18:09:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: setuser = ceph 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: rgw data notify interval msec = 0 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 10 2023-10-17T18:09:47.675 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-10-17T18:09:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 1000 2023-10-17T18:09:47.676 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-10-17T18:09:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 10000 2023-10-17T18:09:47.677 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-10-17T18:09:47.677 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-10-17T18:09:47.678 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T18:09:47.678 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-10-17T18:09:47.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.678 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-10-17T18:09:47.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.679 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-10-17T18:09:47.680 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-10-17T18:09:47.680 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.680 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T18:09:47.680 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-10-17T18:09:47.680 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.681 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-10-17T18:09:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-10-17T18:09:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-10-17T18:09:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-10-17T18:09:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-10-17T18:09:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-10-17T18:09:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-10-17T18:09:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T18:09:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T18:09:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-10-17T18:09:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-10-17T18:09:47.684 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.685 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-10-17T18:09:47.685 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-10-17T18:09:47.685 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-10-17T18:09:47.685 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.49 2023-10-17T18:09:47.685 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = ec7c8da4-7f75-454e-8466-c20b932240f2 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-10-17T18:09:47.686 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-10-17T18:09:47.687 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-10-17T18:09:47.687 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.687 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-10-17T18:09:47.687 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-10-17T18:09:47.687 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-10-17T18:09:47.688 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-10-17T18:09:47.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.689 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-10-17T18:09:47.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.689 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-10-17T18:09:47.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.690 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-10-17T18:09:47.691 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-10-17T18:09:47.691 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:09:47.691 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-10-17T18:09:47.691 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2023-10-17T18:09:47.691 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-10-17T18:09:47.692 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:09:47.693 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-10-17T18:09:47.694 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-10-17T18:09:47.694 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-10-17T18:09:47.694 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-10-17T18:09:47.694 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-10-17T18:09:47.694 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T18:09:47.695 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T18:09:47.696 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.697 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-10-17T18:09:47.697 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-10-17T18:09:47.697 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.697 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-10-17T18:09:47.697 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T18:09:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T18:09:47.699 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2023-10-17T18:09:47.699 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2023-10-17T18:09:47.699 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2023-10-17T18:09:47.699 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2023-10-17T18:09:47.699 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2023-10-17T18:09:47.700 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-10-17T18:09:47.701 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2023-10-17T18:09:47.702 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-10-17T18:09:47.702 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2023-10-17T18:09:47.702 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-10-17T18:09:47.702 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2023-10-17T18:09:47.702 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-10-17T18:09:47.703 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2023-10-17T18:09:47.703 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-10-17T18:09:47.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-10-17T18:09:47.777 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-17T18:09:47.777 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:47.778 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-10-17T18:09:47.796 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:47.796 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-10-17T18:09:47.853 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T18:09:47.854 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:47.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/c1.keyring 2023-10-17T18:09:47.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-17T18:09:47.931 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:47.932 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:47.990 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T18:09:47.990 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:47.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c1.keyring 2023-10-17T18:09:47.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-17T18:09:48.035 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:09:48.035 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:48.095 INFO:tasks.ceph:Setting up mon nodes... 2023-10-17T18:09:48.095 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-17T18:09:48.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-10-17T18:09:48.151 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-10-17T18:09:48.153 INFO:tasks.ceph:Setting up mds nodes... 2023-10-17T18:09:48.154 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-17T18:09:48.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-10-17T18:09:48.199 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/c1.client.0.keyring 2023-10-17T18:09:48.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-10-17T18:09:48.256 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/c1.client.1.keyring 2023-10-17T18:09:48.269 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-17T18:09:48.269 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T18:09:48.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-10-17T18:09:48.297 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:09:48.297 INFO:tasks.ceph:role: c1.osd.0 2023-10-17T18:09:48.297 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T18:09:48.298 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-17T18:09:48.364 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:09:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:09:48.365 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:09:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:09:48.366 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:09:48.391 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T18:09:48.393 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T18:09:48.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-10-17T18:09:48.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-10-17T18:09:48.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-10-17T18:09:48.614 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:09:48.614 INFO:tasks.ceph:role: c1.osd.1 2023-10-17T18:09:48.614 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T18:09:48.614 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-17T18:09:48.681 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:09:48.681 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:09:48.681 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:09:48.682 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:09:48.683 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:09:48.713 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T18:09:48.715 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T18:09:48.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-10-17T18:09:48.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-10-17T18:09:48.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-10-17T18:09:48.935 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:09:48.936 INFO:tasks.ceph:role: c1.osd.2 2023-10-17T18:09:48.936 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T18:09:48.936 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-17T18:09:49.003 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:09:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:09:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:09:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:09:49.004 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:09:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:09:49.004 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:09:49.004 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:09:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:09:49.005 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:09:49.030 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-17T18:09:49.031 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-17T18:09:49.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-10-17T18:09:49.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-10-17T18:09:49.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:49.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-10-17T18:09:49.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-10-17T18:09:49.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-17T18:09:49.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-17T18:09:49.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-17T18:09:49.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:49.281+0000 7f869b47c640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-10-17T18:09:51.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-10-17T18:09:51.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:51.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.483+0000 7fedbe208640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-10-17T18:09:51.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.483+0000 7fedbe208640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-10-17T18:09:51.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.483+0000 7fedbe208640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-17T18:09:51.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.483+0000 7fedbe208640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-17T18:09:51.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.483+0000 7fedbe208640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-17T18:09:51.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:51.484+0000 7fedbe208640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-10-17T18:09:53.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-10-17T18:09:53.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.694+0000 7fcd894a2640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-10-17T18:09:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.694+0000 7fcd894a2640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-10-17T18:09:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.694+0000 7fcd894a2640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-17T18:09:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.694+0000 7fcd894a2640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-17T18:09:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.694+0000 7fcd894a2640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-17T18:09:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:09:53.695+0000 7fcd894a2640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-10-17T18:09:55.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-10-17T18:09:55.805 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-17T18:09:55.805 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:55.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-10-17T18:09:55.877 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:55.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-10-17T18:09:55.948 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:55.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-10-17T18:09:56.018 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:56.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-10-17T18:09:56.087 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:56.087 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-10-17T18:09:56.146 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:56.146 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-10-17T18:09:56.204 INFO:tasks.ceph:Adding keys to all mons... 2023-10-17T18:09:56.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/c1.keyring 2023-10-17T18:09:56.272 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2023-10-17T18:09:56.272 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDszS5lFgUMCRAAk6KFLYdL5NjDs2+q2g8aKA== 2023-10-17T18:09:56.272 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-10-17T18:09:56.273 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDtzS5lb7TMEBAAtb0v+/r3pQFr61cXvMpFZg== 2023-10-17T18:09:56.273 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-10-17T18:09:56.273 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDvzS5lul3YHBAAlRKzYWVtk0wvIUP+UD6D1w== 2023-10-17T18:09:56.273 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-10-17T18:09:56.273 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDxzS5lBBd+KRAA0bH2MJUGSCjj8iABSpaV5w== 2023-10-17T18:09:56.274 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-10-17T18:09:56.274 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDszS5laojpCxAACqmpygp0L0ajuTfHtodPbg== 2023-10-17T18:09:56.274 INFO:teuthology.orchestra.run.smithi049.stdout:[client.1] 2023-10-17T18:09:56.274 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDszS5laNFSDxAAknxDvN46v+QaIovekgWGdg== 2023-10-17T18:09:56.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-17T18:09:56.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:09:56.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:09:56.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:09:56.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T18:09:56.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T18:09:56.651 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-17T18:09:56.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-10-17T18:09:56.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-10-17T18:09:56.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-10-17T18:09:56.808 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-10-17T18:09:56.867 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-10-17T18:09:56.867 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-10-17T18:09:56.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-10-17T18:09:56.909 INFO:tasks.ceph.c1.mon.a:Started 2023-10-17T18:09:56.910 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-10-17T18:09:56.910 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-10-17T18:09:56.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-10-17T18:09:56.912 INFO:tasks.ceph.c1.mgr.x:Started 2023-10-17T18:09:56.913 INFO:tasks.ceph:Setting crush tunables to default 2023-10-17T18:09:56.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd crush tunables default 2023-10-17T18:09:57.126 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2023-10-17T18:09:57.137 INFO:tasks.ceph:check_enable_crimson: False 2023-10-17T18:09:57.137 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-10-17T18:09:57.137 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:57.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-10-17T18:09:57.165 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:57.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-10-17T18:09:57.233 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:09:57.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-10-17T18:09:57.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd new df221d56-2316-496c-a2fe-ca92353f0a02 0 2023-10-17T18:09:57.317 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:57.316+0000 7f4c7d64b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-17T18:09:57.504 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:57.503+0000 7f4c7d64b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-17T18:09:57.581 INFO:teuthology.orchestra.run.smithi049.stdout:0 2023-10-17T18:09:57.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd new 8d64bc08-d3ee-4de7-98ba-3e2910bd27ab 1 2023-10-17T18:09:57.813 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-10-17T18:09:57.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd new 0f0f352e-4d48-43dd-9e29-e16aaebb4889 2 2023-10-17T18:09:57.868 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:57.867+0000 7f4c7d64b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-17T18:09:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:2 2023-10-17T18:09:58.060 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-10-17T18:09:58.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-10-17T18:09:58.103 INFO:tasks.ceph.c1.osd.0:Started 2023-10-17T18:09:58.104 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-10-17T18:09:58.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-10-17T18:09:58.106 INFO:tasks.ceph.c1.osd.1:Started 2023-10-17T18:09:58.106 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-10-17T18:09:58.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-10-17T18:09:58.109 INFO:tasks.ceph.c1.osd.2:Started 2023-10-17T18:09:58.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-17T18:09:58.343 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:09:58.344 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":5,"fsid":"ec7c8da4-7f75-454e-8466-c20b932240f2","created":"2023-10-17T18:09:57.017995+0000","modified":"2023-10-17T18:09:58.047230+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T18:09:58.047230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df221d56-2316-496c-a2fe-ca92353f0a02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8d64bc08-d3ee-4de7-98ba-3e2910bd27ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f0f352e-4d48-43dd-9e29-e16aaebb4889","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:09:58.356 INFO:tasks.ceph.ceph_manager.c1:[] 2023-10-17T18:09:58.356 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-17T18:09:58.429 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:58.428+0000 7f4c7d64b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-17T18:09:58.632 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:58.631+0000 7f4c7d64b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-17T18:09:58.659 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-17T18:09:58.719 INFO:tasks.ceph.c1.osd.1.smithi049.stderr:2023-10-17T18:09:58.718+0000 7f5a16339640 -1 Falling back to public interface 2023-10-17T18:09:58.723 INFO:tasks.ceph.c1.osd.2.smithi049.stderr:2023-10-17T18:09:58.722+0000 7f6878a4c640 -1 Falling back to public interface 2023-10-17T18:09:58.732 INFO:tasks.ceph.c1.osd.0.smithi049.stderr:2023-10-17T18:09:58.731+0000 7fd226cc1640 -1 Falling back to public interface 2023-10-17T18:09:58.859 INFO:teuthology.misc.health.smithi049.stdout: 2023-10-17T18:09:58.859 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":5,"fsid":"ec7c8da4-7f75-454e-8466-c20b932240f2","created":"2023-10-17T18:09:57.017995+0000","modified":"2023-10-17T18:09:58.047230+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T18:09:58.047230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df221d56-2316-496c-a2fe-ca92353f0a02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8d64bc08-d3ee-4de7-98ba-3e2910bd27ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f0f352e-4d48-43dd-9e29-e16aaebb4889","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:09:58.870 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-17T18:09:59.017 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:59.016+0000 7f4c7d64b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-17T18:09:59.715 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:59.714+0000 7f4c7d64b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-17T18:09:59.792 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:59.791+0000 7f4c7d64b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-17T18:09:59.935 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:09:59.934+0000 7f4c7d64b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-17T18:10:00.009 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.008+0000 7f4c7d64b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-17T18:10:00.212 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.211+0000 7f4c7d64b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-17T18:10:00.295 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.294+0000 7f4c7d64b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-17T18:10:00.378 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.377+0000 7f4c7d64b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-17T18:10:00.476 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.475+0000 7f4c7d64b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-17T18:10:00.667 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.666+0000 7f4c7d64b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-17T18:10:00.752 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:00.751+0000 7f4c7d64b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-17T18:10:01.068 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:01.067+0000 7f4c7d64b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-17T18:10:01.144 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:01.143+0000 7f4c7d64b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-17T18:10:01.431 INFO:tasks.ceph.c1.osd.1.smithi049.stderr:2023-10-17T18:10:01.429+0000 7f5a16339640 -1 osd.1 0 log_to_monitors true 2023-10-17T18:10:01.445 INFO:tasks.ceph.c1.osd.0.smithi049.stderr:2023-10-17T18:10:01.444+0000 7fd226cc1640 -1 osd.0 0 log_to_monitors true 2023-10-17T18:10:01.463 INFO:tasks.ceph.c1.osd.2.smithi049.stderr:2023-10-17T18:10:01.462+0000 7f6878a4c640 -1 osd.2 0 log_to_monitors true 2023-10-17T18:10:01.698 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:01.697+0000 7f4c7d64b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-17T18:10:02.030 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:02.029+0000 7f4c7d64b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-17T18:10:02.181 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:02.180+0000 7f4c7d64b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-17T18:10:02.679 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:02.678+0000 7f4c7d64b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-17T18:10:02.852 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:02.851+0000 7f4c7d64b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-17T18:10:03.031 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:03.030+0000 7f4c7d64b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-17T18:10:03.299 INFO:tasks.ceph.c1.osd.0.smithi049.stderr:2023-10-17T18:10:03.297+0000 7fd21b062700 -1 osd.0 0 waiting for initial osdmap 2023-10-17T18:10:03.322 INFO:tasks.ceph.c1.osd.0.smithi049.stderr:2023-10-17T18:10:03.321+0000 7fd212c3c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:03.378 INFO:tasks.ceph.c1.osd.1.smithi049.stderr:2023-10-17T18:10:03.377+0000 7f5a0a6da700 -1 osd.1 0 waiting for initial osdmap 2023-10-17T18:10:03.426 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:03.424+0000 7f4c7d64b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-17T18:10:03.479 INFO:tasks.ceph.c1.osd.1.smithi049.stderr:2023-10-17T18:10:03.478+0000 7f5a022b4700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:03.654 INFO:tasks.ceph.c1.osd.2.smithi049.stderr:2023-10-17T18:10:03.653+0000 7f686cded700 -1 osd.2 0 waiting for initial osdmap 2023-10-17T18:10:03.665 INFO:tasks.ceph.c1.osd.2.smithi049.stderr:2023-10-17T18:10:03.664+0000 7f68649c7700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:04.436 INFO:tasks.ceph.c1.mgr.x.smithi049.stderr:2023-10-17T18:10:04.435+0000 7f4c3ea64700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-17T18:10:05.174 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-17T18:10:05.516 INFO:teuthology.misc.health.smithi049.stdout: 2023-10-17T18:10:05.516 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":9,"fsid":"ec7c8da4-7f75-454e-8466-c20b932240f2","created":"2023-10-17T18:09:57.017995+0000","modified":"2023-10-17T18:10:05.430110+0000","last_up_change":"2023-10-17T18:10:04.299964+0000","last_in_change":"2023-10-17T18:09:58.047230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df221d56-2316-496c-a2fe-ca92353f0a02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":114267},{"type":"v1","addr":"172.21.15.49:6817","nonce":114267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":114267},{"type":"v1","addr":"172.21.15.49:6819","nonce":114267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":114267},{"type":"v1","addr":"172.21.15.49:6823","nonce":114267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":114267},{"type":"v1","addr":"172.21.15.49:6821","nonce":114267}]},"public_addr":"172.21.15.49:6817/114267","cluster_addr":"172.21.15.49:6819/114267","heartbeat_back_addr":"172.21.15.49:6823/114267","heartbeat_front_addr":"172.21.15.49:6821/114267","state":["exists","up"]},{"osd":1,"uuid":"8d64bc08-d3ee-4de7-98ba-3e2910bd27ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":114265},{"type":"v1","addr":"172.21.15.49:6801","nonce":114265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":114265},{"type":"v1","addr":"172.21.15.49:6803","nonce":114265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":114265},{"type":"v1","addr":"172.21.15.49:6807","nonce":114265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":114265},{"type":"v1","addr":"172.21.15.49:6805","nonce":114265}]},"public_addr":"172.21.15.49:6801/114265","cluster_addr":"172.21.15.49:6803/114265","heartbeat_back_addr":"172.21.15.49:6807/114265","heartbeat_front_addr":"172.21.15.49:6805/114265","state":["exists","up"]},{"osd":2,"uuid":"0f0f352e-4d48-43dd-9e29-e16aaebb4889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":114266},{"type":"v1","addr":"172.21.15.49:6809","nonce":114266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":114266},{"type":"v1","addr":"172.21.15.49:6811","nonce":114266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":114266},{"type":"v1","addr":"172.21.15.49:6815","nonce":114266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":114266},{"type":"v1","addr":"172.21.15.49:6813","nonce":114266}]},"public_addr":"172.21.15.49:6809/114266","cluster_addr":"172.21.15.49:6811/114266","heartbeat_back_addr":"172.21.15.49:6815/114266","heartbeat_front_addr":"172.21.15.49:6813/114266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.442549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.424943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:03.655328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:05.528 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-17T18:10:05.529 INFO:tasks.ceph:Creating RBD pool 2023-10-17T18:10:05.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd pool create rbd 8 2023-10-17T18:10:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2023-10-17T18:10:06.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-17T18:10:07.445 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-17T18:10:07.462 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-10-17T18:10:07.463 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-10-17T18:10:07.463 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-10-17T18:10:07.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-10-17T18:10:07.464 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-17T18:10:07.927 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:07.943 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":113853},{"type":"v1","addr":"172.21.15.49:6825","nonce":113853}]},"active_addr":"172.21.15.49:6825/113853","active_change":"2023-10-17T18:10:03.427056+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3239901045}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1140960267}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2966202344}]}]} 2023-10-17T18:10:07.946 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-10-17T18:10:07.946 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-10-17T18:10:07.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-17T18:10:08.284 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:08.285 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"fsid":"ec7c8da4-7f75-454e-8466-c20b932240f2","created":"2023-10-17T18:09:57.017995+0000","modified":"2023-10-17T18:10:07.433650+0000","last_up_change":"2023-10-17T18:10:04.299964+0000","last_in_change":"2023-10-17T18:09:58.047230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T18:10:05.472993+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T18:10:05.898315+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df221d56-2316-496c-a2fe-ca92353f0a02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":114267},{"type":"v1","addr":"172.21.15.49:6817","nonce":114267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":114267},{"type":"v1","addr":"172.21.15.49:6819","nonce":114267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":114267},{"type":"v1","addr":"172.21.15.49:6823","nonce":114267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":114267},{"type":"v1","addr":"172.21.15.49:6821","nonce":114267}]},"public_addr":"172.21.15.49:6817/114267","cluster_addr":"172.21.15.49:6819/114267","heartbeat_back_addr":"172.21.15.49:6823/114267","heartbeat_front_addr":"172.21.15.49:6821/114267","state":["exists","up"]},{"osd":1,"uuid":"8d64bc08-d3ee-4de7-98ba-3e2910bd27ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":114265},{"type":"v1","addr":"172.21.15.49:6801","nonce":114265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":114265},{"type":"v1","addr":"172.21.15.49:6803","nonce":114265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":114265},{"type":"v1","addr":"172.21.15.49:6807","nonce":114265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":114265},{"type":"v1","addr":"172.21.15.49:6805","nonce":114265}]},"public_addr":"172.21.15.49:6801/114265","cluster_addr":"172.21.15.49:6803/114265","heartbeat_back_addr":"172.21.15.49:6807/114265","heartbeat_front_addr":"172.21.15.49:6805/114265","state":["exists","up"]},{"osd":2,"uuid":"0f0f352e-4d48-43dd-9e29-e16aaebb4889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":114266},{"type":"v1","addr":"172.21.15.49:6809","nonce":114266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":114266},{"type":"v1","addr":"172.21.15.49:6811","nonce":114266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":114266},{"type":"v1","addr":"172.21.15.49:6815","nonce":114266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":114266},{"type":"v1","addr":"172.21.15.49:6813","nonce":114266}]},"public_addr":"172.21.15.49:6809/114266","cluster_addr":"172.21.15.49:6811/114266","heartbeat_back_addr":"172.21.15.49:6815/114266","heartbeat_front_addr":"172.21.15.49:6813/114266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.442549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.424943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:03.655328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:08.298 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-10-17T18:10:08.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-17T18:10:08.635 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:08.635 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"fsid":"ec7c8da4-7f75-454e-8466-c20b932240f2","created":"2023-10-17T18:09:57.017995+0000","modified":"2023-10-17T18:10:08.434295+0000","last_up_change":"2023-10-17T18:10:04.299964+0000","last_in_change":"2023-10-17T18:09:58.047230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T18:10:05.472993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T18:10:05.898315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df221d56-2316-496c-a2fe-ca92353f0a02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":114267},{"type":"v1","addr":"172.21.15.49:6817","nonce":114267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":114267},{"type":"v1","addr":"172.21.15.49:6819","nonce":114267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":114267},{"type":"v1","addr":"172.21.15.49:6823","nonce":114267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":114267},{"type":"v1","addr":"172.21.15.49:6821","nonce":114267}]},"public_addr":"172.21.15.49:6817/114267","cluster_addr":"172.21.15.49:6819/114267","heartbeat_back_addr":"172.21.15.49:6823/114267","heartbeat_front_addr":"172.21.15.49:6821/114267","state":["exists","up"]},{"osd":1,"uuid":"8d64bc08-d3ee-4de7-98ba-3e2910bd27ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":114265},{"type":"v1","addr":"172.21.15.49:6801","nonce":114265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":114265},{"type":"v1","addr":"172.21.15.49:6803","nonce":114265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":114265},{"type":"v1","addr":"172.21.15.49:6807","nonce":114265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":114265},{"type":"v1","addr":"172.21.15.49:6805","nonce":114265}]},"public_addr":"172.21.15.49:6801/114265","cluster_addr":"172.21.15.49:6803/114265","heartbeat_back_addr":"172.21.15.49:6807/114265","heartbeat_front_addr":"172.21.15.49:6805/114265","state":["exists","up"]},{"osd":2,"uuid":"0f0f352e-4d48-43dd-9e29-e16aaebb4889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":114266},{"type":"v1","addr":"172.21.15.49:6809","nonce":114266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":114266},{"type":"v1","addr":"172.21.15.49:6811","nonce":114266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":114266},{"type":"v1","addr":"172.21.15.49:6815","nonce":114266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":114266},{"type":"v1","addr":"172.21.15.49:6813","nonce":114266}]},"public_addr":"172.21.15.49:6809/114266","cluster_addr":"172.21.15.49:6811/114266","heartbeat_back_addr":"172.21.15.49:6815/114266","heartbeat_front_addr":"172.21.15.49:6813/114266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.442549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:02.424943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:03.655328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:08.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-10-17T18:10:08.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-10-17T18:10:08.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-10-17T18:10:08.822 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:08.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-17T18:10:08.823 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:08.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-17T18:10:08.825 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:08.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-17T18:10:09.208 INFO:teuthology.orchestra.run.smithi049.stdout:34359738369 2023-10-17T18:10:09.212 INFO:teuthology.orchestra.run.smithi049.stdout:34359738369 2023-10-17T18:10:09.215 INFO:teuthology.orchestra.run.smithi049.stdout:34359738369 2023-10-17T18:10:09.225 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2023-10-17T18:10:09.229 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-10-17T18:10:09.232 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-10-17T18:10:10.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-17T18:10:10.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-17T18:10:10.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-17T18:10:10.597 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:10.611 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2023-10-17T18:10:10.612 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:10.627 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:10.638 INFO:teuthology.orchestra.run.smithi049.stdout:34359738370 2023-10-17T18:10:10.641 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-10-17T18:10:10.641 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:10.652 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2023-10-17T18:10:10.652 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:10.653 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-10-17T18:10:10.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-17T18:10:11.020 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:11.022 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-10-17T18:10:11.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-17T18:10:09.432770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82624,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490176,"statfs":{"total":322122547200,"available":322037940224,"internally_reserved":0,"allocated":1486848,"data_stored":1258566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.442936+0000","last_change":"2023-10-17T18:10:07.441267+0000","last_active":"2023-10-17T18:10:07.442936+0000","last_peered":"2023-10-17T18:10:07.442936+0000","last_clean":"2023-10-17T18:10:07.442936+0000","last_became_active":"2023-10-17T18:10:07.440812+0000","last_became_peered":"2023-10-17T18:10:07.440812+0000","last_unstale":"2023-10-17T18:10:07.442936+0000","last_undegraded":"2023-10-17T18:10:07.442936+0000","last_fullsized":"2023-10-17T18:10:07.442936+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T06:07:25.714738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443478+0000","last_change":"2023-10-17T18:10:07.442399+0000","last_active":"2023-10-17T18:10:07.443478+0000","last_peered":"2023-10-17T18:10:07.443478+0000","last_clean":"2023-10-17T18:10:07.443478+0000","last_became_active":"2023-10-17T18:10:07.442093+0000","last_became_peered":"2023-10-17T18:10:07.442093+0000","last_unstale":"2023-10-17T18:10:07.443478+0000","last_undegraded":"2023-10-17T18:10:07.443478+0000","last_fullsized":"2023-10-17T18:10:07.443478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T05:52:00.760732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443417+0000","last_change":"2023-10-17T18:10:07.441842+0000","last_active":"2023-10-17T18:10:07.443417+0000","last_peered":"2023-10-17T18:10:07.443417+0000","last_clean":"2023-10-17T18:10:07.443417+0000","last_became_active":"2023-10-17T18:10:07.441561+0000","last_became_peered":"2023-10-17T18:10:07.441561+0000","last_unstale":"2023-10-17T18:10:07.443417+0000","last_undegraded":"2023-10-17T18:10:07.443417+0000","last_fullsized":"2023-10-17T18:10:07.443417+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T01:46:03.558593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:08.436097+0000","last_change":"2023-10-17T18:10:07.444378+0000","last_active":"2023-10-17T18:10:08.436097+0000","last_peered":"2023-10-17T18:10:08.436097+0000","last_clean":"2023-10-17T18:10:08.436097+0000","last_became_active":"2023-10-17T18:10:07.444157+0000","last_became_peered":"2023-10-17T18:10:07.444157+0000","last_unstale":"2023-10-17T18:10:08.436097+0000","last_undegraded":"2023-10-17T18:10:08.436097+0000","last_fullsized":"2023-10-17T18:10:08.436097+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:36:50.174739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.937498+0000","last_change":"2023-10-17T18:10:07.441657+0000","last_active":"2023-10-17T18:10:07.937498+0000","last_peered":"2023-10-17T18:10:07.937498+0000","last_clean":"2023-10-17T18:10:07.937498+0000","last_became_active":"2023-10-17T18:10:07.441378+0000","last_became_peered":"2023-10-17T18:10:07.441378+0000","last_unstale":"2023-10-17T18:10:07.937498+0000","last_undegraded":"2023-10-17T18:10:07.937498+0000","last_fullsized":"2023-10-17T18:10:07.937498+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T00:29:22.516840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.442962+0000","last_change":"2023-10-17T18:10:07.441437+0000","last_active":"2023-10-17T18:10:07.442962+0000","last_peered":"2023-10-17T18:10:07.442962+0000","last_clean":"2023-10-17T18:10:07.442962+0000","last_became_active":"2023-10-17T18:10:07.441093+0000","last_became_peered":"2023-10-17T18:10:07.441093+0000","last_unstale":"2023-10-17T18:10:07.442962+0000","last_undegraded":"2023-10-17T18:10:07.442962+0000","last_fullsized":"2023-10-17T18:10:07.442962+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:40:55.453442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:08.436117+0000","last_change":"2023-10-17T18:10:07.443303+0000","last_active":"2023-10-17T18:10:08.436117+0000","last_peered":"2023-10-17T18:10:08.436117+0000","last_clean":"2023-10-17T18:10:08.436117+0000","last_became_active":"2023-10-17T18:10:07.443050+0000","last_became_peered":"2023-10-17T18:10:07.443050+0000","last_unstale":"2023-10-17T18:10:08.436117+0000","last_undegraded":"2023-10-17T18:10:08.436117+0000","last_fullsized":"2023-10-17T18:10:08.436117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:42:32.024987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.444097+0000","last_change":"2023-10-17T18:10:07.443204+0000","last_active":"2023-10-17T18:10:07.444097+0000","last_peered":"2023-10-17T18:10:07.444097+0000","last_clean":"2023-10-17T18:10:07.444097+0000","last_became_active":"2023-10-17T18:10:07.442911+0000","last_became_peered":"2023-10-17T18:10:07.442911+0000","last_unstale":"2023-10-17T18:10:07.444097+0000","last_undegraded":"2023-10-17T18:10:07.444097+0000","last_fullsized":"2023-10-17T18:10:07.444097+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:21:58.237924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443490+0000","last_change":"2023-10-17T18:10:07.442461+0000","last_active":"2023-10-17T18:10:07.443490+0000","last_peered":"2023-10-17T18:10:07.443490+0000","last_clean":"2023-10-17T18:10:07.443490+0000","last_became_active":"2023-10-17T18:10:07.442166+0000","last_became_peered":"2023-10-17T18:10:07.442166+0000","last_unstale":"2023-10-17T18:10:07.443490+0000","last_undegraded":"2023-10-17T18:10:07.443490+0000","last_fullsized":"2023-10-17T18:10:07.443490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T05:29:06.099489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27756,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829844,"statfs":{"total":107374182400,"available":107345760256,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":614388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T18:10:11.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-17T18:10:11.366 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:11.369 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2023-10-17T18:10:11.380 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-17T18:10:09.432770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82624,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490176,"statfs":{"total":322122547200,"available":322037940224,"internally_reserved":0,"allocated":1486848,"data_stored":1258566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.442936+0000","last_change":"2023-10-17T18:10:07.441267+0000","last_active":"2023-10-17T18:10:07.442936+0000","last_peered":"2023-10-17T18:10:07.442936+0000","last_clean":"2023-10-17T18:10:07.442936+0000","last_became_active":"2023-10-17T18:10:07.440812+0000","last_became_peered":"2023-10-17T18:10:07.440812+0000","last_unstale":"2023-10-17T18:10:07.442936+0000","last_undegraded":"2023-10-17T18:10:07.442936+0000","last_fullsized":"2023-10-17T18:10:07.442936+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T06:07:25.714738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443478+0000","last_change":"2023-10-17T18:10:07.442399+0000","last_active":"2023-10-17T18:10:07.443478+0000","last_peered":"2023-10-17T18:10:07.443478+0000","last_clean":"2023-10-17T18:10:07.443478+0000","last_became_active":"2023-10-17T18:10:07.442093+0000","last_became_peered":"2023-10-17T18:10:07.442093+0000","last_unstale":"2023-10-17T18:10:07.443478+0000","last_undegraded":"2023-10-17T18:10:07.443478+0000","last_fullsized":"2023-10-17T18:10:07.443478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T05:52:00.760732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443417+0000","last_change":"2023-10-17T18:10:07.441842+0000","last_active":"2023-10-17T18:10:07.443417+0000","last_peered":"2023-10-17T18:10:07.443417+0000","last_clean":"2023-10-17T18:10:07.443417+0000","last_became_active":"2023-10-17T18:10:07.441561+0000","last_became_peered":"2023-10-17T18:10:07.441561+0000","last_unstale":"2023-10-17T18:10:07.443417+0000","last_undegraded":"2023-10-17T18:10:07.443417+0000","last_fullsized":"2023-10-17T18:10:07.443417+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T01:46:03.558593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:08.436097+0000","last_change":"2023-10-17T18:10:07.444378+0000","last_active":"2023-10-17T18:10:08.436097+0000","last_peered":"2023-10-17T18:10:08.436097+0000","last_clean":"2023-10-17T18:10:08.436097+0000","last_became_active":"2023-10-17T18:10:07.444157+0000","last_became_peered":"2023-10-17T18:10:07.444157+0000","last_unstale":"2023-10-17T18:10:08.436097+0000","last_undegraded":"2023-10-17T18:10:08.436097+0000","last_fullsized":"2023-10-17T18:10:08.436097+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:36:50.174739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.937498+0000","last_change":"2023-10-17T18:10:07.441657+0000","last_active":"2023-10-17T18:10:07.937498+0000","last_peered":"2023-10-17T18:10:07.937498+0000","last_clean":"2023-10-17T18:10:07.937498+0000","last_became_active":"2023-10-17T18:10:07.441378+0000","last_became_peered":"2023-10-17T18:10:07.441378+0000","last_unstale":"2023-10-17T18:10:07.937498+0000","last_undegraded":"2023-10-17T18:10:07.937498+0000","last_fullsized":"2023-10-17T18:10:07.937498+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T00:29:22.516840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.442962+0000","last_change":"2023-10-17T18:10:07.441437+0000","last_active":"2023-10-17T18:10:07.442962+0000","last_peered":"2023-10-17T18:10:07.442962+0000","last_clean":"2023-10-17T18:10:07.442962+0000","last_became_active":"2023-10-17T18:10:07.441093+0000","last_became_peered":"2023-10-17T18:10:07.441093+0000","last_unstale":"2023-10-17T18:10:07.442962+0000","last_undegraded":"2023-10-17T18:10:07.442962+0000","last_fullsized":"2023-10-17T18:10:07.442962+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:40:55.453442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:08.436117+0000","last_change":"2023-10-17T18:10:07.443303+0000","last_active":"2023-10-17T18:10:08.436117+0000","last_peered":"2023-10-17T18:10:08.436117+0000","last_clean":"2023-10-17T18:10:08.436117+0000","last_became_active":"2023-10-17T18:10:07.443050+0000","last_became_peered":"2023-10-17T18:10:07.443050+0000","last_unstale":"2023-10-17T18:10:08.436117+0000","last_undegraded":"2023-10-17T18:10:08.436117+0000","last_fullsized":"2023-10-17T18:10:08.436117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:42:32.024987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.444097+0000","last_change":"2023-10-17T18:10:07.443204+0000","last_active":"2023-10-17T18:10:07.444097+0000","last_peered":"2023-10-17T18:10:07.444097+0000","last_clean":"2023-10-17T18:10:07.444097+0000","last_became_active":"2023-10-17T18:10:07.442911+0000","last_became_peered":"2023-10-17T18:10:07.442911+0000","last_unstale":"2023-10-17T18:10:07.444097+0000","last_undegraded":"2023-10-17T18:10:07.444097+0000","last_fullsized":"2023-10-17T18:10:07.444097+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:21:58.237924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-17T18:10:07.443490+0000","last_change":"2023-10-17T18:10:07.442461+0000","last_active":"2023-10-17T18:10:07.443490+0000","last_peered":"2023-10-17T18:10:07.443490+0000","last_clean":"2023-10-17T18:10:07.443490+0000","last_became_active":"2023-10-17T18:10:07.442166+0000","last_became_peered":"2023-10-17T18:10:07.442166+0000","last_unstale":"2023-10-17T18:10:07.443490+0000","last_undegraded":"2023-10-17T18:10:07.443490+0000","last_fullsized":"2023-10-17T18:10:07.443490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:06.431207+0000","last_clean_scrub_stamp":"2023-10-17T18:10:06.431207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T05:29:06.099489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27756,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829844,"statfs":{"total":107374182400,"available":107345760256,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":614388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T18:10:11.381 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-10-17T18:10:11.381 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-10-17T18:10:11.382 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-10-17T18:10:11.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-10-17T18:10:11.758 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-17T18:10:11.772 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-10-17T18:10:11.773 INFO:teuthology.run_tasks:Running task ceph... 2023-10-17T18:10:11.783 INFO:tasks.ceph:Creating ceph cluster c2... 2023-10-17T18:10:11.783 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-17T18:10:11.783 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7430137', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-16-1247-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 993, 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-10-16-1247-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKx21743cUlmo6TKUbvR5PAR3YJyl6RDmMH8t+R6LplmFaEgnLfqs3jO/mdESrHA1/7zybwQDizFBDdcbLTm/18=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxePA3TlvYqb+4IRWm+eI1CkcB6eXCdh5zeTPXhaTJWuWW7972TipdWZLJP4OU5UiJN2/V8uTym1MqHpDcG0e4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cff47d96488123dae42a0c6258323ad6422fe0a6', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_21:55:28', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-17T18:10:11.784 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-17T18:10:11.786 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-17T18:10:11.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T18:10:11.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-10-17T18:10:11.882 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:11.883 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2023-10-17T18:10:11.942 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-17T18:10:11.942 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2023-10-17T18:10:12.001 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-17T18:10:12.002 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:10:12.002 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 395481 Links: 1 2023-10-17T18:10:12.002 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:10:12.002 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:10:12.002 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-10-17 18:09:30.142380482 +0000 2023-10-17T18:10:12.003 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-10-17 18:05:12.057592210 +0000 2023-10-17T18:10:12.003 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-10-17 18:05:12.057592210 +0000 2023-10-17T18:10:12.003 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-10-17T18:10:12.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-17T18:10:12.069 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-10-17T18:10:12.070 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-10-17T18:10:12.070 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000247567 s, 2.1 MB/s 2023-10-17T18:10:12.071 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-17T18:10:12.131 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2023-10-17T18:10:12.189 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-17T18:10:12.190 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:10:12.190 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 400461 Links: 1 2023-10-17T18:10:12.190 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:10:12.190 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:10:12.190 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-10-17 18:09:30.142380482 +0000 2023-10-17T18:10:12.191 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-10-17 18:05:12.555583524 +0000 2023-10-17T18:10:12.191 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-10-17 18:05:12.555583524 +0000 2023-10-17T18:10:12.191 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-10-17T18:10:12.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-17T18:10:12.257 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-10-17T18:10:12.257 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-10-17T18:10:12.257 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000195514 s, 2.6 MB/s 2023-10-17T18:10:12.258 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-17T18:10:12.318 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2023-10-17T18:10:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-17T18:10:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:10:12.377 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 393874 Links: 1 2023-10-17T18:10:12.377 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:10:12.377 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:10:12.377 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-10-17 18:09:30.142380482 +0000 2023-10-17T18:10:12.377 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-10-17 18:05:13.072574507 +0000 2023-10-17T18:10:12.378 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-10-17 18:05:13.072574507 +0000 2023-10-17T18:10:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-10-17T18:10:12.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-17T18:10:12.442 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-10-17T18:10:12.443 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-10-17T18:10:12.443 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000221539 s, 2.3 MB/s 2023-10-17T18:10:12.444 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-17T18:10:12.503 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2023-10-17T18:10:12.560 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-17T18:10:12.561 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-17T18:10:12.561 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 400701 Links: 1 2023-10-17T18:10:12.561 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-17T18:10:12.561 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-10-17T18:10:12.561 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-10-17 18:09:30.142380482 +0000 2023-10-17T18:10:12.562 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-10-17 18:05:13.574565752 +0000 2023-10-17T18:10:12.562 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-10-17 18:05:13.574565752 +0000 2023-10-17T18:10:12.562 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-10-17T18:10:12.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-17T18:10:12.627 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-10-17T18:10:12.627 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-10-17T18:10:12.627 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000231718 s, 2.2 MB/s 2023-10-17T18:10:12.628 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-17T18:10:12.688 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:10:12.688 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T18:10:12.688 INFO:tasks.ceph:Generating config... 2023-10-17T18:10:12.690 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-17T18:10:12.690 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-17T18:10:12.690 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-17T18:10:12.690 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-17T18:10:12.691 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:10:12.691 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-17T18:10:12.691 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-17T18:10:12.692 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-17T18:10:12.692 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-17T18:10:12.692 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-17T18:10:12.692 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-17T18:10:12.693 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-17T18:10:12.693 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-17T18:10:12.693 INFO:tasks.ceph:[client] setuser = ceph 2023-10-17T18:10:12.694 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-17T18:10:12.694 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-17T18:10:12.694 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-17T18:10:12.694 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-17T18:10:12.695 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-17T18:10:12.695 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-17T18:10:12.695 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-17T18:10:12.695 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-17T18:10:12.696 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-17T18:10:12.696 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-17T18:10:12.696 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-17T18:10:12.696 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-17T18:10:12.697 INFO:tasks.ceph:Setting up c2.mon.a... 2023-10-17T18:10:12.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-10-17T18:10:12.783 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.keyring 2023-10-17T18:10:12.786 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-10-17T18:10:12.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-17T18:10:12.859 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.71')] 2023-10-17T18:10:12.860 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-17T18:10:12.861 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:12.861 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T18:10:12.918 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --print /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:12.999 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:13.000 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:13.001 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:13.001 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 3e63af52-2e83-45bf-8b85-954ab730fd36 2023-10-17T18:10:13.001 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = pacific 2023-10-17T18:10:13.001 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2023-10-17T18:10:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 3e63af52-2e83-45bf-8b85-954ab730fd36 2023-10-17T18:10:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2023-10-17T18:10:13.000347+0000 2023-10-17T18:10:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:created 2023-10-17T18:10:13.000347+0000 2023-10-17T18:10:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 16 (pacific) 2023-10-17T18:10:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2023-10-17T18:10:13.003 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2023-10-17T18:10:13.003 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-10-17T18:10:13.003 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-17T18:10:13.059 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 3e63af52-2e83-45bf-8b85-954ab730fd36... 2023-10-17T18:10:13.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-17T18:10:13.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-17T18:10:13.116 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-10-17T18:10:13.116 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2023-10-17T18:10:13.117 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T18:10:13.117 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2023-10-17T18:10:13.117 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.117 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2023-10-17T18:10:13.117 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.118 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2023-10-17T18:10:13.118 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.118 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2023-10-17T18:10:13.118 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2023-10-17T18:10:13.119 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.119 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2023-10-17T18:10:13.119 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2023-10-17T18:10:13.119 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.119 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T18:10:13.120 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2023-10-17T18:10:13.120 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.120 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T18:10:13.120 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T18:10:13.121 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.121 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2023-10-17T18:10:13.121 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.121 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2023-10-17T18:10:13.121 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2023-10-17T18:10:13.122 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2023-10-17T18:10:13.122 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2023-10-17T18:10:13.122 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2023-10-17T18:10:13.122 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2023-10-17T18:10:13.123 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2023-10-17T18:10:13.123 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T18:10:13.123 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T18:10:13.123 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2023-10-17T18:10:13.123 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.124 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T18:10:13.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.124 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2023-10-17T18:10:13.125 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.125 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2023-10-17T18:10:13.125 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.125 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2023-10-17T18:10:13.125 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2023-10-17T18:10:13.126 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2023-10-17T18:10:13.126 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.71 2023-10-17T18:10:13.126 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 3e63af52-2e83-45bf-8b85-954ab730fd36 2023-10-17T18:10:13.126 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.126 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2023-10-17T18:10:13.127 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2023-10-17T18:10:13.127 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.127 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2023-10-17T18:10:13.127 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2023-10-17T18:10:13.127 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2023-10-17T18:10:13.128 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.128 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2023-10-17T18:10:13.128 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2023-10-17T18:10:13.128 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.128 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2023-10-17T18:10:13.129 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2023-10-17T18:10:13.129 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2023-10-17T18:10:13.129 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.129 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2023-10-17T18:10:13.129 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2023-10-17T18:10:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2023-10-17T18:10:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2023-10-17T18:10:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.131 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2023-10-17T18:10:13.131 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.131 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2023-10-17T18:10:13.131 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2023-10-17T18:10:13.132 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.132 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2023-10-17T18:10:13.132 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2023-10-17T18:10:13.132 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:10:13.132 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2023-10-17T18:10:13.133 INFO:teuthology.orchestra.run.smithi049.stdout: rocksdb delete range threshold = 0 2023-10-17T18:10:13.133 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.133 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2023-10-17T18:10:13.133 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:10:13.133 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2023-10-17T18:10:13.134 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T18:10:13.134 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T18:10:13.134 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:10:13.135 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:10:13.135 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2023-10-17T18:10:13.135 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.135 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2023-10-17T18:10:13.135 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-17T18:10:13.136 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-17T18:10:13.136 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2023-10-17T18:10:13.136 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-17T18:10:13.136 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2023-10-17T18:10:13.136 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.137 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T18:10:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T18:10:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T18:10:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T18:10:13.138 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T18:10:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T18:10:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2023-10-17T18:10:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2023-10-17T18:10:13.139 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2023-10-17T18:10:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:10:13.140 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2023-10-17T18:10:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2023-10-17T18:10:13.140 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2023-10-17T18:10:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2023-10-17T18:10:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T18:10:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T18:10:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw = 20 2023-10-17T18:10:13.141 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw sync = 20 2023-10-17T18:10:13.142 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt require ssl = False 2023-10-17T18:10:13.142 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt s3 kms backend = testing 2023-10-17T18:10:13.142 INFO:teuthology.orchestra.run.smithi049.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:10:13.142 INFO:teuthology.orchestra.run.smithi049.stdout: rgw curl low speed time = 300 2023-10-17T18:10:13.142 INFO:teuthology.orchestra.run.smithi049.stdout: rgw data log num shards = 4 2023-10-17T18:10:13.143 INFO:teuthology.orchestra.run.smithi049.stdout: rgw md log max shards = 4 2023-10-17T18:10:13.143 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync data inject err probability = 0.1 2023-10-17T18:10:13.143 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync log trim interval = 0 2023-10-17T18:10:13.143 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync meta inject err probability = 0.1 2023-10-17T18:10:13.143 INFO:teuthology.orchestra.run.smithi049.stdout: rgw sync obj etag verify = True 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout: setgroup = ceph 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout: setuser = ceph 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout: rgw data notify interval msec = 0 2023-10-17T18:10:13.144 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-10-17T18:10:13.145 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 10 2023-10-17T18:10:13.145 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-10-17T18:10:13.145 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 1000 2023-10-17T18:10:13.145 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-10-17T18:10:13.146 INFO:teuthology.orchestra.run.smithi049.stdout: osd_max_omap_entries_per_request = 10000 2023-10-17T18:10:13.146 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-10-17T18:10:13.147 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-10-17T18:10:13.147 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-17T18:10:13.147 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-10-17T18:10:13.147 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.147 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-10-17T18:10:13.148 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.148 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-10-17T18:10:13.148 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.148 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-10-17T18:10:13.149 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-10-17T18:10:13.149 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.149 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-10-17T18:10:13.149 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-10-17T18:10:13.149 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.150 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-17T18:10:13.150 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-10-17T18:10:13.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.150 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-10-17T18:10:13.151 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-17T18:10:13.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.151 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-10-17T18:10:13.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.151 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-10-17T18:10:13.152 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-10-17T18:10:13.152 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-10-17T18:10:13.152 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-10-17T18:10:13.152 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-10-17T18:10:13.152 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-10-17T18:10:13.153 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-10-17T18:10:13.153 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-17T18:10:13.153 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-17T18:10:13.153 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-10-17T18:10:13.153 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.154 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-17T18:10:13.154 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.154 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-10-17T18:10:13.154 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.154 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-10-17T18:10:13.155 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.155 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-10-17T18:10:13.155 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-10-17T18:10:13.155 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-10-17T18:10:13.155 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.71 2023-10-17T18:10:13.156 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 3e63af52-2e83-45bf-8b85-954ab730fd36 2023-10-17T18:10:13.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.156 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-10-17T18:10:13.156 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-10-17T18:10:13.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.157 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-10-17T18:10:13.157 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-10-17T18:10:13.157 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-10-17T18:10:13.157 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.157 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-10-17T18:10:13.158 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-10-17T18:10:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-10-17T18:10:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-10-17T18:10:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-10-17T18:10:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-10-17T18:10:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-10-17T18:10:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:10:13.162 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-10-17T18:10:13.162 INFO:teuthology.orchestra.run.smithi071.stdout: rocksdb delete range threshold = 0 2023-10-17T18:10:13.162 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.162 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-10-17T18:10:13.162 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:10:13.163 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-10-17T18:10:13.163 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-10-17T18:10:13.163 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-10-17T18:10:13.163 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:10:13.163 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-10-17T18:10:13.164 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-10-17T18:10:13.165 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-10-17T18:10:13.165 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-10-17T18:10:13.165 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-10-17T18:10:13.165 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-10-17T18:10:13.165 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-10-17T18:10:13.166 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-10-17T18:10:13.166 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-10-17T18:10:13.166 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.166 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-17T18:10:13.166 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-10-17T18:10:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-10-17T18:10:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-10-17T18:10:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-17T18:10:13.167 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-17T18:10:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-10-17T18:10:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-10-17T18:10:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-10-17T18:10:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:13.169 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-10-17T18:10:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-10-17T18:10:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-10-17T18:10:13.169 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-10-17T18:10:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-17T18:10:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-17T18:10:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2023-10-17T18:10:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw sync = 20 2023-10-17T18:10:13.170 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2023-10-17T18:10:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2023-10-17T18:10:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-17T18:10:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: rgw curl low speed time = 300 2023-10-17T18:10:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data log num shards = 4 2023-10-17T18:10:13.171 INFO:teuthology.orchestra.run.smithi071.stdout: rgw md log max shards = 4 2023-10-17T18:10:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync data inject err probability = 0.1 2023-10-17T18:10:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync log trim interval = 0 2023-10-17T18:10:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync meta inject err probability = 0.1 2023-10-17T18:10:13.172 INFO:teuthology.orchestra.run.smithi071.stdout: rgw sync obj etag verify = True 2023-10-17T18:10:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2023-10-17T18:10:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2023-10-17T18:10:13.173 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-10-17T18:10:13.173 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-10-17T18:10:13.173 INFO:teuthology.orchestra.run.smithi071.stdout: rgw data notify interval msec = 0 2023-10-17T18:10:13.174 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-10-17T18:10:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10 2023-10-17T18:10:13.174 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-10-17T18:10:13.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 1000 2023-10-17T18:10:13.174 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-10-17T18:10:13.175 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_omap_entries_per_request = 10000 2023-10-17T18:10:13.176 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-10-17T18:10:13.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-10-17T18:10:13.274 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-17T18:10:13.275 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:13.275 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-10-17T18:10:13.293 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:13.294 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-10-17T18:10:13.351 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T18:10:13.351 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:10:13.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/c2.keyring 2023-10-17T18:10:13.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-17T18:10:13.397 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-17T18:10:13.398 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:13.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T18:10:13.460 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:13.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/c2.keyring 2023-10-17T18:10:13.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-17T18:10:13.496 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:13.496 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:13.554 INFO:tasks.ceph:Setting up mon nodes... 2023-10-17T18:10:13.554 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-17T18:10:13.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-10-17T18:10:13.650 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-10-17T18:10:13.651 INFO:tasks.ceph:Setting up mds nodes... 2023-10-17T18:10:13.651 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-17T18:10:13.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-10-17T18:10:13.695 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.client.0.keyring 2023-10-17T18:10:13.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-10-17T18:10:13.750 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/c2.client.1.keyring 2023-10-17T18:10:13.763 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-17T18:10:13.763 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-17T18:10:13.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-10-17T18:10:13.830 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:10:13.830 INFO:tasks.ceph:role: c2.osd.0 2023-10-17T18:10:13.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T18:10:13.831 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-17T18:10:13.904 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:10:13.904 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:10:13.905 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:10:13.906 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:10:13.906 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:10:14.332 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-10-17T18:10:14.334 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-10-17T18:10:14.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-10-17T18:10:14.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-10-17T18:10:14.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-10-17T18:10:14.620 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:10:14.621 INFO:tasks.ceph:role: c2.osd.1 2023-10-17T18:10:14.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T18:10:14.621 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-17T18:10:14.695 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:10:14.695 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:10:14.696 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:10:14.697 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:10:14.697 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:10:15.120 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-10-17T18:10:15.121 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-10-17T18:10:15.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-10-17T18:10:15.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-10-17T18:10:15.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-10-17T18:10:15.329 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-17T18:10:15.329 INFO:tasks.ceph:role: c2.osd.2 2023-10-17T18:10:15.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T18:10:15.330 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-17T18:10:15.403 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-17T18:10:15.403 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-17T18:10:15.404 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-17T18:10:15.404 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-17T18:10:15.404 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-17T18:10:15.404 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-10-17T18:10:15.404 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-17T18:10:15.405 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-17T18:10:15.405 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-17T18:10:15.405 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-17T18:10:15.859 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-10-17T18:10:15.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-10-17T18:10:15.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-10-17T18:10:16.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-10-17T18:10:16.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:16.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.137+0000 7f26f379a640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-10-17T18:10:16.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.139+0000 7f26f379a640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-10-17T18:10:16.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.139+0000 7f26f379a640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-17T18:10:16.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.139+0000 7f26f379a640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-17T18:10:16.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.139+0000 7f26f379a640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-17T18:10:16.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:16.147+0000 7f26f379a640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-10-17T18:10:18.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-10-17T18:10:18.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:18.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.349+0000 7f49f0ecd640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-10-17T18:10:18.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.349+0000 7f49f0ecd640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-10-17T18:10:18.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.349+0000 7f49f0ecd640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-17T18:10:18.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.349+0000 7f49f0ecd640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-17T18:10:18.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.349+0000 7f49f0ecd640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-17T18:10:18.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:18.350+0000 7f49f0ecd640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-10-17T18:10:20.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-10-17T18:10:20.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:20.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-10-17T18:10:20.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-10-17T18:10:20.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-17T18:10:20.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-17T18:10:20.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-17T18:10:20.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:20.559+0000 7f9cf1758640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-10-17T18:10:22.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-10-17T18:10:22.667 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-17T18:10:22.668 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:22.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-10-17T18:10:22.738 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:22.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-10-17T18:10:22.808 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:22.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-10-17T18:10:22.877 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:22.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-10-17T18:10:22.945 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:22.946 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-10-17T18:10:23.005 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:23.005 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-10-17T18:10:23.063 INFO:tasks.ceph:Adding keys to all mons... 2023-10-17T18:10:23.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/c2.keyring 2023-10-17T18:10:23.129 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2023-10-17T18:10:23.129 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAFzi5lydqYJhAAvpT8rMMWH54Ar4tjGct5kw== 2023-10-17T18:10:23.130 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-10-17T18:10:23.130 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAIzi5lweFJCBAAlJALTM3eZHOCwXTe2mxGzA== 2023-10-17T18:10:23.130 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-10-17T18:10:23.131 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAKzi5lDa3mFBAA/BGugRg9Ty55PV0IJeKAOg== 2023-10-17T18:10:23.131 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-10-17T18:10:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAMzi5l02JnIRAAZ4U2UEfxp5JTrSbqkBKOlw== 2023-10-17T18:10:23.132 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-10-17T18:10:23.132 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAFzi5l2TJ5KRAAuJOqHAZvHSCpIiTP6Jb0Lw== 2023-10-17T18:10:23.133 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2023-10-17T18:10:23.133 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAFzi5lnkrFLBAA+8KNhanql5J2i5s0eUQ26Q== 2023-10-17T18:10:23.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-17T18:10:23.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:10:23.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:10:23.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-17T18:10:23.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T18:10:23.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-17T18:10:23.503 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-17T18:10:23.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-10-17T18:10:23.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-10-17T18:10:23.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-10-17T18:10:23.656 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-10-17T18:10:23.714 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-10-17T18:10:23.714 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-10-17T18:10:23.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-10-17T18:10:23.756 INFO:tasks.ceph.c2.mon.a:Started 2023-10-17T18:10:23.757 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-10-17T18:10:23.757 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-10-17T18:10:23.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-10-17T18:10:23.760 INFO:tasks.ceph.c2.mgr.x:Started 2023-10-17T18:10:23.760 INFO:tasks.ceph:Setting crush tunables to default 2023-10-17T18:10:23.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd crush tunables default 2023-10-17T18:10:23.965 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2023-10-17T18:10:23.976 INFO:tasks.ceph:check_enable_crimson: False 2023-10-17T18:10:23.976 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-10-17T18:10:23.976 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:23.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-10-17T18:10:24.004 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:24.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-10-17T18:10:24.070 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-10-17T18:10:24.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-10-17T18:10:24.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new 2a645805-b941-44c0-9fa2-ba41227af2da 0 2023-10-17T18:10:24.158 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:24.157+0000 7f9ada3f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-17T18:10:24.340 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:24.339+0000 7f9ada3f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-17T18:10:24.401 INFO:teuthology.orchestra.run.smithi071.stdout:0 2023-10-17T18:10:24.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new 1a0b39c2-084f-4cd9-a96a-e7836ef8c279 1 2023-10-17T18:10:24.626 INFO:teuthology.orchestra.run.smithi071.stdout:1 2023-10-17T18:10:24.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd new bf7071b3-55cb-48ce-9258-b760e12a9547 2 2023-10-17T18:10:24.701 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:24.700+0000 7f9ada3f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-17T18:10:24.853 INFO:teuthology.orchestra.run.smithi071.stdout:2 2023-10-17T18:10:24.865 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-10-17T18:10:24.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-10-17T18:10:24.868 INFO:tasks.ceph.c2.osd.0:Started 2023-10-17T18:10:24.868 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-10-17T18:10:24.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-10-17T18:10:24.871 INFO:tasks.ceph.c2.osd.1:Started 2023-10-17T18:10:24.871 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-10-17T18:10:24.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-10-17T18:10:24.873 INFO:tasks.ceph.c2.osd.2:Started 2023-10-17T18:10:24.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-17T18:10:25.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:25.118 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"fsid":"3e63af52-2e83-45bf-8b85-954ab730fd36","created":"2023-10-17T18:10:23.870596+0000","modified":"2023-10-17T18:10:24.852340+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T18:10:24.852340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2a645805-b941-44c0-9fa2-ba41227af2da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a0b39c2-084f-4cd9-a96a-e7836ef8c279","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf7071b3-55cb-48ce-9258-b760e12a9547","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:25.130 INFO:tasks.ceph.ceph_manager.c2:[] 2023-10-17T18:10:25.131 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-17T18:10:25.234 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2023-10-17T18:10:25.232+0000 7f1696ba4640 -1 Falling back to public interface 2023-10-17T18:10:25.259 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:25.257+0000 7f9ada3f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-17T18:10:25.433 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-17T18:10:25.460 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:25.459+0000 7f9ada3f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-17T18:10:25.480 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2023-10-17T18:10:25.479+0000 7fb642b34640 -1 Falling back to public interface 2023-10-17T18:10:25.488 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2023-10-17T18:10:25.487+0000 7f7069ce0640 -1 Falling back to public interface 2023-10-17T18:10:25.627 INFO:teuthology.misc.health.smithi071.stdout: 2023-10-17T18:10:25.627 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":5,"fsid":"3e63af52-2e83-45bf-8b85-954ab730fd36","created":"2023-10-17T18:10:23.870596+0000","modified":"2023-10-17T18:10:24.852340+0000","last_up_change":"0.000000","last_in_change":"2023-10-17T18:10:24.852340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2a645805-b941-44c0-9fa2-ba41227af2da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a0b39c2-084f-4cd9-a96a-e7836ef8c279","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf7071b3-55cb-48ce-9258-b760e12a9547","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:25.638 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-17T18:10:25.833 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:25.832+0000 7f9ada3f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-17T18:10:26.496 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:26.495+0000 7f9ada3f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-17T18:10:26.568 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:26.567+0000 7f9ada3f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-17T18:10:26.704 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:26.702+0000 7f9ada3f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-17T18:10:26.773 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:26.772+0000 7f9ada3f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-17T18:10:26.954 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:26.953+0000 7f9ada3f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-17T18:10:27.032 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.031+0000 7f9ada3f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-17T18:10:27.109 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.108+0000 7f9ada3f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-17T18:10:27.208 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.207+0000 7f9ada3f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-17T18:10:27.390 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.388+0000 7f9ada3f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-17T18:10:27.472 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.472+0000 7f9ada3f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-17T18:10:27.761 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.761+0000 7f9ada3f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-17T18:10:27.831 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:27.831+0000 7f9ada3f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-17T18:10:27.976 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2023-10-17T18:10:27.976+0000 7f1696ba4640 -1 osd.0 0 log_to_monitors true 2023-10-17T18:10:28.213 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2023-10-17T18:10:28.213+0000 7fb642b34640 -1 osd.1 0 log_to_monitors true 2023-10-17T18:10:28.238 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2023-10-17T18:10:28.238+0000 7f7069ce0640 -1 osd.2 0 log_to_monitors true 2023-10-17T18:10:28.366 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:28.366+0000 7f9ada3f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-17T18:10:28.681 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:28.681+0000 7f9ada3f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-17T18:10:28.824 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:28.824+0000 7f9ada3f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-17T18:10:29.245 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:29.245+0000 7f9ada3f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-17T18:10:29.401 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:29.401+0000 7f9ada3f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-17T18:10:29.562 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:29.562+0000 7f9ada3f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-17T18:10:29.573 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2023-10-17T18:10:29.573+0000 7f168af45700 -1 osd.0 0 waiting for initial osdmap 2023-10-17T18:10:29.588 INFO:tasks.ceph.c2.osd.0.smithi071.stderr:2023-10-17T18:10:29.588+0000 7f1682b1f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:29.871 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:29.871+0000 7f9ada3f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-17T18:10:30.880 INFO:tasks.ceph.c2.mgr.x.smithi071.stderr:2023-10-17T18:10:30.880+0000 7f9a9b80f700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-17T18:10:30.925 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2023-10-17T18:10:30.925+0000 7f705e882700 -1 osd.2 0 waiting for initial osdmap 2023-10-17T18:10:30.929 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2023-10-17T18:10:30.929+0000 7fb636ed5700 -1 osd.1 0 waiting for initial osdmap 2023-10-17T18:10:30.939 INFO:tasks.ceph.c2.osd.2.smithi071.stderr:2023-10-17T18:10:30.939+0000 7f7055c5b700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:30.947 INFO:tasks.ceph.c2.osd.1.smithi071.stderr:2023-10-17T18:10:30.947+0000 7fb62eaaf700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-17T18:10:31.942 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-17T18:10:32.266 INFO:teuthology.misc.health.smithi071.stdout: 2023-10-17T18:10:32.266 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"3e63af52-2e83-45bf-8b85-954ab730fd36","created":"2023-10-17T18:10:23.870596+0000","modified":"2023-10-17T18:10:32.043413+0000","last_up_change":"2023-10-17T18:10:31.034329+0000","last_in_change":"2023-10-17T18:10:24.852340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T18:10:31.915965+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a645805-b941-44c0-9fa2-ba41227af2da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":114402},{"type":"v1","addr":"172.21.15.71:6801","nonce":114402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":114402},{"type":"v1","addr":"172.21.15.71:6803","nonce":114402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":114402},{"type":"v1","addr":"172.21.15.71:6807","nonce":114402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":114402},{"type":"v1","addr":"172.21.15.71:6805","nonce":114402}]},"public_addr":"172.21.15.71:6801/114402","cluster_addr":"172.21.15.71:6803/114402","heartbeat_back_addr":"172.21.15.71:6807/114402","heartbeat_front_addr":"172.21.15.71:6805/114402","state":["exists","up"]},{"osd":1,"uuid":"1a0b39c2-084f-4cd9-a96a-e7836ef8c279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":114403},{"type":"v1","addr":"172.21.15.71:6809","nonce":114403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":114403},{"type":"v1","addr":"172.21.15.71:6811","nonce":114403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":114403},{"type":"v1","addr":"172.21.15.71:6815","nonce":114403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":114403},{"type":"v1","addr":"172.21.15.71:6813","nonce":114403}]},"public_addr":"172.21.15.71:6809/114403","cluster_addr":"172.21.15.71:6811/114403","heartbeat_back_addr":"172.21.15.71:6815/114403","heartbeat_front_addr":"172.21.15.71:6813/114403","state":["exists","up"]},{"osd":2,"uuid":"bf7071b3-55cb-48ce-9258-b760e12a9547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":114420},{"type":"v1","addr":"172.21.15.71:6817","nonce":114420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":114420},{"type":"v1","addr":"172.21.15.71:6819","nonce":114420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":114420},{"type":"v1","addr":"172.21.15.71:6823","nonce":114420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":114420},{"type":"v1","addr":"172.21.15.71:6821","nonce":114420}]},"public_addr":"172.21.15.71:6817/114420","cluster_addr":"172.21.15.71:6819/114420","heartbeat_back_addr":"172.21.15.71:6823/114420","heartbeat_front_addr":"172.21.15.71:6821/114420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:28.950378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.250412+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.257797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:32.278 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-17T18:10:32.279 INFO:tasks.ceph:Creating RBD pool 2023-10-17T18:10:32.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd pool create rbd 8 2023-10-17T18:10:33.054 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2023-10-17T18:10:33.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-17T18:10:34.059 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-17T18:10:34.071 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-10-17T18:10:34.071 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-10-17T18:10:34.072 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-10-17T18:10:34.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-10-17T18:10:34.073 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-17T18:10:34.448 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:34.461 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":113990},{"type":"v1","addr":"172.21.15.71:6825","nonce":113990}]},"active_addr":"172.21.15.71:6825/113990","active_change":"2023-10-17T18:10:29.872577+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2458200168}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1182980126}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1783112356}]}]} 2023-10-17T18:10:34.467 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-10-17T18:10:34.467 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-10-17T18:10:34.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-17T18:10:34.785 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:34.785 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"fsid":"3e63af52-2e83-45bf-8b85-954ab730fd36","created":"2023-10-17T18:10:23.870596+0000","modified":"2023-10-17T18:10:34.049917+0000","last_up_change":"2023-10-17T18:10:31.034329+0000","last_in_change":"2023-10-17T18:10:24.852340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T18:10:31.915965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T18:10:32.612553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a645805-b941-44c0-9fa2-ba41227af2da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":114402},{"type":"v1","addr":"172.21.15.71:6801","nonce":114402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":114402},{"type":"v1","addr":"172.21.15.71:6803","nonce":114402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":114402},{"type":"v1","addr":"172.21.15.71:6807","nonce":114402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":114402},{"type":"v1","addr":"172.21.15.71:6805","nonce":114402}]},"public_addr":"172.21.15.71:6801/114402","cluster_addr":"172.21.15.71:6803/114402","heartbeat_back_addr":"172.21.15.71:6807/114402","heartbeat_front_addr":"172.21.15.71:6805/114402","state":["exists","up"]},{"osd":1,"uuid":"1a0b39c2-084f-4cd9-a96a-e7836ef8c279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":114403},{"type":"v1","addr":"172.21.15.71:6809","nonce":114403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":114403},{"type":"v1","addr":"172.21.15.71:6811","nonce":114403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":114403},{"type":"v1","addr":"172.21.15.71:6815","nonce":114403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":114403},{"type":"v1","addr":"172.21.15.71:6813","nonce":114403}]},"public_addr":"172.21.15.71:6809/114403","cluster_addr":"172.21.15.71:6811/114403","heartbeat_back_addr":"172.21.15.71:6815/114403","heartbeat_front_addr":"172.21.15.71:6813/114403","state":["exists","up"]},{"osd":2,"uuid":"bf7071b3-55cb-48ce-9258-b760e12a9547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":114420},{"type":"v1","addr":"172.21.15.71:6817","nonce":114420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":114420},{"type":"v1","addr":"172.21.15.71:6819","nonce":114420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":114420},{"type":"v1","addr":"172.21.15.71:6823","nonce":114420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":114420},{"type":"v1","addr":"172.21.15.71:6821","nonce":114420}]},"public_addr":"172.21.15.71:6817/114420","cluster_addr":"172.21.15.71:6819/114420","heartbeat_back_addr":"172.21.15.71:6823/114420","heartbeat_front_addr":"172.21.15.71:6821/114420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:28.950378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.250412+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.257797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:34.798 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-10-17T18:10:34.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-17T18:10:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:35.125 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":13,"fsid":"3e63af52-2e83-45bf-8b85-954ab730fd36","created":"2023-10-17T18:10:23.870596+0000","modified":"2023-10-17T18:10:35.052914+0000","last_up_change":"2023-10-17T18:10:31.034329+0000","last_in_change":"2023-10-17T18:10:24.852340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-17T18:10:31.915965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-17T18:10:32.612553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a645805-b941-44c0-9fa2-ba41227af2da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":114402},{"type":"v1","addr":"172.21.15.71:6801","nonce":114402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":114402},{"type":"v1","addr":"172.21.15.71:6803","nonce":114402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":114402},{"type":"v1","addr":"172.21.15.71:6807","nonce":114402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":114402},{"type":"v1","addr":"172.21.15.71:6805","nonce":114402}]},"public_addr":"172.21.15.71:6801/114402","cluster_addr":"172.21.15.71:6803/114402","heartbeat_back_addr":"172.21.15.71:6807/114402","heartbeat_front_addr":"172.21.15.71:6805/114402","state":["exists","up"]},{"osd":1,"uuid":"1a0b39c2-084f-4cd9-a96a-e7836ef8c279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":114403},{"type":"v1","addr":"172.21.15.71:6809","nonce":114403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":114403},{"type":"v1","addr":"172.21.15.71:6811","nonce":114403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":114403},{"type":"v1","addr":"172.21.15.71:6815","nonce":114403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":114403},{"type":"v1","addr":"172.21.15.71:6813","nonce":114403}]},"public_addr":"172.21.15.71:6809/114403","cluster_addr":"172.21.15.71:6811/114403","heartbeat_back_addr":"172.21.15.71:6815/114403","heartbeat_front_addr":"172.21.15.71:6813/114403","state":["exists","up"]},{"osd":2,"uuid":"bf7071b3-55cb-48ce-9258-b760e12a9547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":114420},{"type":"v1","addr":"172.21.15.71:6817","nonce":114420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":114420},{"type":"v1","addr":"172.21.15.71:6819","nonce":114420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":114420},{"type":"v1","addr":"172.21.15.71:6823","nonce":114420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":114420},{"type":"v1","addr":"172.21.15.71:6821","nonce":114420}]},"public_addr":"172.21.15.71:6817/114420","cluster_addr":"172.21.15.71:6819/114420","heartbeat_back_addr":"172.21.15.71:6823/114420","heartbeat_front_addr":"172.21.15.71:6821/114420","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:28.950378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.250412+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-17T18:10:29.257797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-17T18:10:35.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-10-17T18:10:35.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-10-17T18:10:35.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-10-17T18:10:35.306 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-10-17T18:10:35.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-17T18:10:35.309 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2023-10-17T18:10:35.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-17T18:10:35.312 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2023-10-17T18:10:35.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-17T18:10:35.655 INFO:teuthology.orchestra.run.smithi071.stdout:34359738369 2023-10-17T18:10:35.668 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2023-10-17T18:10:35.686 INFO:teuthology.orchestra.run.smithi071.stdout:38654705665 2023-10-17T18:10:35.696 INFO:teuthology.orchestra.run.smithi071.stdout:38654705665 2023-10-17T18:10:35.700 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.1 2023-10-17T18:10:35.710 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.2 2023-10-17T18:10:36.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-17T18:10:36.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-17T18:10:36.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-17T18:10:37.033 INFO:teuthology.orchestra.run.smithi071.stdout:34359738370 2023-10-17T18:10:37.047 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2023-10-17T18:10:37.048 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:37.058 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2023-10-17T18:10:37.068 INFO:teuthology.orchestra.run.smithi071.stdout:38654705666 2023-10-17T18:10:37.072 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.2 2023-10-17T18:10:37.072 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:37.083 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.1 2023-10-17T18:10:37.083 DEBUG:teuthology.parallel:result is None 2023-10-17T18:10:37.084 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-10-17T18:10:37.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-17T18:10:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:37.442 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-10-17T18:10:37.453 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-17T18:10:35.877394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82460,"kb_used_data":1476,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490340,"statfs":{"total":322122547200,"available":322038108160,"internally_reserved":0,"allocated":1511424,"data_stored":1267493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060031+0000","last_change":"2023-10-17T18:10:34.058432+0000","last_active":"2023-10-17T18:10:35.060031+0000","last_peered":"2023-10-17T18:10:35.060031+0000","last_clean":"2023-10-17T18:10:35.060031+0000","last_became_active":"2023-10-17T18:10:34.058194+0000","last_became_peered":"2023-10-17T18:10:34.058194+0000","last_unstale":"2023-10-17T18:10:35.060031+0000","last_undegraded":"2023-10-17T18:10:35.060031+0000","last_fullsized":"2023-10-17T18:10:35.060031+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T04:00:39.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060073+0000","last_change":"2023-10-17T18:10:34.057889+0000","last_active":"2023-10-17T18:10:35.060073+0000","last_peered":"2023-10-17T18:10:35.060073+0000","last_clean":"2023-10-17T18:10:35.060073+0000","last_became_active":"2023-10-17T18:10:34.057710+0000","last_became_peered":"2023-10-17T18:10:34.057710+0000","last_unstale":"2023-10-17T18:10:35.060073+0000","last_undegraded":"2023-10-17T18:10:35.060073+0000","last_fullsized":"2023-10-17T18:10:35.060073+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T21:42:23.647903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060037+0000","last_change":"2023-10-17T18:10:34.057978+0000","last_active":"2023-10-17T18:10:35.060037+0000","last_peered":"2023-10-17T18:10:35.060037+0000","last_clean":"2023-10-17T18:10:35.060037+0000","last_became_active":"2023-10-17T18:10:34.057736+0000","last_became_peered":"2023-10-17T18:10:34.057736+0000","last_unstale":"2023-10-17T18:10:35.060037+0000","last_undegraded":"2023-10-17T18:10:35.060037+0000","last_fullsized":"2023-10-17T18:10:35.060037+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:29:30.008455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059373+0000","last_change":"2023-10-17T18:10:34.056874+0000","last_active":"2023-10-17T18:10:34.059373+0000","last_peered":"2023-10-17T18:10:34.059373+0000","last_clean":"2023-10-17T18:10:34.059373+0000","last_became_active":"2023-10-17T18:10:34.056616+0000","last_became_peered":"2023-10-17T18:10:34.056616+0000","last_unstale":"2023-10-17T18:10:34.059373+0000","last_undegraded":"2023-10-17T18:10:34.059373+0000","last_fullsized":"2023-10-17T18:10:34.059373+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:10:06.127763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059962+0000","last_change":"2023-10-17T18:10:33.052983+0000","last_active":"2023-10-17T18:10:35.059962+0000","last_peered":"2023-10-17T18:10:35.059962+0000","last_clean":"2023-10-17T18:10:35.059962+0000","last_became_active":"2023-10-17T18:10:33.052745+0000","last_became_peered":"2023-10-17T18:10:33.052745+0000","last_unstale":"2023-10-17T18:10:35.059962+0000","last_undegraded":"2023-10-17T18:10:35.059962+0000","last_fullsized":"2023-10-17T18:10:35.059962+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:32.043413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:32.043413+0000","last_clean_scrub_stamp":"2023-10-17T18:10:32.043413+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:10:05.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059961+0000","last_change":"2023-10-17T18:10:34.060096+0000","last_active":"2023-10-17T18:10:35.059961+0000","last_peered":"2023-10-17T18:10:35.059961+0000","last_clean":"2023-10-17T18:10:35.059961+0000","last_became_active":"2023-10-17T18:10:34.059839+0000","last_became_peered":"2023-10-17T18:10:34.059839+0000","last_unstale":"2023-10-17T18:10:35.059961+0000","last_undegraded":"2023-10-17T18:10:35.059961+0000","last_fullsized":"2023-10-17T18:10:35.059961+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:49:27.848742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059427+0000","last_change":"2023-10-17T18:10:34.056937+0000","last_active":"2023-10-17T18:10:34.059427+0000","last_peered":"2023-10-17T18:10:34.059427+0000","last_clean":"2023-10-17T18:10:34.059427+0000","last_became_active":"2023-10-17T18:10:34.056690+0000","last_became_peered":"2023-10-17T18:10:34.056690+0000","last_unstale":"2023-10-17T18:10:34.059427+0000","last_undegraded":"2023-10-17T18:10:34.059427+0000","last_fullsized":"2023-10-17T18:10:34.059427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:38:15.242150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059410+0000","last_change":"2023-10-17T18:10:34.056342+0000","last_active":"2023-10-17T18:10:34.059410+0000","last_peered":"2023-10-17T18:10:34.059410+0000","last_clean":"2023-10-17T18:10:34.059410+0000","last_became_active":"2023-10-17T18:10:34.056077+0000","last_became_peered":"2023-10-17T18:10:34.056077+0000","last_unstale":"2023-10-17T18:10:34.059410+0000","last_undegraded":"2023-10-17T18:10:34.059410+0000","last_fullsized":"2023-10-17T18:10:34.059410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T23:20:31.445320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059988+0000","last_change":"2023-10-17T18:10:34.058729+0000","last_active":"2023-10-17T18:10:35.059988+0000","last_peered":"2023-10-17T18:10:35.059988+0000","last_clean":"2023-10-17T18:10:35.059988+0000","last_became_active":"2023-10-17T18:10:34.058525+0000","last_became_peered":"2023-10-17T18:10:34.058525+0000","last_unstale":"2023-10-17T18:10:35.059988+0000","last_undegraded":"2023-10-17T18:10:35.059988+0000","last_fullsized":"2023-10-17T18:10:35.059988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:47:27.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27760,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829840,"statfs":{"total":107374182400,"available":107345756160,"internally_reserved":0,"allocated":696320,"data_stored":617485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T18:10:37.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-17T18:10:37.768 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:37.771 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-10-17T18:10:37.782 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-17T18:10:35.877394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82460,"kb_used_data":1476,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490340,"statfs":{"total":322122547200,"available":322038108160,"internally_reserved":0,"allocated":1511424,"data_stored":1267493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060031+0000","last_change":"2023-10-17T18:10:34.058432+0000","last_active":"2023-10-17T18:10:35.060031+0000","last_peered":"2023-10-17T18:10:35.060031+0000","last_clean":"2023-10-17T18:10:35.060031+0000","last_became_active":"2023-10-17T18:10:34.058194+0000","last_became_peered":"2023-10-17T18:10:34.058194+0000","last_unstale":"2023-10-17T18:10:35.060031+0000","last_undegraded":"2023-10-17T18:10:35.060031+0000","last_fullsized":"2023-10-17T18:10:35.060031+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T04:00:39.602540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060073+0000","last_change":"2023-10-17T18:10:34.057889+0000","last_active":"2023-10-17T18:10:35.060073+0000","last_peered":"2023-10-17T18:10:35.060073+0000","last_clean":"2023-10-17T18:10:35.060073+0000","last_became_active":"2023-10-17T18:10:34.057710+0000","last_became_peered":"2023-10-17T18:10:34.057710+0000","last_unstale":"2023-10-17T18:10:35.060073+0000","last_undegraded":"2023-10-17T18:10:35.060073+0000","last_fullsized":"2023-10-17T18:10:35.060073+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T21:42:23.647903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.060037+0000","last_change":"2023-10-17T18:10:34.057978+0000","last_active":"2023-10-17T18:10:35.060037+0000","last_peered":"2023-10-17T18:10:35.060037+0000","last_clean":"2023-10-17T18:10:35.060037+0000","last_became_active":"2023-10-17T18:10:34.057736+0000","last_became_peered":"2023-10-17T18:10:34.057736+0000","last_unstale":"2023-10-17T18:10:35.060037+0000","last_undegraded":"2023-10-17T18:10:35.060037+0000","last_fullsized":"2023-10-17T18:10:35.060037+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:29:30.008455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059373+0000","last_change":"2023-10-17T18:10:34.056874+0000","last_active":"2023-10-17T18:10:34.059373+0000","last_peered":"2023-10-17T18:10:34.059373+0000","last_clean":"2023-10-17T18:10:34.059373+0000","last_became_active":"2023-10-17T18:10:34.056616+0000","last_became_peered":"2023-10-17T18:10:34.056616+0000","last_unstale":"2023-10-17T18:10:34.059373+0000","last_undegraded":"2023-10-17T18:10:34.059373+0000","last_fullsized":"2023-10-17T18:10:34.059373+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:10:06.127763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059962+0000","last_change":"2023-10-17T18:10:33.052983+0000","last_active":"2023-10-17T18:10:35.059962+0000","last_peered":"2023-10-17T18:10:35.059962+0000","last_clean":"2023-10-17T18:10:35.059962+0000","last_became_active":"2023-10-17T18:10:33.052745+0000","last_became_peered":"2023-10-17T18:10:33.052745+0000","last_unstale":"2023-10-17T18:10:35.059962+0000","last_undegraded":"2023-10-17T18:10:35.059962+0000","last_fullsized":"2023-10-17T18:10:35.059962+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:32.043413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:32.043413+0000","last_clean_scrub_stamp":"2023-10-17T18:10:32.043413+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T03:10:05.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059961+0000","last_change":"2023-10-17T18:10:34.060096+0000","last_active":"2023-10-17T18:10:35.059961+0000","last_peered":"2023-10-17T18:10:35.059961+0000","last_clean":"2023-10-17T18:10:35.059961+0000","last_became_active":"2023-10-17T18:10:34.059839+0000","last_became_peered":"2023-10-17T18:10:34.059839+0000","last_unstale":"2023-10-17T18:10:35.059961+0000","last_undegraded":"2023-10-17T18:10:35.059961+0000","last_fullsized":"2023-10-17T18:10:35.059961+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:49:27.848742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059427+0000","last_change":"2023-10-17T18:10:34.056937+0000","last_active":"2023-10-17T18:10:34.059427+0000","last_peered":"2023-10-17T18:10:34.059427+0000","last_clean":"2023-10-17T18:10:34.059427+0000","last_became_active":"2023-10-17T18:10:34.056690+0000","last_became_peered":"2023-10-17T18:10:34.056690+0000","last_unstale":"2023-10-17T18:10:34.059427+0000","last_undegraded":"2023-10-17T18:10:34.059427+0000","last_fullsized":"2023-10-17T18:10:34.059427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T02:38:15.242150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-17T18:10:34.059410+0000","last_change":"2023-10-17T18:10:34.056342+0000","last_active":"2023-10-17T18:10:34.059410+0000","last_peered":"2023-10-17T18:10:34.059410+0000","last_clean":"2023-10-17T18:10:34.059410+0000","last_became_active":"2023-10-17T18:10:34.056077+0000","last_became_peered":"2023-10-17T18:10:34.056077+0000","last_unstale":"2023-10-17T18:10:34.059410+0000","last_undegraded":"2023-10-17T18:10:34.059410+0000","last_fullsized":"2023-10-17T18:10:34.059410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T23:20:31.445320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-17T18:10:35.059988+0000","last_change":"2023-10-17T18:10:34.058729+0000","last_active":"2023-10-17T18:10:35.059988+0000","last_peered":"2023-10-17T18:10:35.059988+0000","last_clean":"2023-10-17T18:10:35.059988+0000","last_became_active":"2023-10-17T18:10:34.058525+0000","last_became_peered":"2023-10-17T18:10:34.058525+0000","last_unstale":"2023-10-17T18:10:35.059988+0000","last_undegraded":"2023-10-17T18:10:35.059988+0000","last_fullsized":"2023-10-17T18:10:35.059988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-17T18:10:33.045868+0000","last_clean_scrub_stamp":"2023-10-17T18:10:33.045868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T20:47:27.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27760,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829840,"statfs":{"total":107374182400,"available":107345756160,"internally_reserved":0,"allocated":696320,"data_stored":617485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-17T18:10:37.783 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-10-17T18:10:37.783 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-10-17T18:10:37.784 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-10-17T18:10:37.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-10-17T18:10:38.147 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-10-17T18:10:38.147 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-17T18:10:38.160 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-10-17T18:10:38.161 INFO:teuthology.run_tasks:Running task rgw... 2023-10-17T18:10:38.175 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-10-17T18:10:38.175 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-10-17T18:10:38.176 INFO:tasks.rgw:Creating data pools 2023-10-17T18:10:38.176 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-10-17T18:10:38.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-17T18:10:39.030 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'default.rgw.buckets.data' created 2023-10-17T18:10:39.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-17T18:10:40.052 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-17T18:10:40.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-17T18:10:41.044 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'default.rgw.buckets.index' created 2023-10-17T18:10:41.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-17T18:10:42.048 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-17T18:10:42.068 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-10-17T18:10:42.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-17T18:10:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-17T18:10:42.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-17T18:10:43.036 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-17T18:10:43.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-17T18:10:43.384 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-17T18:10:43.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-17T18:10:44.040 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-17T18:10:44.052 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-10-17T18:10:44.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-17T18:10:44.863 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' created 2023-10-17T18:10:44.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-17T18:10:45.873 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-17T18:10:45.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-17T18:10:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' created 2023-10-17T18:10:46.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-17T18:10:47.885 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-17T18:10:47.901 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-10-17T18:10:47.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-17T18:10:48.228 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-17T18:10:48.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-17T18:10:48.870 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-17T18:10:48.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-17T18:10:49.207 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-17T18:10:49.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-17T18:10:49.871 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-17T18:10:49.882 DEBUG:tasks.rgw:Pools created 2023-10-17T18:10:49.883 INFO:tasks.rgw:Configuring compression type = random 2023-10-17T18:10:49.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-10-17T18:10:49.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-10-17T18:10:49.884 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-10-17T18:10:50.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:50.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:50.016+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.124+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.124+0000 7f77ddbfe780 20 realm 2023-10-17T18:10:51.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.124+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.125+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.125+0000 7f77ddbfe780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-17T18:10:51.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.125+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.125+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.125+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.126+0000 7f77ddbfe780 10 Creating default zonegroup 2023-10-17T18:10:51.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.127+0000 7f77ddbfe780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-17T18:10:51.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.137+0000 7f77ddbfe780 10 WARNING: driver->list_zones() returned r=-2 2023-10-17T18:10:51.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.137+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.137+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.139+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.139+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.139+0000 7f77ddbfe780 10 could not read realm id: (2) No such file or directory 2023-10-17T18:10:51.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.139+0000 7f77ddbfe780 10 WARNING: failed to set zone as default, r=-22 2023-10-17T18:10:51.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.139+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.140+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.142+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.142+0000 7f77ddbfe780 20 rados_obj.operate() r=0 bl.length=874 2023-10-17T18:10:51.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.142+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados_obj.operate() r=0 bl.length=420 2023-10-17T18:10:51.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 10 Using default name default 2023-10-17T18:10:51.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:10:51.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados_obj.operate() r=0 bl.length=874 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 zone default found 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 4 Realm: () 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 4 ZoneGroup: default (8f887301-dafc-4336-aaab-38d9b1b8adb5) 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 4 Zone: default (7913abe9-1a8a-4474-baae-2cc46eec8d6b) 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-17T18:10:51.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 zonegroup default 2023-10-17T18:10:51.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.143+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:51.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.144+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:51.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:51.144+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:53.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:53.127+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:53.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:53.127+0000 7f77ddbfe780 20 rados->read ofs=0 len=0 2023-10-17T18:10:53.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:53.128+0000 7f77ddbfe780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:53.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:53.128+0000 7f77ddbfe780 20 started sync module instance, tier type = 2023-10-17T18:10:53.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:53.128+0000 7f77ddbfe780 20 started zone id=7913abe9-1a8a-4474-baae-2cc46eec8d6b (name=default) with tier type = 2023-10-17T18:10:55.154 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=0 2023-10-17T18:10:55.154 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=1 2023-10-17T18:10:55.154 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=2 2023-10-17T18:10:55.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=3 2023-10-17T18:10:55.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=4 2023-10-17T18:10:55.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=5 2023-10-17T18:10:55.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.154+0000 7f77ddbfe780 20 add_watcher() i=6 2023-10-17T18:10:55.156 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 add_watcher() i=7 2023-10-17T18:10:55.157 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 2 all 8 watchers are set, enabling cache 2023-10-17T18:10:55.157 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-17T18:10:55.157 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-17T18:10:55.157 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:10:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-17T18:10:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-17T18:10:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:10:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-17T18:10:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-17T18:10:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.155+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:10:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.156+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-17T18:10:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.156+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-17T18:10:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.156+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:10:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.156+0000 7f77ddbfe780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-17T18:10:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.156+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-17T18:10:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.157+0000 7f77ddbfe780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:10:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.160+0000 7f77c2b7a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:10:55.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.160+0000 7f77ddbfe780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:10:55.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:55.160+0000 7f77ddbfe780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:10:58.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.131+0000 7f77ddbfe780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-17T18:10:58.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.131+0000 7f77ddbfe780 5 note: GC not initialized 2023-10-17T18:10:58.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.133+0000 7f77ddbfe780 20 init_complete bucket index max shards: 11 2023-10-17T18:10:58.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.133+0000 7f77ddbfe780 10 rgw notify: Started notification manager with: 1 workers 2023-10-17T18:10:58.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.133+0000 7f77bc36d700 20 rgw notify: INFO: next queues processing will happen at: Tue Oct 17 18:11:28 2023 2023-10-17T18:10:58.136 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T18:10:58.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.133+0000 7f77bbb4b700 20 reqs_thread_entry: start 2023-10-17T18:10:58.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.133+0000 7f77be371700 20 reqs_thread_entry: start 2023-10-17T18:10:58.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.155+0000 7f77c2b7a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:10:58.157 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.157+0000 7f77ddbfe780 20 remove_watcher() i=0 2023-10-17T18:10:58.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.157+0000 7f77ddbfe780 2 removed watcher, disabling cache 2023-10-17T18:10:58.158 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.158+0000 7f77ddbfe780 20 remove_watcher() i=1 2023-10-17T18:10:58.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.159+0000 7f77ddbfe780 20 remove_watcher() i=2 2023-10-17T18:10:58.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.160+0000 7f77ddbfe780 20 remove_watcher() i=3 2023-10-17T18:10:58.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.161+0000 7f77ddbfe780 20 remove_watcher() i=4 2023-10-17T18:10:58.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.163+0000 7f77ddbfe780 20 remove_watcher() i=5 2023-10-17T18:10:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.164+0000 7f77ddbfe780 20 remove_watcher() i=6 2023-10-17T18:10:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.165+0000 7f77ddbfe780 20 remove_watcher() i=7 2023-10-17T18:10:58.167 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:10:58.170 DEBUG:tasks.util.rgw: json result: [] 2023-10-17T18:10:58.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:10:58.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:10:58.171 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-17T18:10:58.226 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:58.227 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:58.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.236+0000 7f128af73780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:10:58.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.236+0000 7f128af73780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:10:58.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.240+0000 7f126c6e8700 20 reqs_thread_entry: start 2023-10-17T18:10:58.247 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:10:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "7913abe9-1a8a-4474-baae-2cc46eec8d6b", 2023-10-17T18:10:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default", 2023-10-17T18:10:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "default.rgw.meta:root", 2023-10-17T18:10:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "default.rgw.control", 2023-10-17T18:10:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-17T18:10:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-17T18:10:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "default.rgw.log", 2023-10-17T18:10:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-17T18:10:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-17T18:10:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-17T18:10:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-17T18:10:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-17T18:10:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-17T18:10:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-17T18:10:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-17T18:10:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "default.rgw.otp", 2023-10-17T18:10:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:10:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "", 2023-10-17T18:10:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "" 2023-10-17T18:10:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:10:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:10:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:10:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:10:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:10:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-17T18:10:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:10:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:10:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-17T18:10:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:10:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:10:58.254 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-17T18:10:58.254 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:10:58.255 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:10:58.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.255 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:10:58.256 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "", 2023-10-17T18:10:58.256 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-17T18:10:58.256 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:10:58.257 DEBUG:tasks.util.rgw: json result: {'id': '7913abe9-1a8a-4474-baae-2cc46eec8d6b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-17T18:10:58.258 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-10-17T18:10:58.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-10-17T18:10:58.258 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-10-17T18:10:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:58.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.313+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 20 realm 2023-10-17T18:10:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-17T18:10:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.314+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:10:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 rados_obj.operate() r=0 bl.length=890 2023-10-17T18:10:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.315+0000 7fc2f0327780 20 searching for the correct realm 2023-10-17T18:10:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 RGWRados::pool_iterate: got zonegroup_info.8f887301-dafc-4336-aaab-38d9b1b8adb5 2023-10-17T18:10:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-17T18:10:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-17T18:10:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 RGWRados::pool_iterate: got zone_info.7913abe9-1a8a-4474-baae-2cc46eec8d6b 2023-10-17T18:10:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:10:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados_obj.operate() r=0 bl.length=420 2023-10-17T18:10:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 zone default found 2023-10-17T18:10:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 4 Realm: () 2023-10-17T18:10:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 4 ZoneGroup: default (8f887301-dafc-4336-aaab-38d9b1b8adb5) 2023-10-17T18:10:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 4 Zone: default (7913abe9-1a8a-4474-baae-2cc46eec8d6b) 2023-10-17T18:10:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 zonegroup default 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.323+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.324+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.324+0000 7fc2f0327780 20 rados->read ofs=0 len=0 2023-10-17T18:10:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.324+0000 7fc2f0327780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.324+0000 7fc2f0327780 20 started sync module instance, tier type = 2023-10-17T18:10:58.329 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.324+0000 7fc2f0327780 20 started zone id=7913abe9-1a8a-4474-baae-2cc46eec8d6b (name=default) with tier type = 2023-10-17T18:10:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=0 2023-10-17T18:10:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=1 2023-10-17T18:10:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=2 2023-10-17T18:10:58.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=3 2023-10-17T18:10:58.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=4 2023-10-17T18:10:58.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=5 2023-10-17T18:10:58.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.332+0000 7fc2f0327780 20 add_watcher() i=6 2023-10-17T18:10:58.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.333+0000 7fc2f0327780 20 add_watcher() i=7 2023-10-17T18:10:58.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.333+0000 7fc2f0327780 2 all 8 watchers are set, enabling cache 2023-10-17T18:10:58.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.334+0000 7fc2d52a3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:10:58.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.334+0000 7fc2f0327780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:10:58.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.334+0000 7fc2f0327780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:10:58.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.338+0000 7fc2f0327780 5 note: GC not initialized 2023-10-17T18:10:58.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.339+0000 7fc2d0a9a700 20 reqs_thread_entry: start 2023-10-17T18:10:58.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.339+0000 7fc2f0327780 20 init_complete bucket index max shards: 11 2023-10-17T18:10:58.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.339+0000 7fc2f0327780 10 rgw notify: Started notification manager with: 1 workers 2023-10-17T18:10:58.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.339+0000 7fc2cea96700 20 rgw notify: INFO: next queues processing will happen at: Tue Oct 17 18:11:28 2023 2023-10-17T18:10:58.340 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-17T18:10:58.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.339+0000 7fc2ce274700 20 reqs_thread_entry: start 2023-10-17T18:10:58.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.347+0000 7fc2d52a3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:10:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.348+0000 7fc2f0327780 20 remove_watcher() i=0 2023-10-17T18:10:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.348+0000 7fc2f0327780 2 removed watcher, disabling cache 2023-10-17T18:10:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.349+0000 7fc2f0327780 20 remove_watcher() i=1 2023-10-17T18:10:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.350+0000 7fc2f0327780 20 remove_watcher() i=2 2023-10-17T18:10:58.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.351+0000 7fc2f0327780 20 remove_watcher() i=3 2023-10-17T18:10:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.352+0000 7fc2f0327780 20 remove_watcher() i=4 2023-10-17T18:10:58.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.353+0000 7fc2f0327780 20 remove_watcher() i=5 2023-10-17T18:10:58.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.353+0000 7fc2f0327780 20 remove_watcher() i=6 2023-10-17T18:10:58.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.354+0000 7fc2f0327780 20 remove_watcher() i=7 2023-10-17T18:10:58.357 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:10:58.359 DEBUG:tasks.util.rgw: json result: [] 2023-10-17T18:10:58.359 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:10:58.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:10:58.360 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-17T18:10:58.411 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:58.411 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:58.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.420+0000 7ff94800d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:10:58.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.420+0000 7ff94800d780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:10:58.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:10:58.423+0000 7ff929782700 20 reqs_thread_entry: start 2023-10-17T18:10:58.429 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:10:58.429 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "7913abe9-1a8a-4474-baae-2cc46eec8d6b", 2023-10-17T18:10:58.429 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default", 2023-10-17T18:10:58.430 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "default.rgw.meta:root", 2023-10-17T18:10:58.430 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "default.rgw.control", 2023-10-17T18:10:58.430 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-17T18:10:58.430 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "default.rgw.log", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-17T18:10:58.431 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-17T18:10:58.432 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-17T18:10:58.432 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-17T18:10:58.432 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-17T18:10:58.432 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "default.rgw.otp", 2023-10-17T18:10:58.432 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:10:58.433 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "", 2023-10-17T18:10:58.433 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "" 2023-10-17T18:10:58.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:10:58.433 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:10:58.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:10:58.434 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:10:58.434 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:10:58.434 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-17T18:10:58.434 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:10:58.434 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:10:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-17T18:10:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:10:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:10:58.435 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:10:58.436 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "", 2023-10-17T18:10:58.437 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-17T18:10:58.437 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:10:58.438 DEBUG:tasks.util.rgw: json result: {'id': '7913abe9-1a8a-4474-baae-2cc46eec8d6b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-17T18:10:58.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-10-17T18:10:58.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-10-17T18:10:58.439 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-10-17T18:10:58.578 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:10:58.579 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:10:58.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:58.587+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.955+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.955+0000 7fd757f15780 20 realm 2023-10-17T18:10:59.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.955+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.956+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.956+0000 7fd757f15780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-17T18:10:59.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.956+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.956+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.956+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.957+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.957+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.958+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.958+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.959+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.959+0000 7fd757f15780 10 Creating default zonegroup 2023-10-17T18:10:59.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.960+0000 7fd757f15780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-17T18:10:59.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.970+0000 7fd757f15780 10 WARNING: driver->list_zones() returned r=-2 2023-10-17T18:10:59.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.970+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.971+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 10 could not read realm id: (2) No such file or directory 2023-10-17T18:10:59.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 10 WARNING: failed to set zone as default, r=-22 2023-10-17T18:10:59.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.975+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.977+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.977+0000 7fd757f15780 20 rados_obj.operate() r=0 bl.length=874 2023-10-17T18:10:59.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.978+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.981+0000 7fd757f15780 20 rados_obj.operate() r=0 bl.length=420 2023-10-17T18:10:59.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.981+0000 7fd757f15780 10 Using default name default 2023-10-17T18:10:59.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.981+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.981+0000 7fd757f15780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:10:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.981+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 rados_obj.operate() r=0 bl.length=874 2023-10-17T18:10:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 zone default found 2023-10-17T18:10:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 4 Realm: () 2023-10-17T18:10:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 4 ZoneGroup: default (dd99dcd1-7383-4898-b7ed-84ee026fbcae) 2023-10-17T18:10:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 4 Zone: default (7c34d3e6-fe52-4bb7-8e2c-ce044263d991) 2023-10-17T18:10:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-17T18:10:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 zonegroup default 2023-10-17T18:10:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:10:59.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:10:59.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:10:59.982+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:11:01.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:01.972+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:01.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:01.972+0000 7fd757f15780 20 rados->read ofs=0 len=0 2023-10-17T18:11:01.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:01.973+0000 7fd757f15780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:01.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:01.973+0000 7fd757f15780 20 started sync module instance, tier type = 2023-10-17T18:11:01.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:01.973+0000 7fd757f15780 20 started zone id=7c34d3e6-fe52-4bb7-8e2c-ce044263d991 (name=default) with tier type = 2023-10-17T18:11:03.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=0 2023-10-17T18:11:04.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=1 2023-10-17T18:11:04.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=2 2023-10-17T18:11:04.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=3 2023-10-17T18:11:04.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=4 2023-10-17T18:11:04.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=5 2023-10-17T18:11:04.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.998+0000 7fd757f15780 20 add_watcher() i=6 2023-10-17T18:11:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 add_watcher() i=7 2023-10-17T18:11:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 2 all 8 watchers are set, enabling cache 2023-10-17T18:11:04.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-17T18:11:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-17T18:11:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:11:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-17T18:11:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-17T18:11:04.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:03.999+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:11:04.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-17T18:11:04.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-17T18:11:04.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:11:04.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-17T18:11:04.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-17T18:11:04.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:11:04.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-17T18:11:04.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.000+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-17T18:11:04.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.001+0000 7fd757f15780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-17T18:11:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.006+0000 7fd73ce91700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:11:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.006+0000 7fd757f15780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:11:04.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:04.006+0000 7fd757f15780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:11:06.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.979+0000 7fd757f15780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-17T18:11:06.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.979+0000 7fd757f15780 5 note: GC not initialized 2023-10-17T18:11:06.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.981+0000 7fd738688700 20 reqs_thread_entry: start 2023-10-17T18:11:06.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.981+0000 7fd757f15780 20 init_complete bucket index max shards: 11 2023-10-17T18:11:06.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.981+0000 7fd757f15780 10 rgw notify: Started notification manager with: 1 workers 2023-10-17T18:11:06.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.981+0000 7fd736684700 20 rgw notify: INFO: next queues processing will happen at: Tue Oct 17 18:11:37 2023 2023-10-17T18:11:06.983 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-10-17T18:11:06.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:06.981+0000 7fd735e62700 20 reqs_thread_entry: start 2023-10-17T18:11:07.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.000+0000 7fd73ce91700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:11:07.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.005+0000 7fd757f15780 20 remove_watcher() i=0 2023-10-17T18:11:07.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.005+0000 7fd757f15780 2 removed watcher, disabling cache 2023-10-17T18:11:07.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.007+0000 7fd757f15780 20 remove_watcher() i=1 2023-10-17T18:11:07.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.008+0000 7fd757f15780 20 remove_watcher() i=2 2023-10-17T18:11:07.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.011+0000 7fd757f15780 20 remove_watcher() i=3 2023-10-17T18:11:07.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.014+0000 7fd757f15780 20 remove_watcher() i=4 2023-10-17T18:11:07.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.015+0000 7fd757f15780 20 remove_watcher() i=5 2023-10-17T18:11:07.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.017+0000 7fd757f15780 20 remove_watcher() i=6 2023-10-17T18:11:07.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.018+0000 7fd757f15780 20 remove_watcher() i=7 2023-10-17T18:11:07.022 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-10-17T18:11:07.024 DEBUG:tasks.util.rgw: json result: [] 2023-10-17T18:11:07.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:11:07.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:11:07.025 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-17T18:11:07.078 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:07.079 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:07.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.086+0000 7f9a3d9e7780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:11:07.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.086+0000 7f9a3d9e7780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:11:07.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.090+0000 7f9a1f15c700 20 reqs_thread_entry: start 2023-10-17T18:11:07.097 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7c34d3e6-fe52-4bb7-8e2c-ce044263d991", 2023-10-17T18:11:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2023-10-17T18:11:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2023-10-17T18:11:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2023-10-17T18:11:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-17T18:11:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-17T18:11:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2023-10-17T18:11:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-17T18:11:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-17T18:11:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-17T18:11:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-17T18:11:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-17T18:11:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-17T18:11:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-17T18:11:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-17T18:11:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2023-10-17T18:11:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:11:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2023-10-17T18:11:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:11:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-17T18:11:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:11:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:11:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-17T18:11:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-10-17T18:11:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-17T18:11:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:11:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:11:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2023-10-17T18:11:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-17T18:11:07.105 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:07.106 DEBUG:tasks.util.rgw: json result: {'id': '7c34d3e6-fe52-4bb7-8e2c-ce044263d991', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-17T18:11:07.107 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-10-17T18:11:07.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-10-17T18:11:07.107 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-10-17T18:11:07.151 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:07.152 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:07.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.159+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.159+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.159+0000 7f71dc881780 20 realm 2023-10-17T18:11:07.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.159+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-17T18:11:07.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:11:07.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.160+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.161+0000 7f71dc881780 20 rados_obj.operate() r=0 bl.length=890 2023-10-17T18:11:07.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.161+0000 7f71dc881780 20 searching for the correct realm 2023-10-17T18:11:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 RGWRados::pool_iterate: got zone_info.7c34d3e6-fe52-4bb7-8e2c-ce044263d991 2023-10-17T18:11:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 RGWRados::pool_iterate: got zonegroup_info.dd99dcd1-7383-4898-b7ed-84ee026fbcae 2023-10-17T18:11:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-17T18:11:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-17T18:11:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.167+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados_obj.operate() r=0 bl.length=46 2023-10-17T18:11:07.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados_obj.operate() r=0 bl.length=420 2023-10-17T18:11:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 zone default found 2023-10-17T18:11:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 4 Realm: () 2023-10-17T18:11:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 4 ZoneGroup: default (dd99dcd1-7383-4898-b7ed-84ee026fbcae) 2023-10-17T18:11:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 4 Zone: default (7c34d3e6-fe52-4bb7-8e2c-ce044263d991) 2023-10-17T18:11:07.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-17T18:11:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 zonegroup default 2023-10-17T18:11:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados->read ofs=0 len=0 2023-10-17T18:11:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-17T18:11:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 started sync module instance, tier type = 2023-10-17T18:11:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.168+0000 7f71dc881780 20 started zone id=7c34d3e6-fe52-4bb7-8e2c-ce044263d991 (name=default) with tier type = 2023-10-17T18:11:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=0 2023-10-17T18:11:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=1 2023-10-17T18:11:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=2 2023-10-17T18:11:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=3 2023-10-17T18:11:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=4 2023-10-17T18:11:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=5 2023-10-17T18:11:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.197+0000 7f71dc881780 20 add_watcher() i=6 2023-10-17T18:11:07.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.200+0000 7f71dc881780 20 add_watcher() i=7 2023-10-17T18:11:07.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.200+0000 7f71dc881780 2 all 8 watchers are set, enabling cache 2023-10-17T18:11:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.201+0000 7f71c17fd700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:11:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.201+0000 7f71dc881780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:11:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.201+0000 7f71dc881780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:11:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.205+0000 7f71dc881780 5 note: GC not initialized 2023-10-17T18:11:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.206+0000 7f71bcff4700 20 reqs_thread_entry: start 2023-10-17T18:11:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.206+0000 7f71dc881780 20 init_complete bucket index max shards: 11 2023-10-17T18:11:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.206+0000 7f71dc881780 10 rgw notify: Started notification manager with: 1 workers 2023-10-17T18:11:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.206+0000 7f71baff0700 20 rgw notify: INFO: next queues processing will happen at: Tue Oct 17 18:11:37 2023 2023-10-17T18:11:07.208 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-10-17T18:11:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.206+0000 7f71ba7ce700 20 reqs_thread_entry: start 2023-10-17T18:11:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.213+0000 7f71c17fd700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-17T18:11:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.215+0000 7f71dc881780 20 remove_watcher() i=0 2023-10-17T18:11:07.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.215+0000 7f71dc881780 2 removed watcher, disabling cache 2023-10-17T18:11:07.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.215+0000 7f71dc881780 20 remove_watcher() i=1 2023-10-17T18:11:07.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.216+0000 7f71dc881780 20 remove_watcher() i=2 2023-10-17T18:11:07.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.217+0000 7f71dc881780 20 remove_watcher() i=3 2023-10-17T18:11:07.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.218+0000 7f71dc881780 20 remove_watcher() i=4 2023-10-17T18:11:07.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.219+0000 7f71dc881780 20 remove_watcher() i=5 2023-10-17T18:11:07.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.219+0000 7f71dc881780 20 remove_watcher() i=6 2023-10-17T18:11:07.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.220+0000 7f71dc881780 20 remove_watcher() i=7 2023-10-17T18:11:07.223 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2023-10-17T18:11:07.225 DEBUG:tasks.util.rgw: json result: [] 2023-10-17T18:11:07.225 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:11:07.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-17T18:11:07.226 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-17T18:11:07.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:07.316 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:07.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.323+0000 7f1cea935780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-17T18:11:07.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.323+0000 7f1cea935780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-17T18:11:07.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:07.326+0000 7f1ccc0aa700 20 reqs_thread_entry: start 2023-10-17T18:11:07.333 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "7c34d3e6-fe52-4bb7-8e2c-ce044263d991", 2023-10-17T18:11:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2023-10-17T18:11:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2023-10-17T18:11:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2023-10-17T18:11:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-17T18:11:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-17T18:11:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2023-10-17T18:11:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-17T18:11:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2023-10-17T18:11:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-17T18:11:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:11:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:11:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-17T18:11:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-10-17T18:11:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2023-10-17T18:11:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-17T18:11:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:07.341 DEBUG:tasks.util.rgw: json result: {'id': '7c34d3e6-fe52-4bb7-8e2c-ce044263d991', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-17T18:11:07.342 INFO:tasks.rgw:Starting rgw... 2023-10-17T18:11:07.342 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-10-17T18:11:07.342 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-17T18:11:07.343 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-17T18:11:07.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-17T18:11:07.346 INFO:tasks.rgw.c1.client.0:Started 2023-10-17T18:11:07.346 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-10-17T18:11:07.347 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-17T18:11:07.347 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T18:11:07.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T18:11:07.349 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T18:11:07.350 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-10-17T18:11:07.351 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-17T18:11:07.351 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-17T18:11:07.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-17T18:11:07.355 INFO:tasks.rgw.c2.client.0:Started 2023-10-17T18:11:07.355 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-10-17T18:11:07.355 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-17T18:11:07.356 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-17T18:11:07.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-17T18:11:07.358 INFO:tasks.rgw.c2.client.1:Started 2023-10-17T18:11:07.358 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi049.front.sepia.ceph.com:8000/ 2023-10-17T18:11:07.359 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8000/ 2023-10-17T18:11:07.435 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:07.435 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:11:07.438 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8000: Connection refused 2023-10-17T18:11:08.439 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8000/ 2023-10-17T18:11:08.464 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:08.464 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:08.466 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:08.467 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:11:08.467 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi049.front.sepia.ceph.com:8001/ 2023-10-17T18:11:08.468 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001/ 2023-10-17T18:11:08.532 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:08.532 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:08.532 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:08.533 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:11:08.533 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8000/ 2023-10-17T18:11:08.534 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000/ 2023-10-17T18:11:08.566 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:08.566 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:08.567 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:08.568 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-10-17T18:11:08.568 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:8001/ 2023-10-17T18:11:08.569 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001/ 2023-10-17T18:11:08.631 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:08.631 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:08.632 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:08.632 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-10-17T18:11:08.633 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-10-17T18:11:10.276 INFO:tasks.rgw_multisite:creating realm.. 2023-10-17T18:11:10.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.278 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:10.332 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:10.333 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:10.361 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "test-realm", 2023-10-17T18:11:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "current_period": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:11:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 1 2023-10-17T18:11:10.362 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:10.363 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-10-17T18:11:10.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.364 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:10.415 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:10.415 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:10.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:11:10.429+0000 7fc138ff8780 -1 old format 2023-10-17T18:11:10.437 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:11:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:11:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:11:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:11:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "", 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [], 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:11:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:11:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:11:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [] 2023-10-17T18:11:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:11:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:11:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:11:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:11:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:10.442 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:10.443 INFO:tasks.rgw_multisite:creating master zone.. 2023-10-17T18:11:10.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi049.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi049.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:10.445 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --endpoints http://smithi049.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:10.556 INFO:teuthology.orchestra.run.smithi049.stderr:NOTICE: set zone's realm_id=19b3e86c-cc39-4be9-99c4-971ddfc4a8c5 2023-10-17T18:11:10.571 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:11:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-17T18:11:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a1.rgw.control", 2023-10-17T18:11:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-17T18:11:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-17T18:11:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a1.rgw.log", 2023-10-17T18:11:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-17T18:11:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-17T18:11:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-17T18:11:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-17T18:11:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-17T18:11:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-17T18:11:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-17T18:11:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-17T18:11:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a1.rgw.otp", 2023-10-17T18:11:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:11:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:11:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:11:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:11:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-17T18:11:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:11:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:11:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a1.rgw.buckets.data" 2023-10-17T18:11:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-17T18:11:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:11:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:11:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-17T18:11:10.580 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:10.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-10-17T18:11:11.161 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'a1.rgw.buckets.data' created 2023-10-17T18:11:11.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-10-17T18:11:12.185 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-10-17T18:11:12.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:12.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:12.201 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:12.258 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:12.278 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:11:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-17T18:11:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a1.rgw.control", 2023-10-17T18:11:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-17T18:11:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a1.rgw.log", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-17T18:11:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-17T18:11:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-17T18:11:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-17T18:11:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-17T18:11:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a1.rgw.otp", 2023-10-17T18:11:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:11:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:11:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:11:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:11:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-17T18:11:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:11:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:11:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a1.rgw.buckets.data", 2023-10-17T18:11:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:11:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-17T18:11:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:11:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:11:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-17T18:11:12.286 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:12.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:12.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:12.288 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:12.338 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:12.339 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:12.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:11:12.350+0000 7faee40f5780 0 period (291121e4-630c-452a-bf9e-7afaaae7cbe0 does not have zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 configured 2023-10-17T18:11:19.233 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:19.233 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:19.233 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 1, 2023-10-17T18:11:19.233 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:11:19.234 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:11:19.234 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:11:19.234 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:19.234 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:11:19.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:19.235 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:11:19.235 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:11:19.235 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:11:19.236 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:11:19.236 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:19.236 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:19.236 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:19.236 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:11:19.237 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:11:19.237 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:19.237 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:11:19.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:19.237 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:19.238 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:11:19.238 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:19.238 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:19.238 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:19.238 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:11:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": false, 2023-10-17T18:11:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:11:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:11:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:11:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:11:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:11:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:11:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:11:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:11:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:11:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:11:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:11:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:11:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:11:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:11:19.244 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:11:19.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.244 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:11:19.244 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:19.245 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:19.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:19.245 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:19.245 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:11:19.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.246 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:11:19.247 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:19.247 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:11:19.247 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:11:19.247 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:19.247 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:19.248 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:19.248 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:19.248 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:19.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.249 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:11:19.249 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:19.249 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:19.249 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:19.250 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:19.250 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:19.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.250 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:11:19.250 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.251 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:11:19.252 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:19.252 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:19.252 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:19.252 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:19.253 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:19.254 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:19.254 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:19.254 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:19.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:19.255 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:19.255 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:11:19.255 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:11:19.255 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:19.256 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-17T18:11:19.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:11:19.258 INFO:tasks.rgw.c1.client.0.smithi049.stdout:2023-10-17T18:11:19.256+0000 7f65843d9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 119700) UID: 0 2023-10-17T18:11:19.258 INFO:tasks.rgw.c1.client.0.smithi049.stdout:2023-10-17T18:11:19.256+0000 7f65886427c0 -1 shutting down 2023-10-17T18:11:21.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-17T18:11:25.359 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-17T18:11:25.359 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-17T18:11:25.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-17T18:11:25.363 INFO:tasks.rgw.c1.client.0:Started 2023-10-17T18:11:25.364 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8000 2023-10-17T18:11:25.390 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:25.391 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:11:25.392 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8000: Connection refused 2023-10-17T18:11:26.393 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8000 2023-10-17T18:11:26.420 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:26.420 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:26.420 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:26.421 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:11:26.422 INFO:tasks.rgw_multisite:creating admin user.. 2023-10-17T18:11:26.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:26.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:26.423 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:26.525 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:26.525 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:26.559 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:26.559 INFO:teuthology.orchestra.run.smithi049.stdout: "user_id": "realm-admin", 2023-10-17T18:11:26.560 INFO:teuthology.orchestra.run.smithi049.stdout: "display_name": "Realm Admin", 2023-10-17T18:11:26.560 INFO:teuthology.orchestra.run.smithi049.stdout: "email": "", 2023-10-17T18:11:26.560 INFO:teuthology.orchestra.run.smithi049.stdout: "suspended": 0, 2023-10-17T18:11:26.560 INFO:teuthology.orchestra.run.smithi049.stdout: "max_buckets": 1000, 2023-10-17T18:11:26.560 INFO:teuthology.orchestra.run.smithi049.stdout: "subusers": [], 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: "keys": [ 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "realm-admin", 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:26.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:26.562 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:26.562 INFO:teuthology.orchestra.run.smithi049.stdout: "swift_keys": [], 2023-10-17T18:11:26.562 INFO:teuthology.orchestra.run.smithi049.stdout: "caps": [], 2023-10-17T18:11:26.562 INFO:teuthology.orchestra.run.smithi049.stdout: "op_mask": "read, write, delete", 2023-10-17T18:11:26.562 INFO:teuthology.orchestra.run.smithi049.stdout: "system": true, 2023-10-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "", 2023-10-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi049.stdout: "default_storage_class": "", 2023-10-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_tags": [], 2023-10-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:11:26.565 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:26.565 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:26.565 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:26.565 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:26.566 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:26.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:26.566 INFO:teuthology.orchestra.run.smithi049.stdout: "temp_url_keys": [], 2023-10-17T18:11:26.566 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "rgw", 2023-10-17T18:11:26.566 INFO:teuthology.orchestra.run.smithi049.stdout: "mfa_ids": [] 2023-10-17T18:11:26.567 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:26.567 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:11:26.571 INFO:tasks.rgw_multisite:creating zone.. 2023-10-17T18:11:26.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi049.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:26.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi049.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:26.572 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --endpoints http://smithi049.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:26.628 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:26.629 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:26.643 INFO:teuthology.orchestra.run.smithi049.stderr:NOTICE: set zone's realm_id=19b3e86c-cc39-4be9-99c4-971ddfc4a8c5 2023-10-17T18:11:26.657 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:26.658 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:26.658 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:11:26.658 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:11:26.658 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:11:26.658 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:11:26.659 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:11:26.659 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:11:26.659 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:11:26.659 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:11:26.659 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:11:26.660 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:11:26.660 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:11:26.660 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:11:26.660 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:11:26.661 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:11:26.661 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:11:26.661 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:11:26.661 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:26.661 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:26.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:26.662 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:11:26.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:26.662 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:11:26.662 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:11:26.663 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:11:26.663 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:11:26.663 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:11:26.663 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data" 2023-10-17T18:11:26.663 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:11:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:11:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:11:26.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:26.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:26.665 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:26.665 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:26.665 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:11:26.665 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:26.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-10-17T18:11:27.200 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'a2.rgw.buckets.data' created 2023-10-17T18:11:27.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-10-17T18:11:27.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-17T18:11:28.223 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-10-17T18:11:28.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:28.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:28.241 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:28.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:28.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:28.314 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:28.314 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:28.314 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:11:28.315 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:11:28.315 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:11:28.315 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:11:28.315 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:11:28.315 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:11:28.316 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:11:28.316 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:11:28.316 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:11:28.316 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:11:28.316 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:11:28.317 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:11:28.317 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:11:28.317 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:11:28.317 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:11:28.317 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:11:28.318 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:28.318 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:28.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:28.318 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:11:28.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:28.319 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:11:28.319 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:11:28.319 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:11:28.319 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:11:28.319 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:11:28.320 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:28.321 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:11:28.322 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:28.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:28.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:28.324 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:28.373 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:28.374 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:11:28.384+0000 7f8acb2ee780 0 period (cfc9d53d-2765-4da9-b7a8-37407f517c55 does not have zone 8f644c5a-db9c-4565-b32b-e3a6790e3bcc configured 2023-10-17T18:11:35.261 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:11:35.281 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:11:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2, 2023-10-17T18:11:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:11:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:11:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:11:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:11:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:11:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:11:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:11:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:11:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:11:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:11:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:11:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:11:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:11:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:11:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:11:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:11:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:11:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:11:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:11:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:11:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:11:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:11:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:11:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:11:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:11:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:11:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:11:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:11:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:11:35.294 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:11:35.294 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:11:35.294 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:35.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:35.294 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.295 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:11:35.295 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:35.295 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:11:35.295 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:11:35.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.296 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:11:35.296 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:11:35.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:35.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:35.296 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:11:35.297 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:11:35.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.297 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:35.297 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:11:35.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:11:35.298 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:35.298 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:11:35.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:35.298 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:11:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:11:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:11:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:11:35.300 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:35.300 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:35.300 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:35.300 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:35.300 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:35.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.301 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:11:35.301 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:11:35.301 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:11:35.302 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:11:35.302 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:11:35.302 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:11:35.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.302 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:11:35.303 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:35.303 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:35.303 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:35.303 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:35.303 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:35.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.304 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:11:35.304 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:35.304 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:35.304 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:35.305 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:35.306 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:35.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.306 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:11:35.306 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:11:35.307 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:11:35.307 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:11:35.307 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:11:35.307 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:11:35.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:11:35.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:11:35.308 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:35.308 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:11:35.308 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:11:35.309 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:11:35.309 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T18:11:35.309 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:11:35.310 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:11:35.310+0000 7fe72f146700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 119704) UID: 0 2023-10-17T18:11:35.310 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:11:35.310+0000 7fe7333af7c0 -1 shutting down 2023-10-17T18:11:38.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T18:11:41.411 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T18:11:41.411 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T18:11:41.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T18:11:41.415 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T18:11:41.415 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:11:41.441 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:41.442 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:11:41.445 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T18:11:42.446 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:11:42.472 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:42.472 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:42.473 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:42.474 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:11:42.474 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-17T18:11:42.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.475 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi049.front.sepia.ceph.com:8000 --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:42.534 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:42.535 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:42.580 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:42.580 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:42.580 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "test-realm", 2023-10-17T18:11:42.580 INFO:teuthology.orchestra.run.smithi071.stdout: "current_period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:42.580 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 2 2023-10-17T18:11:42.581 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:42.582 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-10-17T18:11:42.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.583 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:42.631 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:42.632 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:42.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:42.645+0000 7fa2e7575780 -1 old format 2023-10-17T18:11:42.655 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:42.655 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-10-17T18:11:42.655 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-10-17T18:11:42.656 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-10-17T18:11:42.656 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-10-17T18:11:42.656 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:42.656 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:11:42.656 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:42.657 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-10-17T18:11:42.657 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-10-17T18:11:42.657 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "", 2023-10-17T18:11:42.657 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [], 2023-10-17T18:11:42.657 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-10-17T18:11:42.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:42.658 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-10-17T18:11:42.658 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-10-17T18:11:42.658 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [] 2023-10-17T18:11:42.658 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:42.659 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:42.659 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-10-17T18:11:42.659 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:42.659 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-10-17T18:11:42.659 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-10-17T18:11:42.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:42.660 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-10-17T18:11:42.660 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:42.660 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:42.660 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:42.661 INFO:tasks.rgw_multisite:creating zone.. 2023-10-17T18:11:42.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:42.663 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --endpoints http://smithi071.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:42.771 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=19b3e86c-cc39-4be9-99c4-971ddfc4a8c5 2023-10-17T18:11:42.787 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:42.788 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:11:42.788 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-10-17T18:11:42.788 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-17T18:11:42.788 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "b1.rgw.control", 2023-10-17T18:11:42.788 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "b1.rgw.log", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-17T18:11:42.789 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-17T18:11:42.790 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-17T18:11:42.790 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-17T18:11:42.790 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-17T18:11:42.790 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-17T18:11:42.791 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "b1.rgw.otp", 2023-10-17T18:11:42.791 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:11:42.791 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:42.791 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:42.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-17T18:11:42.792 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:11:42.793 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:11:42.793 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "b1.rgw.buckets.data" 2023-10-17T18:11:42.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:42.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:42.793 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-17T18:11:42.794 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:11:42.794 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:11:42.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:42.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:42.795 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:42.795 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:42.795 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-17T18:11:42.795 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:42.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-10-17T18:11:43.959 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'b1.rgw.buckets.data' created 2023-10-17T18:11:43.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-10-17T18:11:44.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T18:11:44.977 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-10-17T18:11:44.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:44.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:44.994 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:45.050 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:45.051 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:45.068 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:45.069 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:11:45.069 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-10-17T18:11:45.069 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-17T18:11:45.069 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "b1.rgw.control", 2023-10-17T18:11:45.070 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-17T18:11:45.070 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-17T18:11:45.070 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "b1.rgw.log", 2023-10-17T18:11:45.070 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-17T18:11:45.070 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-17T18:11:45.071 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-17T18:11:45.071 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-17T18:11:45.071 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-17T18:11:45.071 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-17T18:11:45.071 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-17T18:11:45.072 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-17T18:11:45.072 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "b1.rgw.otp", 2023-10-17T18:11:45.072 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:11:45.072 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:45.072 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:45.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:45.073 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:11:45.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:45.073 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:11:45.073 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:11:45.074 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-17T18:11:45.074 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:11:45.074 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:11:45.074 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "b1.rgw.buckets.data", 2023-10-17T18:11:45.074 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:11:45.075 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:45.076 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:45.076 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:45.076 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:45.076 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-17T18:11:45.076 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:45.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:45.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:45.078 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:45.126 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:45.127 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:45.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:11:45.138+0000 7ffb32dab780 0 period (cfc9d53d-2765-4da9-b7a8-37407f517c55 does not have zone ad5bf4b6-5c62-4abf-bddb-0259cafe1aab configured 2023-10-17T18:11:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:11:52.069 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:52.070 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:52.070 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 3, 2023-10-17T18:11:52.070 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:11:52.070 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-10-17T18:11:52.071 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-10-17T18:11:52.072 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-10-17T18:11:52.072 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-10-17T18:11:52.072 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:52.072 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:52.072 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.073 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-10-17T18:11:52.073 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-10-17T18:11:52.073 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:52.073 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-10-17T18:11:52.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.074 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:11:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:11:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:11:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:11:52.075 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:11:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:11:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:11:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:11:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:11:52.076 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-10-17T18:11:52.077 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:11:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:11:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:11:52.078 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:52.079 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:11:52.079 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:11:52.079 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:11:52.079 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:11:52.079 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:11:52.080 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:11:52.080 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:11:52.080 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:52.080 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.081 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.081 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-10-17T18:11:52.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.081 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-10-17T18:11:52.081 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-10-17T18:11:52.082 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-10-17T18:11:52.082 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-10-17T18:11:52.082 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.082 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.082 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.083 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-10-17T18:11:52.083 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:52.083 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-10-17T18:11:52.083 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-10-17T18:11:52.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.084 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-10-17T18:11:52.084 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:52.084 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.085 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-10-17T18:11:52.085 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-10-17T18:11:52.085 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-10-17T18:11:52.085 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-10-17T18:11:52.086 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:11:52.087 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:11:52.088 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.088 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:11:52.088 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": false, 2023-10-17T18:11:52.088 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:11:52.088 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:11:52.089 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:11:52.089 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:11:52.089 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:11:52.089 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:11:52.089 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:11:52.090 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:11:52.091 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:52.091 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.092 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.092 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-10-17T18:11:52.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.092 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-10-17T18:11:52.092 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-10-17T18:11:52.093 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-10-17T18:11:52.093 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-10-17T18:11:52.093 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.093 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.093 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.094 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-10-17T18:11:52.094 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:52.094 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-10-17T18:11:52.094 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-10-17T18:11:52.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.095 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-10-17T18:11:52.095 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:11:52.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.095 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:52.096 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-10-17T18:11:52.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.096 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:52.096 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1846350003 2023-10-17T18:11:52.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:11:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3451171648 2023-10-17T18:11:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:52.098 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:11:52.098 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1989104912 2023-10-17T18:11:52.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.098 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:11:52.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.099 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-10-17T18:11:52.099 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:11:52.099 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-10-17T18:11:52.099 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-10-17T18:11:52.099 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-10-17T18:11:52.100 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-10-17T18:11:52.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-10-17T18:11:52.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-10-17T18:11:52.100 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-10-17T18:11:52.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-10-17T18:11:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-10-17T18:11:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-10-17T18:11:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-10-17T18:11:52.101 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-10-17T18:11:52.102 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-10-17T18:11:52.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.102 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-10-17T18:11:52.102 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:11:52.102 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:11:52.103 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:11:52.103 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:11:52.103 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:11:52.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.103 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-10-17T18:11:52.104 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:11:52.104 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:11:52.104 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:11:52.104 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:11:52.104 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:11:52.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.105 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-10-17T18:11:52.105 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:11:52.105 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:11:52.105 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:52.106 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-10-17T18:11:52.107 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:11:52.107 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:52.107 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-17T18:11:52.108 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:11:52.109 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:11:52.107+0000 7fee70d0f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 119894) UID: 0 2023-10-17T18:11:52.109 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:11:52.107+0000 7fee74f787c0 -1 shutting down 2023-10-17T18:11:53.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-17T18:11:58.209 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-17T18:11:58.210 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-17T18:11:58.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-17T18:11:58.214 INFO:tasks.rgw.c2.client.0:Started 2023-10-17T18:11:58.214 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2023-10-17T18:11:58.239 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:58.239 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:11:58.240 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8000: Connection refused 2023-10-17T18:11:58.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-17T18:11:59.242 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8000 2023-10-17T18:11:59.263 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:11:59.263 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:11:59.265 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:11:59.265 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-10-17T18:11:59.266 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-17T18:11:59.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:59.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--url', 'http://smithi049.front.sepia.ceph.com:8000', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:59.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi049.front.sepia.ceph.com:8000 --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --url http://smithi049.front.sepia.ceph.com:8000 --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:59.362 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:59.362 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:59.403 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "test-realm", 2023-10-17T18:11:59.403 INFO:teuthology.orchestra.run.smithi071.stdout: "current_period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:11:59.404 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 2 2023-10-17T18:11:59.404 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:59.405 INFO:tasks.rgw_multisite:creating zone.. 2023-10-17T18:11:59.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:59.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--endpoints', 'http://smithi071.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:11:59.406 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --endpoints http://smithi071.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:11:59.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:11:59.462 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:11:59.475 INFO:teuthology.orchestra.run.smithi071.stderr:NOTICE: set zone's realm_id=19b3e86c-cc39-4be9-99c4-971ddfc4a8c5 2023-10-17T18:11:59.489 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:11:59.489 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:11:59.489 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-10-17T18:11:59.489 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-17T18:11:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "b2.rgw.control", 2023-10-17T18:11:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-17T18:11:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-17T18:11:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "b2.rgw.log", 2023-10-17T18:11:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-17T18:11:59.491 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-17T18:11:59.491 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-17T18:11:59.491 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-17T18:11:59.491 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-17T18:11:59.491 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "b2.rgw.otp", 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:11:59.492 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:11:59.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:59.493 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:11:59.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:11:59.493 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:11:59.493 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:11:59.494 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-17T18:11:59.494 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:11:59.494 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:11:59.494 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "b2.rgw.buckets.data" 2023-10-17T18:11:59.494 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:59.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:11:59.495 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-17T18:11:59.495 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:11:59.495 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:11:59.495 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:59.496 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:11:59.496 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:11:59.496 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:11:59.496 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-17T18:11:59.496 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:11:59.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-10-17T18:11:59.975 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'b2.rgw.buckets.data' created 2023-10-17T18:11:59.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-10-17T18:12:01.001 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-10-17T18:12:01.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:01.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:01.018 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:12:01.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:01.072 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:01.090 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:12:01.090 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:12:01.090 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-10-17T18:12:01.090 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-17T18:12:01.091 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "b2.rgw.control", 2023-10-17T18:12:01.091 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-17T18:12:01.091 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-17T18:12:01.091 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "b2.rgw.log", 2023-10-17T18:12:01.093 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-17T18:12:01.093 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-17T18:12:01.093 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-17T18:12:01.094 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-17T18:12:01.094 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-17T18:12:01.094 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-17T18:12:01.094 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-17T18:12:01.094 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-17T18:12:01.095 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "b2.rgw.otp", 2023-10-17T18:12:01.095 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2023-10-17T18:12:01.095 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:12:01.095 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:12:01.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-17T18:12:01.096 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2023-10-17T18:12:01.097 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2023-10-17T18:12:01.097 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "b2.rgw.buckets.data", 2023-10-17T18:12:01.097 INFO:teuthology.orchestra.run.smithi071.stdout: "compression_type": "random" 2023-10-17T18:12:01.097 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:01.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:01.098 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-17T18:12:01.098 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2023-10-17T18:12:01.098 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": true 2023-10-17T18:12:01.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:01.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:01.099 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:01.099 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:12:01.099 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-17T18:12:01.099 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:12:01.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:01.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:01.101 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-17T18:12:01.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:01.149 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:01.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:12:01.159+0000 7fc3a1be9780 0 period (cfc9d53d-2765-4da9-b7a8-37407f517c55 does not have zone 049bc79f-3bb6-4b5b-bc52-f8b1db99140f configured 2023-10-17T18:12:08.055 INFO:teuthology.orchestra.run.smithi071.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:12:08.093 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:12:08.093 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:08.093 INFO:teuthology.orchestra.run.smithi071.stdout: "epoch": 4, 2023-10-17T18:12:08.094 INFO:teuthology.orchestra.run.smithi071.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:12:08.094 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": [], 2023-10-17T18:12:08.094 INFO:teuthology.orchestra.run.smithi071.stdout: "period_map": { 2023-10-17T18:12:08.094 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:08.094 INFO:teuthology.orchestra.run.smithi071.stdout: "zonegroups": [ 2023-10-17T18:12:08.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.095 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "a", 2023-10-17T18:12:08.095 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a", 2023-10-17T18:12:08.095 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-10-17T18:12:08.095 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": true, 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-10-17T18:12:08.096 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:12:08.097 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-10-17T18:12:08.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.097 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:12:08.097 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a1", 2023-10-17T18:12:08.097 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.098 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:12:08.098 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.098 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:12:08.098 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:12:08.098 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:12:08.099 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:12:08.100 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:12:08.100 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.100 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "a2", 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.101 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:12:08.102 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:12:08.102 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:12:08.102 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:12:08.102 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:12:08.102 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:12:08.103 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:12:08.103 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:12:08.103 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:12:08.103 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:12:08.103 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.104 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-10-17T18:12:08.105 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-10-17T18:12:08.105 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-10-17T18:12:08.105 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-10-17T18:12:08.105 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.105 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-10-17T18:12:08.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-10-17T18:12:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "b", 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b", 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "api_name": "", 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "is_master": false, 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.108 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:12:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames": [], 2023-10-17T18:12:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: "hostnames_s3website": [], 2023-10-17T18:12:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:12:08.109 INFO:teuthology.orchestra.run.smithi071.stdout: "zones": [ 2023-10-17T18:12:08.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.110 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:12:08.110 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b2", 2023-10-17T18:12:08.110 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.110 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:12:08.111 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:12:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:12:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:12:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:12:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:12:08.112 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:12:08.113 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.114 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.114 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "b1", 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: "endpoints": [ 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: "log_meta": false, 2023-10-17T18:12:08.115 INFO:teuthology.orchestra.run.smithi071.stdout: "log_data": true, 2023-10-17T18:12:08.116 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:12:08.116 INFO:teuthology.orchestra.run.smithi071.stdout: "read_only": false, 2023-10-17T18:12:08.116 INFO:teuthology.orchestra.run.smithi071.stdout: "tier_type": "", 2023-10-17T18:12:08.116 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from_all": true, 2023-10-17T18:12:08.116 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_from": [], 2023-10-17T18:12:08.117 INFO:teuthology.orchestra.run.smithi071.stdout: "redirect_zone": "", 2023-10-17T18:12:08.117 INFO:teuthology.orchestra.run.smithi071.stdout: "supported_features": [ 2023-10-17T18:12:08.117 INFO:teuthology.orchestra.run.smithi071.stdout: "compress-encrypted", 2023-10-17T18:12:08.117 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.117 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_targets": [ 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2023-10-17T18:12:08.118 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2023-10-17T18:12:08.119 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2023-10-17T18:12:08.119 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2023-10-17T18:12:08.119 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.119 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.119 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.120 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "default-placement", 2023-10-17T18:12:08.120 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:12:08.120 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_policy": { 2023-10-17T18:12:08.120 INFO:teuthology.orchestra.run.smithi071.stdout: "groups": [] 2023-10-17T18:12:08.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled_features": [ 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: "resharding" 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2023-10-17T18:12:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: "short_zone_ids": [ 2023-10-17T18:12:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:12:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3843991460 2023-10-17T18:12:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1846350003 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:12:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 3451171648 2023-10-17T18:12:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:12:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:12:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 1989104912 2023-10-17T18:12:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zonegroup": "a", 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: "period_config": { 2023-10-17T18:12:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2023-10-17T18:12:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-10-17T18:12:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-10-17T18:12:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-10-17T18:12:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-10-17T18:12:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2023-10-17T18:12:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2023-10-17T18:12:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2023-10-17T18:12:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: "user_ratelimit": { 2023-10-17T18:12:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:12:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_ratelimit": { 2023-10-17T18:12:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:12:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:12:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:12:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:12:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:12:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: "anonymous_ratelimit": { 2023-10-17T18:12:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_ops": 0, 2023-10-17T18:12:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_ops": 0, 2023-10-17T18:12:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: "max_read_bytes": 0, 2023-10-17T18:12:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: "max_write_bytes": 0, 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_name": "test-realm", 2023-10-17T18:12:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:12:08.133 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:12:08.133 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-17T18:12:08.134 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:12:08.135 INFO:tasks.rgw.c2.client.1.smithi071.stdout:2023-10-17T18:12:08.133+0000 7f3bc6e8d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 119898) UID: 0 2023-10-17T18:12:08.135 INFO:tasks.rgw.c2.client.1.smithi071.stdout:2023-10-17T18:12:08.133+0000 7f3bcb0f67c0 -1 shutting down 2023-10-17T18:12:10.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-17T18:12:14.237 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-17T18:12:14.237 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-17T18:12:14.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-17T18:12:14.241 INFO:tasks.rgw.c2.client.1:Started 2023-10-17T18:12:14.241 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001 2023-10-17T18:12:14.265 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:12:14.266 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:12:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:12:14.268 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T18:12:15.268 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:8001 2023-10-17T18:12:15.291 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:12:15.291 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:12:15.293 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:12:15.294 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2023-10-17T18:12:15.294 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-10-17T18:12:15.294 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-10-17T18:12:15.306 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi 2023-10-17T18:12:15.352 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-10-17T18:12:15.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:15.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:12:15.354 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:12:15.419 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:15.420 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:15.457 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "user_id": "rgw-multisite-test-user", 2023-10-17T18:12:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "display_name": "Multisite Test User", 2023-10-17T18:12:15.457 INFO:teuthology.orchestra.run.smithi049.stdout: "email": "", 2023-10-17T18:12:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "suspended": 0, 2023-10-17T18:12:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "max_buckets": 1000, 2023-10-17T18:12:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "subusers": [], 2023-10-17T18:12:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: "keys": [ 2023-10-17T18:12:15.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "rgw-multisite-test-user", 2023-10-17T18:12:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "EERKR7CLBH40TV9B4PL4", 2023-10-17T18:12:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "43u85u81ko3FPTI0zzFP7QcMlfdDqOIYc6eaA5Iw" 2023-10-17T18:12:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:12:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "swift_keys": [], 2023-10-17T18:12:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "caps": [ 2023-10-17T18:12:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "roles", 2023-10-17T18:12:15.460 INFO:teuthology.orchestra.run.smithi049.stdout: "perm": "*" 2023-10-17T18:12:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:12:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "op_mask": "read, write, delete", 2023-10-17T18:12:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "", 2023-10-17T18:12:15.461 INFO:teuthology.orchestra.run.smithi049.stdout: "default_storage_class": "", 2023-10-17T18:12:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_tags": [], 2023-10-17T18:12:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:12:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:12:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:12:15.462 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:12:15.463 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:12:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:12:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:12:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:12:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.464 INFO:teuthology.orchestra.run.smithi049.stdout: "temp_url_keys": [], 2023-10-17T18:12:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "rgw", 2023-10-17T18:12:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: "mfa_ids": [] 2023-10-17T18:12:15.465 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:15.465 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:12:15.469 INFO:rgw_multi.tests:meta checkpoint 2023-10-17T18:12:15.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:15.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:15.472 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:15.522 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:15.523 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:15.552 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:12:15.564 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:12:15.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.564 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:12:15.564 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:12:15.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.565 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.565 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:12:15.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.566 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.566 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:12:15.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.567 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.567 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:12:15.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.568 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:12:15.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:12:15.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:15.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:15.570 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:15.620 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:15.620 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:15.647+0000 7fe08a6e8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:15.648 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:15.648+0000 7fe08a6e8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:15.659 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:15.659 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:15.659 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:15.659 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:15.660 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:15.660 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:15.660 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:15.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.660 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:15.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.661 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:15.661 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:15.661 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:15.661 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.662 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:15.662 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:15.662 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:15.662 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:15.662 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:15.663 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.663 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:15.663 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:15.664 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:15.664 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.664 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:15.664 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:15.664 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:15.665 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:15.665 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:15.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.666 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:15.666 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:15.666 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:15.666 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.667 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:15.667 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:15.667 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:15.667 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:15.667 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:15.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:15.668 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:15.669 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:15.670 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:15.670 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:15.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:15.670 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:15.672 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:15Z" 2023-10-17T18:12:15.672 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:15.673 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:15.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-17T18:12:20.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:20.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:20.675 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:20.733 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:20.733 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:20.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:20.763+0000 7fddb12f6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:20.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:20.764+0000 7fddb12f6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:20.778 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:20.778 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:20.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:20.780 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:20.781 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:20.782 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:20.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:20.785 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:20.785 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:20.785 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:20.785 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:20.786 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:20.786 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:20.786 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:20.786 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:20.786 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:20.787 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:20.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:20.787 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:20.787 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:20.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.790 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:20.790 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:20.790 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:20.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:20.791 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:20Z" 2023-10-17T18:12:20.791 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:20.792 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:25.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:25.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:25.794 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:25.853 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:25.854 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:25.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:25.884+0000 7f45bac50780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:25.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:25.885+0000 7f45bac50780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:25.898 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:25Z" 2023-10-17T18:12:25.910 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:25.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:30.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:30.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:30.913 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:30.972 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:30.972 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:31.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:31.002+0000 7f64ecc73780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:31.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:31.003+0000 7f64ecc73780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:31.025 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:31.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:31.037 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:31.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.037 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:31.037 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:31.037 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:31.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:31.038 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:31Z" 2023-10-17T18:12:31.038 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:31.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:36.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:36.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:36.041 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:36.100 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:36.101 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:36.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:36.132+0000 7f7a0eb18780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:36.133+0000 7f7a0eb18780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:36.146 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:36Z" 2023-10-17T18:12:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:36.158 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:41.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:41.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:41.160 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:41.218 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:41.219 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:41.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:41.249+0000 7fad7544e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:41.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:41.250+0000 7fad7544e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:41Z" 2023-10-17T18:12:41.276 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:41.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:46.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:46.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:46.280 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:46.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:46.367+0000 7f62b5358780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:46.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:46.368+0000 7f62b5358780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:46.380 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:46Z" 2023-10-17T18:12:46.394 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:46.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:51.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:51.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:51.400 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:51.459 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:51.459 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:51.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:51.490+0000 7fa82c623780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:51.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:51.491+0000 7fa82c623780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:51.503 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:51.504 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:51.504 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:51.504 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:51.504 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:51.504 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:51.505 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:51.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.505 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:51.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:51.506 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:51.506 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:51.506 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:51.506 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:51.507 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:51.507 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:51.507 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:51.507 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:51.507 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:51.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:51.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:51.508 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:51.508 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:51.510 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:51.510 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:51.510 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:51.510 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:51.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:51.511 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:51.511 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:51.511 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:51.511 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:51.512 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:51.512 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:51.512 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:51.512 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:51.514 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:51.514 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:51.514 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:51.514 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:51.515 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:51.515 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:51.515 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:51.517 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:51.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:51.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:51.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:51.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.518 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:51.518 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:51.518 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:51.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:51.519 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:51Z" 2023-10-17T18:12:51.519 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:51.520 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:12:56.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:56.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:12:56.523 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:12:56.583 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:12:56.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:12:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:56.614+0000 7f7f79ef2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:12:56.615+0000 7f7f79ef2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:12:56.628 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:12:56.628 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:12:56.629 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:12:56.629 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:12:56.629 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:12:56.629 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:12:56.630 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:56.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.631 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:12:56.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:56.631 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:12:56.632 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:56.632 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:56.632 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:56.633 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:56.633 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:12:56.633 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:56.633 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:56.634 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:56.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:56.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:56.635 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:12:56.635 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:56.635 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:56.636 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:56.636 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:56.636 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:56.636 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:56.637 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:56.637 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:56.637 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:56.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:12:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:56.640 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:56.640 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:56.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:56.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:12:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:12:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:12:56.643 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:12:56.643 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:12:56.643 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:12:56.643 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:12:56.643 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:12:56.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:12:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:12:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:12:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:12:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:12:56Z" 2023-10-17T18:12:56.646 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:12:56.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:01.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:01.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:01.648 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:01.707 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:01.708 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:01.738+0000 7f54a2ab5780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:01.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:01.739+0000 7f54a2ab5780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:01.753 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:01.753 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:01.753 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:01.753 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:01.753 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:01.754 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:01.754 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:01.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.754 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:01.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:01.755 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:01.755 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:01.755 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:01.755 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:01.756 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:01.756 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:01.756 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:01.756 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:01.756 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:01.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:01.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:01.757 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:01.758 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:01.758 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:01.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:01.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:01.758 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:01.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:01.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:01.759 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:01.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:01.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:01.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:01.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:01.760 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:01.760 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:01.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:01.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:01.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:01.761 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:01.761 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:01.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:01.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:01.762 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:01.762 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:01.763 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.764 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:01.765 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:01.765 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:01.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:01.765 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:01Z" 2023-10-17T18:13:01.765 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:01.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:06.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:06.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:06.768 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:06.826 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:06.827 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:06.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:06.857+0000 7fb223d39780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:06.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:06.858+0000 7fb223d39780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:06.871 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:06.871 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:06.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.873 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:06.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:06.873 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:06.873 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:06.873 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:06.874 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:06.874 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:06.874 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:06.874 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:06.874 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:06.875 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:06.876 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:06.877 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:06.878 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:06.878 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:06.878 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:06.878 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:06.878 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:06.880 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:06.880 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:06.880 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:06.880 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:06.880 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:06.881 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:06.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:06.883 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:06Z" 2023-10-17T18:13:06.883 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:06.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:11.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:11.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:11.886 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:11.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:11.976+0000 7f789ff2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:11.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:11.977+0000 7f789ff2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:11.990 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:11Z" 2023-10-17T18:13:12.002 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:12.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:17.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:17.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:17.005 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:17.064 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:17.065 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:17.095+0000 7fed7c5b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:17.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:17.096+0000 7fed7c5b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:17.109 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:17.120 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:17Z" 2023-10-17T18:13:17.122 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:17.123 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:22.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:22.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:22.125 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:22.184 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:22.185 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:22.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:22.217+0000 7feee0e49780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:22.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:22.218+0000 7feee0e49780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:22.231 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:22.231 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:22.232 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:22.232 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:22.232 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:22.232 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:22.234 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:22.234 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:22.234 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:22.234 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:22.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:22.235 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:22.235 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:22.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:22.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:22.236 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:22.237 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:22.237 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:22.237 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:22.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:22.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:22.238 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:22.238 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:22.238 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:22.238 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:22.239 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:22.239 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:22.239 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:22.239 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:22.239 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:22.240 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:22.241 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:22.241 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:22.241 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:22.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:22.241 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:22.242 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:22.242 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:22.242 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:22.242 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:22.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.243 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:22.243 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:22.245 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:22.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:22.245 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:22Z" 2023-10-17T18:13:22.246 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:22.247 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:27.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:27.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:27.248 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:27.307 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:27.307 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:27.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:27.339+0000 7fc8235d4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:27.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:27.340+0000 7fc8235d4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:27.354 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:27.354 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:27.354 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:27.354 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:27.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:27.356 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:27.358 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:27.358 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:27.358 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:27.358 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:27.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:27.360 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:27.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:27.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:27.362 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:27.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:27.362 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:27.363 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:27.363 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:27.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:27.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:27.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.364 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:27Z" 2023-10-17T18:13:27.365 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:27.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:32.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:32.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:32.368 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:32.426 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:32.427 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:32.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:32.457+0000 7f29c4f44780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:32.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:32.458+0000 7f29c4f44780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:32.471 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:32Z" 2023-10-17T18:13:32.483 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:32.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:37.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:37.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:37.485 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:37.545 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:37.545 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:37.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:37.576+0000 7f77ed75b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:37.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:37.577+0000 7f77ed75b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:37.590 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:37.590 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:37.590 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:37.590 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:37.591 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:37.591 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:37.591 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:37.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.591 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:37.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:37.592 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:37.592 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:37.592 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:37.592 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:37.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:37.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:37.594 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:37.594 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:37.594 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:37.595 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:37.595 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:37.595 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:37.595 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:37.595 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:37.596 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:37.597 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:37.597 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:37.597 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:37.597 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:37.597 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:37.598 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:37.598 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:37.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:37.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:37.599 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:37.600 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:37.600 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:37.600 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:37.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:37.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:37.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:37.602 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:37Z" 2023-10-17T18:13:37.602 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:37.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:42.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:42.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:42.605 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:42.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:42.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:42.695+0000 7fd006d65780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:42.696+0000 7fd006d65780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:42.709 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:42.711 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:42.711 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:42.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.711 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:42.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:42.712 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:42.712 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:42.712 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:42.712 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:42.713 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:42.713 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:42.713 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:42.715 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:42.716 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:42.716 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:42.716 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:42.716 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:42.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:42.717 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:42.717 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:42.717 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:42.718 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:42.718 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:42.718 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:42.718 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:42.718 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:42.719 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:42.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:42.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:42.719 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:42.720 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:42.720 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:42.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:42.720 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:42.720 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:42.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:42.721 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:42.721 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:42.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:42.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:42.722 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:42.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.722 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:42.722 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:42.723 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:42.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:42.723 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:42Z" 2023-10-17T18:13:42.723 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:42.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:47.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:47.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:47.726 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:47.785 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:47.786 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:47.817 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:47.816+0000 7f5e9b254780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:47.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:47.817+0000 7f5e9b254780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:47Z" 2023-10-17T18:13:47.844 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:47.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:52.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:52.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:52.847 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:52.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:52.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:52.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:52.936+0000 7f38aa66d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:52.938 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:52.937+0000 7f38aa66d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:52.951 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:52.951 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:52.951 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:52.952 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:52.952 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:52.952 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:52.952 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:52.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.953 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:52.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:52.953 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:52.953 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:52.953 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:52.954 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:52.955 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:52.956 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:52.956 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:52.956 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:52.956 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:52.956 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:52.958 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:52.958 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:52.958 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:52.958 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:52.958 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:52.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:52.959 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:52.959 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:52.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:52.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:52.960 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:52.960 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:52.960 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:52.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:52.961 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:52.961 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:52.961 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:52.961 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:52.961 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:52.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:52.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:52.962 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:52.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.962 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:52.963 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:52.963 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:52.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:52.963 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:52Z" 2023-10-17T18:13:52.963 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:52.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:13:57.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:57.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:13:57.967 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:13:58.026 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:13:58.026 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:13:58.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:58.057+0000 7fbfed5ae780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:58.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:13:58.058+0000 7fbfed5ae780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:13:58.071 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:13:58.071 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:13:58.071 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:13:58.071 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:13:58.071 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:13:58.072 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:13:58.072 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:58.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.072 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:13:58.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:58.073 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:13:58.073 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:58.073 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:58.073 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:58.073 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:58.074 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:13:58.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:58.074 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:58.074 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:58.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:58.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:58.075 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:13:58.075 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:58.075 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:58.076 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:58.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:58.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:58.077 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:13:58.077 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:58.079 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:58.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:58.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:13:58.079 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:13:58.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:13:58.081 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:13:58.081 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:13:58.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:58.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:13:58.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:13:58.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.082 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:13:58.082 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:13:58.082 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:13:58.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:13:58.083 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:13:58Z" 2023-10-17T18:13:58.083 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:13:58.084 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:14:03.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:03.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:03.086 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:03.147 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:03.147 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:03.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:03.179+0000 7f5472f38780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:03.180+0000 7f5472f38780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:03.193 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:03.193 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:03.193 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:03.193 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:03.194 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:03.194 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:03.194 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:03.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.195 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:03.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:03.195 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:03.195 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:03.195 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:03.196 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:03.196 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:03.196 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:03.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:03.196 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:03.197 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:03.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:03.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:03.198 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:03.198 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:03.198 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:03.198 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:03.198 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:03.199 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:03.199 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:03.199 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:03.199 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:03.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:03.200 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:03.201 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:03.201 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:03.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:03.201 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:03.201 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:03.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:03.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:03.202 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:03.202 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:03.203 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:03.203 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:03.203 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:03.203 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:03.203 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:03.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:03.205 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:03.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.205 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:03.205 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:03.205 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:03.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:03.206 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:03Z" 2023-10-17T18:14:03.206 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:03.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:14:08.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:08.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:08.209 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:08.270 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:08.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:08.300+0000 7fbbd548e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:08.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:08.302+0000 7fbbd548e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:08.314 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:08.314 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:08.314 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:08.314 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:08.314 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:08.315 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:08.315 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:08.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.315 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:08.316 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:08.317 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:08.317 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:08.317 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:08.317 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:08.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:08.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:08.318 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:08.318 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:08.318 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:08.319 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:08.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:08.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:08.320 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:08.320 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:08.321 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:08.322 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:08.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:08.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:08.322 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:08.323 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:08.323 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:08.323 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:08.323 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:08.323 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:08.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:08.324 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:08.324 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:08.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:08.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:08.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:08.326 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:08Z" 2023-10-17T18:14:08.326 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:08.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566335.454808_63.1 target= 2023-10-17T18:14:13.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.329 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.388 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.388 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:13.420+0000 7f50153d8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:13.421+0000 7f50153d8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.434 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:13.434 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:13.434 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.435 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.436 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.437 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:13.437 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.437 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:13.437 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.437 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.438 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:13.438 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.438 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.439 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.440 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:13.440 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.443 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.444 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.444 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.444 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.445 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:13Z" 2023-10-17T18:14:13.446 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:13.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:13.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-17T18:14:13.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.448 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.504 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.505 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.534+0000 7f96f11d8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.535+0000 7f96f11d8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.547 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:14:13.548 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-10-17T18:14:13.548 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-10-17T18:14:13.548 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-10-17T18:14:13.548 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-10-17T18:14:13.548 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-10-17T18:14:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.549 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-10-17T18:14:13.550 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.551 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:13.551 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.552 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-10-17T18:14:13.552 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.552 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.552 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.552 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.553 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.553 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.553 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.553 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.553 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.554 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.555 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.556 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-10-17T18:14:13.556 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.557 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.558 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.558 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.558 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.558 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:14:13.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.559 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-10-17T18:14:13.559 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-10-17T18:14:13.559 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-10-17T18:14:13.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.559 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-10-17T18:14:13Z" 2023-10-17T18:14:13.560 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:14:13.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-17T18:14:13.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-17T18:14:13.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.612 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.612 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.644+0000 7ff01f8d4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.646 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.645+0000 7ff01f8d4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.658 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:14:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-10-17T18:14:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-10-17T18:14:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-10-17T18:14:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-10-17T18:14:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-10-17T18:14:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-10-17T18:14:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-10-17T18:14:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-10-17T18:14:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-10-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-10-17T18:14:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-10-17T18:14:13.670 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-10-17T18:14:13.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.670 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-10-17T18:14:13Z" 2023-10-17T18:14:13.670 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:14:13.672 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-17T18:14:13.672 INFO:rgw_multi.tests:meta checkpoint 2023-10-17T18:14:13.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.673 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.726 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.726 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.757 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:13.769 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:13.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.769 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:13.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.770 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.770 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:13.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.771 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.772 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:13.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.772 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:13.773 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:13.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.774 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.825 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.825 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:13.851+0000 7f258f3cf780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:13.852+0000 7f258f3cf780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.864 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:13.864 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:13.865 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.866 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:13.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.866 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:13.866 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.867 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.867 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.867 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.867 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:13.867 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.868 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:13.868 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.868 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.869 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:13.869 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.869 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.869 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.870 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.870 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.870 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.870 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.870 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.871 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.872 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.872 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.872 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.872 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.872 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.873 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:13.873 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:13.874 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:13.874 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:13.874 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:13.874 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:13.876 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:13.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.876 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:13.876 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:13.876 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:13.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:13.877 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:13Z" 2023-10-17T18:14:13.877 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:13.878 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:13.879 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-17T18:14:13.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.880 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:13.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:13.929 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:13.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.957+0000 7f4d4c635780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:13.958+0000 7f4d4c635780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:13.970 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:14:13.970 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-10-17T18:14:13.970 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-10-17T18:14:13.971 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-10-17T18:14:13.971 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-10-17T18:14:13.971 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:13.971 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.972 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-10-17T18:14:13.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.972 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-10-17T18:14:13.972 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-10-17T18:14:13.973 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.974 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:13.974 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.974 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.975 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-10-17T18:14:13.975 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.975 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.975 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.976 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.976 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.976 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.976 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.976 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.977 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.977 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-10-17T18:14:13.977 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.978 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.978 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.978 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.978 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.978 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:13.979 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-10-17T18:14:13.980 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:13.980 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:13.980 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:13.980 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:13.981 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:13.981 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:13.981 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:13.981 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:13.982 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.982 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:13.982 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:14:13.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.982 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-10-17T18:14:13.983 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-10-17T18:14:13.983 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-10-17T18:14:13.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:13.983 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-10-17T18:14:13Z" 2023-10-17T18:14:13.984 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:14:13.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-17T18:14:13.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-17T18:14:13.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:13.986 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:14.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:14.035 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:14.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:14.061+0000 7f368f7aa780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:14.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-10-17T18:14:14.062+0000 7f368f7aa780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:14.073 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-10-17T18:14:14.073 INFO:teuthology.orchestra.run.smithi071.stdout: "sync_status": { 2023-10-17T18:14:14.073 INFO:teuthology.orchestra.run.smithi071.stdout: "info": { 2023-10-17T18:14:14.074 INFO:teuthology.orchestra.run.smithi071.stdout: "status": "sync", 2023-10-17T18:14:14.074 INFO:teuthology.orchestra.run.smithi071.stdout: "num_shards": 4, 2023-10-17T18:14:14.074 INFO:teuthology.orchestra.run.smithi071.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:14.074 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:14.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.075 INFO:teuthology.orchestra.run.smithi071.stdout: "markers": [ 2023-10-17T18:14:14.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:14.075 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 0, 2023-10-17T18:14:14.075 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 1, 2023-10-17T18:14:14.076 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:14.078 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 1, 2023-10-17T18:14:14.078 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:14.078 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:14.078 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:14.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:14.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:14.080 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 2, 2023-10-17T18:14:14.080 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:14.080 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:14.081 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-10-17T18:14:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: "key": 3, 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "state": 1, 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "marker": "", 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "next_step_marker": "", 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "total_entries": 0, 2023-10-17T18:14:14.083 INFO:teuthology.orchestra.run.smithi071.stdout: "pos": 0, 2023-10-17T18:14:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: "timestamp": "0.000000", 2023-10-17T18:14:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_epoch": 2 2023-10-17T18:14:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-10-17T18:14:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: "full_sync": { 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: "total": 1, 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: "complete": 1 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-10-17T18:14:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: "current_time": "2023-10-17T18:14:14Z" 2023-10-17T18:14:14.086 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-10-17T18:14:14.087 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-17T18:14:14.087 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-10-17T18:14:14.311 DEBUG:boto:Using access key provided by client. 2023-10-17T18:14:14.444 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:14:14.446 DEBUG:boto:Using access key provided by client. 2023-10-17T18:14:14.482 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:14:14.483 DEBUG:boto:Using access key provided by client. 2023-10-17T18:14:14.483 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:14:14.484 DEBUG:boto:Using access key provided by client. 2023-10-17T18:14:14.486 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:14:14.487 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-1 2023-10-17T18:14:14.487 DEBUG:boto:path=/hbmuog-1/ 2023-10-17T18:14:14.488 DEBUG:boto:auth_path=/hbmuog-1/ 2023-10-17T18:14:14.488 DEBUG:boto:Method: PUT 2023-10-17T18:14:14.488 DEBUG:boto:Path: /hbmuog-1/ 2023-10-17T18:14:14.489 DEBUG:boto:Data: 2023-10-17T18:14:14.489 DEBUG:boto:Headers: {} 2023-10-17T18:14:14.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:14.490 DEBUG:boto:Port: 8000 2023-10-17T18:14:14.490 DEBUG:boto:Params: {} 2023-10-17T18:14:14.491 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:14:14.491 DEBUG:boto:Token: None 2023-10-17T18:14:14.492 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:14 GMT /hbmuog-1/ 2023-10-17T18:14:14.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IWy8vBWURH6ZIqD2e4Xp0ZXhjtA= 2023-10-17T18:14:14.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IWy8vBWURH6ZIqD2e4Xp0ZXhjtA=', 'Content-Length': '0'} 2023-10-17T18:14:17.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dec6958e83f29bb0-00652ecef6-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:17.193 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-2 2023-10-17T18:14:17.193 DEBUG:boto:path=/hbmuog-2/ 2023-10-17T18:14:17.193 DEBUG:boto:auth_path=/hbmuog-2/ 2023-10-17T18:14:17.194 DEBUG:boto:Method: PUT 2023-10-17T18:14:17.194 DEBUG:boto:Path: /hbmuog-2/ 2023-10-17T18:14:17.195 DEBUG:boto:Data: 2023-10-17T18:14:17.195 DEBUG:boto:Headers: {} 2023-10-17T18:14:17.195 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:17.196 DEBUG:boto:Port: 8001 2023-10-17T18:14:17.196 DEBUG:boto:Params: {} 2023-10-17T18:14:17.197 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:14:17.197 DEBUG:boto:Token: None 2023-10-17T18:14:17.197 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:17 GMT /hbmuog-2/ 2023-10-17T18:14:17.198 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0Y0aEiopDEyMFq6fWMWhnMR/B/c= 2023-10-17T18:14:17.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0Y0aEiopDEyMFq6fWMWhnMR/B/c=', 'Content-Length': '0'} 2023-10-17T18:14:20.204 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f3899d73d178988-00652ecef9-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:20.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:20.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:20.205 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:20.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:20.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:20.292 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:20.304 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566457.215885_76.1", 2023-10-17T18:14:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:17.215885Z" 2023-10-17T18:14:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.308 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:20.308 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566457.215885_76.1', 2: '', 3: ''} 2023-10-17T18:14:20.309 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:20.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:20.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:20.310 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:20.361 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:20.361 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:20.387+0000 7ffbf0ab4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:20.388+0000 7ffbf0ab4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:20.401 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566457.183694_69.1", 2023-10-17T18:14:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:17.183694Z", 2023-10-17T18:14:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:20Z" 2023-10-17T18:14:20.414 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:20.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566457.183694_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:17.183694Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:20Z" } 2023-10-17T18:14:20.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566457.183694_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:17.183694Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:20.417 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566457.215885_76.1', 2: '', 3: ''} 2023-10-17T18:14:20.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566457.183694_69.1', 2: '', 3: ''} 2023-10-17T18:14:20.417 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697566457.215885_76.1 target=1_1697566457.183694_69.1 2023-10-17T18:14:25.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.420 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:25.480 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:25.481 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:25.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:25.511+0000 7f112a2e3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:25.513+0000 7f112a2e3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:25.526 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:25.526 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:25.527 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:25.527 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:25.527 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:25.527 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:25.527 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.528 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:25.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.528 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:25.528 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.529 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.529 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:25.529 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.529 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:25.529 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.530 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:25.530 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566457.215885_76.1", 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.531 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.532 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.532 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:17.215885Z", 2023-10-17T18:14:25.532 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.533 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.534 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.534 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.534 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.534 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:25.535 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.535 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:25Z" 2023-10-17T18:14:25.539 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:25.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566457.215885_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:17.215885Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:25Z" } 2023-10-17T18:14:25.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566457.215885_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:17.215885Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:25.541 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566457.215885_76.1', 2: '', 3: ''} 2023-10-17T18:14:25.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566457.215885_76.1', 2: '', 3: ''} 2023-10-17T18:14:25.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:25.542 DEBUG:boto:path=/hbmuog-1/ 2023-10-17T18:14:25.543 DEBUG:boto:auth_path=/hbmuog-1/ 2023-10-17T18:14:25.543 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.544 DEBUG:boto:Path: /hbmuog-1/ 2023-10-17T18:14:25.544 DEBUG:boto:Data: 2023-10-17T18:14:25.544 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.545 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.545 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.546 DEBUG:boto:Params: {} 2023-10-17T18:14:25.546 DEBUG:boto:Token: None 2023-10-17T18:14:25.547 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-1/ 2023-10-17T18:14:25.547 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wZCznVp/JDySedZTlU+sk05O63A= 2023-10-17T18:14:25.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wZCznVp/JDySedZTlU+sk05O63A=', 'Content-Length': '0'} 2023-10-17T18:14:25.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bda23e6cd6e0467-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.551 DEBUG:boto:path=/hbmuog-2/ 2023-10-17T18:14:25.551 DEBUG:boto:auth_path=/hbmuog-2/ 2023-10-17T18:14:25.551 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.552 DEBUG:boto:Path: /hbmuog-2/ 2023-10-17T18:14:25.552 DEBUG:boto:Data: 2023-10-17T18:14:25.553 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.554 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.554 DEBUG:boto:Params: {} 2023-10-17T18:14:25.554 DEBUG:boto:Token: None 2023-10-17T18:14:25.555 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-2/ 2023-10-17T18:14:25.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SsYc18j3RSHz13pNWF3oyMinuv4= 2023-10-17T18:14:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SsYc18j3RSHz13pNWF3oyMinuv4=', 'Content-Length': '0'} 2023-10-17T18:14:25.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ac6dcaa00693171-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.559 DEBUG:boto:path=/hbmuog-1/ 2023-10-17T18:14:25.559 DEBUG:boto:auth_path=/hbmuog-1/ 2023-10-17T18:14:25.559 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.560 DEBUG:boto:Path: /hbmuog-1/ 2023-10-17T18:14:25.560 DEBUG:boto:Data: 2023-10-17T18:14:25.561 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.561 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.562 DEBUG:boto:Params: {} 2023-10-17T18:14:25.562 DEBUG:boto:Token: None 2023-10-17T18:14:25.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-1/ 2023-10-17T18:14:25.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wZCznVp/JDySedZTlU+sk05O63A= 2023-10-17T18:14:25.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wZCznVp/JDySedZTlU+sk05O63A=', 'Content-Length': '0'} 2023-10-17T18:14:25.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063666a81241c7a7d-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.566 DEBUG:boto:path=/hbmuog-2/ 2023-10-17T18:14:25.567 DEBUG:boto:auth_path=/hbmuog-2/ 2023-10-17T18:14:25.567 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.567 DEBUG:boto:Path: /hbmuog-2/ 2023-10-17T18:14:25.568 DEBUG:boto:Data: 2023-10-17T18:14:25.568 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.569 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.569 DEBUG:boto:Params: {} 2023-10-17T18:14:25.569 DEBUG:boto:Token: None 2023-10-17T18:14:25.570 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-2/ 2023-10-17T18:14:25.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SsYc18j3RSHz13pNWF3oyMinuv4= 2023-10-17T18:14:25.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SsYc18j3RSHz13pNWF3oyMinuv4=', 'Content-Length': '0'} 2023-10-17T18:14:25.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a7155a16395246d-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-10-17T18:14:25.580 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-3 2023-10-17T18:14:25.580 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.581 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.581 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.581 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.582 DEBUG:boto:Data: 2023-10-17T18:14:25.582 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.583 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.583 DEBUG:boto:Params: {} 2023-10-17T18:14:25.584 DEBUG:boto:Token: None 2023-10-17T18:14:25.584 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk= 2023-10-17T18:14:25.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk=', 'Content-Length': '0'} 2023-10-17T18:14:25.597 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004477b26b0ae7a0c1-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.598 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-4 2023-10-17T18:14:25.598 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.599 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.599 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.599 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.600 DEBUG:boto:Data: 2023-10-17T18:14:25.600 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.600 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.601 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.601 DEBUG:boto:Params: {} 2023-10-17T18:14:25.601 DEBUG:boto:Token: None 2023-10-17T18:14:25.602 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.602 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s= 2023-10-17T18:14:25.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s=', 'Content-Length': '0'} 2023-10-17T18:14:25.624 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000022eff2d0e29d88e6-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.625 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:25.675 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:25.675 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:25.704 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:25.719 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566465.615353_84.1", 2023-10-17T18:14:25.721 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:25.615353Z" 2023-10-17T18:14:25.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.721 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.721 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:25.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.722 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.722 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:25.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.723 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:25.724 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:25.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:25.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.725 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:25.776 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:25.777 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:25.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:25.804+0000 7fd63bb52780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:25.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:25.805+0000 7fd63bb52780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:25.817 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:25.817 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:25.818 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:25.818 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:25.818 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:25.818 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:25.819 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.819 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:25.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.819 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:25.820 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.821 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:25.821 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566465.615353_84.1", 2023-10-17T18:14:25.822 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:25.615353Z", 2023-10-17T18:14:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.824 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.825 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:25.826 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:25.827 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:25.828 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:25.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.829 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:25.829 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:25.829 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:25.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:25.830 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:25Z" 2023-10-17T18:14:25.830 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:25.831 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566465.615353_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:25.615353Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:25Z" } 2023-10-17T18:14:25.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566465.615353_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:25.615353Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:25.832 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:25.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:25.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:25.833 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.833 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.833 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.834 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.834 DEBUG:boto:Data: 2023-10-17T18:14:25.834 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.835 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.836 DEBUG:boto:Params: {} 2023-10-17T18:14:25.836 DEBUG:boto:Token: None 2023-10-17T18:14:25.836 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek= 2023-10-17T18:14:25.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek=', 'Content-Length': '0'} 2023-10-17T18:14:25.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dff0f009068ee3e-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.840 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.841 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.841 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.841 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.842 DEBUG:boto:Data: 2023-10-17T18:14:25.842 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.842 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.843 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.843 DEBUG:boto:Params: {} 2023-10-17T18:14:25.844 DEBUG:boto:Token: None 2023-10-17T18:14:25.844 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM= 2023-10-17T18:14:25.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM=', 'Content-Length': '0'} 2023-10-17T18:14:25.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000094c097898d3b184-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.847 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.848 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.848 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.849 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.849 DEBUG:boto:Data: 2023-10-17T18:14:25.849 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.850 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.850 DEBUG:boto:Params: {} 2023-10-17T18:14:25.851 DEBUG:boto:Token: None 2023-10-17T18:14:25.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek= 2023-10-17T18:14:25.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek=', 'Content-Length': '0'} 2023-10-17T18:14:25.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24aaa829ed9cb79-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.855 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.855 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.856 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.856 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.856 DEBUG:boto:Data: 2023-10-17T18:14:25.857 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.857 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.858 DEBUG:boto:Params: {} 2023-10-17T18:14:25.858 DEBUG:boto:Token: None 2023-10-17T18:14:25.859 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM= 2023-10-17T18:14:25.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM=', 'Content-Length': '0'} 2023-10-17T18:14:25.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d31cbd70bb42feb3-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.862 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.862 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.863 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.863 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.863 DEBUG:boto:Data: 2023-10-17T18:14:25.864 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.864 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.864 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.865 DEBUG:boto:Params: {} 2023-10-17T18:14:25.865 DEBUG:boto:Token: None 2023-10-17T18:14:25.866 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.866 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk= 2023-10-17T18:14:25.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk=', 'Content-Length': '0'} 2023-10-17T18:14:25.869 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009cd166c17c1f24d4-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.869 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.870 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.870 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.871 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.871 DEBUG:boto:Data: 2023-10-17T18:14:25.871 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.872 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.872 DEBUG:boto:Params: {} 2023-10-17T18:14:25.873 DEBUG:boto:Token: None 2023-10-17T18:14:25.873 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s= 2023-10-17T18:14:25.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s=', 'Content-Length': '0'} 2023-10-17T18:14:25.876 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d92110b3af1e5f3d-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.877 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.877 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.877 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.878 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.878 DEBUG:boto:Data: 2023-10-17T18:14:25.878 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.879 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.879 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.880 DEBUG:boto:Params: {} 2023-10-17T18:14:25.880 DEBUG:boto:Token: None 2023-10-17T18:14:25.880 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk= 2023-10-17T18:14:25.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QbClQY+byvlhjqYvELAQip5ICbk=', 'Content-Length': '0'} 2023-10-17T18:14:25.886 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008fe3b2f79c68c12d-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.886 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.886 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.887 DEBUG:boto:Method: PUT 2023-10-17T18:14:25.887 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.888 DEBUG:boto:Data: 2023-10-17T18:14:25.888 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.889 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.889 DEBUG:boto:Params: {} 2023-10-17T18:14:25.889 DEBUG:boto:Token: None 2023-10-17T18:14:25.890 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s= 2023-10-17T18:14:25.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BcB85hmdD/VmTjMgH0NzQmPIU0s=', 'Content-Length': '0'} 2023-10-17T18:14:25.936 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000370f693fcd50e846-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.936 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.937 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.937 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.937 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.938 DEBUG:boto:Data: 2023-10-17T18:14:25.938 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.939 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.939 DEBUG:boto:Params: {} 2023-10-17T18:14:25.940 DEBUG:boto:Token: None 2023-10-17T18:14:25.940 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek= 2023-10-17T18:14:25.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek=', 'Content-Length': '0'} 2023-10-17T18:14:25.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009667c6f194cf7c20-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.944 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.944 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.945 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.945 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.945 DEBUG:boto:Data: 2023-10-17T18:14:25.946 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:25.946 DEBUG:boto:Port: 8000 2023-10-17T18:14:25.947 DEBUG:boto:Params: {} 2023-10-17T18:14:25.947 DEBUG:boto:Token: None 2023-10-17T18:14:25.948 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM= 2023-10-17T18:14:25.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM=', 'Content-Length': '0'} 2023-10-17T18:14:25.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2e03c5633b4dba6-00652ecf01-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.951 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:25.951 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:25.951 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.952 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:25.952 DEBUG:boto:Data: 2023-10-17T18:14:25.952 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.953 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.953 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.953 DEBUG:boto:Params: {} 2023-10-17T18:14:25.954 DEBUG:boto:Token: None 2023-10-17T18:14:25.954 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-3/ 2023-10-17T18:14:25.955 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek= 2023-10-17T18:14:25.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZlH2EZTpXjV7miJrZXFhBXuofek=', 'Content-Length': '0'} 2023-10-17T18:14:25.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6bd0ad32f327655-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.958 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:25.958 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:25.958 DEBUG:boto:Method: HEAD 2023-10-17T18:14:25.959 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:25.959 DEBUG:boto:Data: 2023-10-17T18:14:25.959 DEBUG:boto:Headers: {} 2023-10-17T18:14:25.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:25.960 DEBUG:boto:Port: 8001 2023-10-17T18:14:25.960 DEBUG:boto:Params: {} 2023-10-17T18:14:25.961 DEBUG:boto:Token: None 2023-10-17T18:14:25.961 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:25 GMT /hbmuog-4/ 2023-10-17T18:14:25.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM= 2023-10-17T18:14:25.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyoyBkJE3YyK0ow3RAz/QItHXgM=', 'Content-Length': '0'} 2023-10-17T18:14:25.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000451a811809b9e66a-00652ecf01-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:25.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:25.965 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.014 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.015 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.044 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:26.057 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:26.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.057 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.057 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:26.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.058 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566465.615353_84.1", 2023-10-17T18:14:26.058 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:25.615353Z" 2023-10-17T18:14:26.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.059 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.059 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.060 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.060 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.060 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:26.061 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:26.061 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:26.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.063 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.113 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.113 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.140+0000 7ff69c945780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.141+0000 7ff69c945780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.153 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:26.153 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:26.154 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:26.154 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:26.154 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:26.154 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:26.155 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.155 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:26.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.155 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:26.156 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.157 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:26.157 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.158 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:26.158 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.158 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.158 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566465.615353_84.1", 2023-10-17T18:14:26.158 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:25.615353Z", 2023-10-17T18:14:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.159 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.160 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:26.160 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.160 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.161 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.162 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:26.163 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.163 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.163 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.163 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.163 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:26.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.165 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:26.165 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:26.165 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:26.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.166 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:26Z" 2023-10-17T18:14:26.166 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:26.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566465.615353_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:25.615353Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:26Z" } 2023-10-17T18:14:26.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566465.615353_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:25.615353Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:26.168 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:26.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566465.615353_84.1', 2: '', 3: ''} 2023-10-17T18:14:26.168 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:26.169 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:26.169 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:26.169 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.170 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:26.170 DEBUG:boto:Data: 2023-10-17T18:14:26.170 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.171 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.171 DEBUG:boto:Params: {} 2023-10-17T18:14:26.172 DEBUG:boto:Token: None 2023-10-17T18:14:26.172 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-3/ 2023-10-17T18:14:26.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+wbUBKtoTnrfcuMGzwbeQCDcbZc= 2023-10-17T18:14:26.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+wbUBKtoTnrfcuMGzwbeQCDcbZc=', 'Content-Length': '0'} 2023-10-17T18:14:26.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf2748aedb8f60d9-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.176 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:26.177 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:26.177 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.177 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:26.178 DEBUG:boto:Data: 2023-10-17T18:14:26.178 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.179 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.179 DEBUG:boto:Params: {} 2023-10-17T18:14:26.180 DEBUG:boto:Token: None 2023-10-17T18:14:26.180 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-4/ 2023-10-17T18:14:26.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hsoD6gl7uGNMPjF4wxZaFteQaiY= 2023-10-17T18:14:26.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hsoD6gl7uGNMPjF4wxZaFteQaiY=', 'Content-Length': '0'} 2023-10-17T18:14:26.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e93e6afddccea8ce-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.183 DEBUG:boto:path=/hbmuog-3/ 2023-10-17T18:14:26.184 DEBUG:boto:auth_path=/hbmuog-3/ 2023-10-17T18:14:26.184 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.184 DEBUG:boto:Path: /hbmuog-3/ 2023-10-17T18:14:26.185 DEBUG:boto:Data: 2023-10-17T18:14:26.185 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.186 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.186 DEBUG:boto:Params: {} 2023-10-17T18:14:26.186 DEBUG:boto:Token: None 2023-10-17T18:14:26.187 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-3/ 2023-10-17T18:14:26.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+wbUBKtoTnrfcuMGzwbeQCDcbZc= 2023-10-17T18:14:26.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+wbUBKtoTnrfcuMGzwbeQCDcbZc=', 'Content-Length': '0'} 2023-10-17T18:14:26.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088707e800142687e-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.190 DEBUG:boto:path=/hbmuog-4/ 2023-10-17T18:14:26.191 DEBUG:boto:auth_path=/hbmuog-4/ 2023-10-17T18:14:26.191 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.191 DEBUG:boto:Path: /hbmuog-4/ 2023-10-17T18:14:26.192 DEBUG:boto:Data: 2023-10-17T18:14:26.192 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.193 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.193 DEBUG:boto:Params: {} 2023-10-17T18:14:26.193 DEBUG:boto:Token: None 2023-10-17T18:14:26.194 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-4/ 2023-10-17T18:14:26.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hsoD6gl7uGNMPjF4wxZaFteQaiY= 2023-10-17T18:14:26.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hsoD6gl7uGNMPjF4wxZaFteQaiY=', 'Content-Length': '0'} 2023-10-17T18:14:26.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000543dda8700b59d07-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-10-17T18:14:26.200 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-5 2023-10-17T18:14:26.201 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.201 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.201 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.202 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.202 DEBUG:boto:Data: 2023-10-17T18:14:26.203 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.203 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.204 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.204 DEBUG:boto:Params: {} 2023-10-17T18:14:26.204 DEBUG:boto:Token: None 2023-10-17T18:14:26.205 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rMZJhrf8/r0JUsTA+1nbAQZ1fgs= 2023-10-17T18:14:26.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rMZJhrf8/r0JUsTA+1nbAQZ1fgs=', 'Content-Length': '0'} 2023-10-17T18:14:26.218 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4456c27c26cd3ba-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.218 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-6 2023-10-17T18:14:26.219 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.219 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.219 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.220 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.220 DEBUG:boto:Data: 2023-10-17T18:14:26.221 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.221 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.222 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.222 DEBUG:boto:Params: {} 2023-10-17T18:14:26.222 DEBUG:boto:Token: None 2023-10-17T18:14:26.223 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.223 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:753PUANEAuZBRDCnMBgMzwapiEg= 2023-10-17T18:14:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:753PUANEAuZBRDCnMBgMzwapiEg=', 'Content-Length': '0'} 2023-10-17T18:14:26.244 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021935b661c5e4338-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.246 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.296 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:26.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:26.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.236784_92.1", 2023-10-17T18:14:26.347 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:26.236784Z" 2023-10-17T18:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.350 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.351 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:26.353 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566466.236784_92.1', 2: '', 3: ''} 2023-10-17T18:14:26.354 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:26.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.355 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.406 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.406 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.432+0000 7fe2d1dc6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.433+0000 7fe2d1dc6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.446 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:26.446 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:26.446 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:26.447 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:26.447 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:26.447 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:26.447 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.448 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:26.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.448 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:26.448 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.449 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.449 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.449 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.449 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:26.449 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.450 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:26.450 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.451 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:26.451 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.452 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.452 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.236784_92.1", 2023-10-17T18:14:26.452 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.452 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.452 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.453 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:26.236784Z", 2023-10-17T18:14:26.453 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.454 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:26.454 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.454 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.454 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.455 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.455 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.455 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.455 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.455 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.456 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.456 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:26.457 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.457 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.457 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.457 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.457 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.458 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.458 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.458 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.459 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:26.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.459 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:26.459 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:26.460 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:26.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.460 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:26Z" 2023-10-17T18:14:26.460 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:26.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566466.236784_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:26.236784Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:26Z" } 2023-10-17T18:14:26.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566466.236784_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:26.236784Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:26.462 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.236784_92.1', 2: '', 3: ''} 2023-10-17T18:14:26.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.236784_92.1', 2: '', 3: ''} 2023-10-17T18:14:26.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:26.464 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.464 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.464 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.465 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.465 DEBUG:boto:Data: 2023-10-17T18:14:26.466 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.466 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.467 DEBUG:boto:Params: {} 2023-10-17T18:14:26.467 DEBUG:boto:Token: None 2023-10-17T18:14:26.468 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc= 2023-10-17T18:14:26.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc=', 'Content-Length': '0'} 2023-10-17T18:14:26.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee12f79706319cfc-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.472 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.472 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.472 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.473 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.473 DEBUG:boto:Data: 2023-10-17T18:14:26.474 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.474 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.475 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.475 DEBUG:boto:Params: {} 2023-10-17T18:14:26.475 DEBUG:boto:Token: None 2023-10-17T18:14:26.476 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.476 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo= 2023-10-17T18:14:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo=', 'Content-Length': '0'} 2023-10-17T18:14:26.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bbf9ab4718e69f8-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.480 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.480 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.481 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.481 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.481 DEBUG:boto:Data: 2023-10-17T18:14:26.482 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.482 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.483 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.483 DEBUG:boto:Params: {} 2023-10-17T18:14:26.484 DEBUG:boto:Token: None 2023-10-17T18:14:26.484 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.485 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc= 2023-10-17T18:14:26.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc=', 'Content-Length': '0'} 2023-10-17T18:14:26.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d342fdbe59393dc1-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.488 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.488 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.489 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.489 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.490 DEBUG:boto:Data: 2023-10-17T18:14:26.490 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.491 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.491 DEBUG:boto:Params: {} 2023-10-17T18:14:26.492 DEBUG:boto:Token: None 2023-10-17T18:14:26.493 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo= 2023-10-17T18:14:26.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo=', 'Content-Length': '0'} 2023-10-17T18:14:26.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dd99827d9c2c36b-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.497 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.497 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.498 DEBUG:boto:Method: DELETE 2023-10-17T18:14:26.498 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.499 DEBUG:boto:Data: 2023-10-17T18:14:26.499 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.500 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.500 DEBUG:boto:Params: {} 2023-10-17T18:14:26.500 DEBUG:boto:Token: None 2023-10-17T18:14:26.501 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3Lcq1vMneweBVMHzJbY0UiMp/m0= 2023-10-17T18:14:26.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3Lcq1vMneweBVMHzJbY0UiMp/m0=', 'Content-Length': '0'} 2023-10-17T18:14:26.527 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e164e20fc6fd5d3-00652ecf02-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.528 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.528 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.529 DEBUG:boto:Method: DELETE 2023-10-17T18:14:26.529 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.529 DEBUG:boto:Data: 2023-10-17T18:14:26.530 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.531 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.531 DEBUG:boto:Params: {} 2023-10-17T18:14:26.531 DEBUG:boto:Token: None 2023-10-17T18:14:26.532 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U0Jo1wJa+5ZSIdvLFDk2mXgkGwo= 2023-10-17T18:14:26.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U0Jo1wJa+5ZSIdvLFDk2mXgkGwo=', 'Content-Length': '0'} 2023-10-17T18:14:26.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000964d5cd908e24770-00652ecf02-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.586 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.665 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:26.676 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.562431_100.1", 2023-10-17T18:14:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:26.562431Z" 2023-10-17T18:14:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.680 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:26.681 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566466.562431_100.1', 2: '', 3: ''} 2023-10-17T18:14:26.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:26.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:26.682 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:26.735 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:26.737 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:26.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.765+0000 7ffb2d68e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:26.766+0000 7ffb2d68e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:26.779 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:26.779 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:26.779 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:26.779 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:26.780 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:26.780 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:26.780 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.780 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:26.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.781 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:26.781 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.781 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.781 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:26.782 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.782 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:26.782 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.782 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:26.782 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.783 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.783 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:26.783 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.562431_100.1", 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.784 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:26.562431Z", 2023-10-17T18:14:26.785 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.785 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.786 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.787 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.787 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.787 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:26.788 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:26.790 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:26.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:26.791 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:26Z" 2023-10-17T18:14:26.791 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:26.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566466.562431_100.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:26.562431Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:26Z" } 2023-10-17T18:14:26.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566466.562431_100.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:26.562431Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:26.793 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.562431_100.1', 2: '', 3: ''} 2023-10-17T18:14:26.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.562431_100.1', 2: '', 3: ''} 2023-10-17T18:14:26.793 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:26.794 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.794 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.794 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.795 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.795 DEBUG:boto:Data: 2023-10-17T18:14:26.795 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.796 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.796 DEBUG:boto:Params: {} 2023-10-17T18:14:26.797 DEBUG:boto:Token: None 2023-10-17T18:14:26.797 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc= 2023-10-17T18:14:26.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc=', 'Content-Length': '0'} 2023-10-17T18:14:26.800 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000000a70dce345a84ef9-00652ecf02-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.800 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.800 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.801 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.801 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.801 DEBUG:boto:Data: 2023-10-17T18:14:26.802 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.803 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.803 DEBUG:boto:Params: {} 2023-10-17T18:14:26.803 DEBUG:boto:Token: None 2023-10-17T18:14:26.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo= 2023-10-17T18:14:26.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo=', 'Content-Length': '0'} 2023-10-17T18:14:26.806 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000008b9f584ce6414804-00652ecf02-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.806 DEBUG:boto:path=/hbmuog-5/ 2023-10-17T18:14:26.806 DEBUG:boto:auth_path=/hbmuog-5/ 2023-10-17T18:14:26.807 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.807 DEBUG:boto:Path: /hbmuog-5/ 2023-10-17T18:14:26.808 DEBUG:boto:Data: 2023-10-17T18:14:26.808 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.808 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.809 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.809 DEBUG:boto:Params: {} 2023-10-17T18:14:26.809 DEBUG:boto:Token: None 2023-10-17T18:14:26.810 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-5/ 2023-10-17T18:14:26.810 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc= 2023-10-17T18:14:26.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lfVwsvi/3+3D9W2RdlLonDsK1Nc=', 'Content-Length': '0'} 2023-10-17T18:14:26.812 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000009ce5a516dde1bc4-00652ecf02-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.813 DEBUG:boto:path=/hbmuog-6/ 2023-10-17T18:14:26.813 DEBUG:boto:auth_path=/hbmuog-6/ 2023-10-17T18:14:26.813 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.814 DEBUG:boto:Path: /hbmuog-6/ 2023-10-17T18:14:26.814 DEBUG:boto:Data: 2023-10-17T18:14:26.815 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.815 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.815 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.816 DEBUG:boto:Params: {} 2023-10-17T18:14:26.816 DEBUG:boto:Token: None 2023-10-17T18:14:26.816 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-6/ 2023-10-17T18:14:26.817 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo= 2023-10-17T18:14:26.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u43s2PlSDGkvp1Q2Ccvb22Ku5mo=', 'Content-Length': '0'} 2023-10-17T18:14:26.818 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000004bd0b9a9b95b0ab-00652ecf02-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-10-17T18:14:26.819 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-7 2023-10-17T18:14:26.820 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:14:26.820 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:14:26.821 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.821 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:14:26.821 DEBUG:boto:Data: 2023-10-17T18:14:26.822 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.822 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.823 DEBUG:boto:Params: {} 2023-10-17T18:14:26.823 DEBUG:boto:Token: None 2023-10-17T18:14:26.824 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/ 2023-10-17T18:14:26.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PWos3+3unKWeJLKkn6cAgunnee8= 2023-10-17T18:14:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PWos3+3unKWeJLKkn6cAgunnee8=', 'Content-Length': '0'} 2023-10-17T18:14:26.836 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8cb1527d68b66e5-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.837 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-8 2023-10-17T18:14:26.838 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:14:26.838 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:14:26.839 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.839 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:14:26.839 DEBUG:boto:Data: 2023-10-17T18:14:26.840 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.840 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:26.841 DEBUG:boto:Port: 8001 2023-10-17T18:14:26.841 DEBUG:boto:Params: {} 2023-10-17T18:14:26.842 DEBUG:boto:Token: None 2023-10-17T18:14:26.843 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-8/ 2023-10-17T18:14:26.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UKY2mMwnixV5ozjY3M07AFKRj1g= 2023-10-17T18:14:26.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UKY2mMwnixV5ozjY3M07AFKRj1g=', 'Content-Length': '0'} 2023-10-17T18:14:26.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f49ef6060ddf523-00652ecf02-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.864 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:14:26.865 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:14:26.865 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.865 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:14:26.866 DEBUG:boto:Data: 2023-10-17T18:14:26.866 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.867 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.868 DEBUG:boto:Params: {} 2023-10-17T18:14:26.868 DEBUG:boto:Token: None 2023-10-17T18:14:26.869 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/ 2023-10-17T18:14:26.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg= 2023-10-17T18:14:26.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg=', 'Content-Length': '0'} 2023-10-17T18:14:26.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c462b02744ae3a33-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.873 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:14:26.913 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:14:26.914 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.915 DEBUG:boto:Path: /hbmuog-7/myobj 2023-10-17T18:14:26.915 DEBUG:boto:Data: 2023-10-17T18:14:26.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:26.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.916 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.917 DEBUG:boto:Params: {} 2023-10-17T18:14:26.917 DEBUG:boto:Token: None 2023-10-17T18:14:26.918 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/myobj 2023-10-17T18:14:26.918 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fy9hWG8XvLlHWMuHQG8alWQOAJA= 2023-10-17T18:14:26.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fy9hWG8XvLlHWMuHQG8alWQOAJA='} 2023-10-17T18:14:26.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8b9b09dab115677-00652ecf02-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.929 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:14:26.929 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:14:26.930 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.930 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:14:26.930 DEBUG:boto:Data: 2023-10-17T18:14:26.931 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.931 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.932 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.932 DEBUG:boto:Params: {} 2023-10-17T18:14:26.932 DEBUG:boto:Token: None 2023-10-17T18:14:26.933 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/ 2023-10-17T18:14:26.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg= 2023-10-17T18:14:26.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg=', 'Content-Length': '0'} 2023-10-17T18:14:26.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064853dfc3d3d8e31-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.937 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:14:26.937 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:14:26.938 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.938 DEBUG:boto:Path: /hbmuog-7/_myobj 2023-10-17T18:14:26.938 DEBUG:boto:Data: 2023-10-17T18:14:26.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:26.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.940 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.940 DEBUG:boto:Params: {} 2023-10-17T18:14:26.940 DEBUG:boto:Token: None 2023-10-17T18:14:26.941 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/_myobj 2023-10-17T18:14:26.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:esz1QqG4lIi1/E9i8mYxfdDAesA= 2023-10-17T18:14:26.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:esz1QqG4lIi1/E9i8mYxfdDAesA='} 2023-10-17T18:14:26.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001eb1e83de9453d0e-00652ecf02-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.951 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:14:26.952 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:14:26.952 DEBUG:boto:Method: HEAD 2023-10-17T18:14:26.952 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:14:26.953 DEBUG:boto:Data: 2023-10-17T18:14:26.953 DEBUG:boto:Headers: {} 2023-10-17T18:14:26.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.954 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.954 DEBUG:boto:Params: {} 2023-10-17T18:14:26.955 DEBUG:boto:Token: None 2023-10-17T18:14:26.955 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/ 2023-10-17T18:14:26.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg= 2023-10-17T18:14:26.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Ay86Zt8TAQQu9Fzt4BU14kvEvg=', 'Content-Length': '0'} 2023-10-17T18:14:26.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b4fc790384968b-00652ecf02-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:26.959 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:14:26.959 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:14:26.959 DEBUG:boto:Method: PUT 2023-10-17T18:14:26.960 DEBUG:boto:Path: /hbmuog-7/%3A 2023-10-17T18:14:26.960 DEBUG:boto:Data: 2023-10-17T18:14:26.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:26.961 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:26.961 DEBUG:boto:Port: 8000 2023-10-17T18:14:26.962 DEBUG:boto:Params: {} 2023-10-17T18:14:26.962 DEBUG:boto:Token: None 2023-10-17T18:14:26.963 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:26 GMT /hbmuog-7/%3A 2023-10-17T18:14:26.963 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V67nmifIt0g2DIAzoP9aeFgCN0o= 2023-10-17T18:14:26.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:26 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V67nmifIt0g2DIAzoP9aeFgCN0o='} 2023-10-17T18:14:27.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af5a82bd011342b0-00652ecf02-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.007 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:14:27.008 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:14:27.008 DEBUG:boto:Method: HEAD 2023-10-17T18:14:27.008 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:14:27.009 DEBUG:boto:Data: 2023-10-17T18:14:27.009 DEBUG:boto:Headers: {} 2023-10-17T18:14:27.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:27.010 DEBUG:boto:Port: 8000 2023-10-17T18:14:27.010 DEBUG:boto:Params: {} 2023-10-17T18:14:27.011 DEBUG:boto:Token: None 2023-10-17T18:14:27.011 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-7/ 2023-10-17T18:14:27.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V0rH6EDBmuy/hi5zIFtO6E8fDc0= 2023-10-17T18:14:27.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V0rH6EDBmuy/hi5zIFtO6E8fDc0=', 'Content-Length': '0'} 2023-10-17T18:14:27.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a0b54ebb3bb5a86-00652ecf03-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.015 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:14:27.015 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:14:27.016 DEBUG:boto:Method: PUT 2023-10-17T18:14:27.016 DEBUG:boto:Path: /hbmuog-7/%26 2023-10-17T18:14:27.016 DEBUG:boto:Data: 2023-10-17T18:14:27.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:27.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:14:27.017 DEBUG:boto:Port: 8000 2023-10-17T18:14:27.018 DEBUG:boto:Params: {} 2023-10-17T18:14:27.018 DEBUG:boto:Token: None 2023-10-17T18:14:27.019 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-7/%26 2023-10-17T18:14:27.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eLrjHSQD5AwI7F16yo6gy2MrczA= 2023-10-17T18:14:27.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eLrjHSQD5AwI7F16yo6gy2MrczA='} 2023-10-17T18:14:27.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037c6ee0ceb523344-00652ecf03-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.025 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:14:27.025 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:14:27.025 DEBUG:boto:Method: HEAD 2023-10-17T18:14:27.026 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:14:27.026 DEBUG:boto:Data: 2023-10-17T18:14:27.026 DEBUG:boto:Headers: {} 2023-10-17T18:14:27.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.027 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.027 DEBUG:boto:Params: {} 2023-10-17T18:14:27.028 DEBUG:boto:Token: None 2023-10-17T18:14:27.028 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/ 2023-10-17T18:14:27.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU= 2023-10-17T18:14:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU=', 'Content-Length': '0'} 2023-10-17T18:14:27.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d61e85ed035e69a0-00652ecf03-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.031 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:14:27.032 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:14:27.032 DEBUG:boto:Method: PUT 2023-10-17T18:14:27.033 DEBUG:boto:Path: /hbmuog-8/myobj 2023-10-17T18:14:27.033 DEBUG:boto:Data: 2023-10-17T18:14:27.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:27.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.034 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.034 DEBUG:boto:Params: {} 2023-10-17T18:14:27.035 DEBUG:boto:Token: None 2023-10-17T18:14:27.035 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/myobj 2023-10-17T18:14:27.036 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xw4Ze7PJLpC+ozUyXa4ocpGLV54= 2023-10-17T18:14:27.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xw4Ze7PJLpC+ozUyXa4ocpGLV54='} 2023-10-17T18:14:27.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eeb71b8bb40a1fd1-00652ecf03-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.079 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:14:27.079 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:14:27.080 DEBUG:boto:Method: HEAD 2023-10-17T18:14:27.080 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:14:27.080 DEBUG:boto:Data: 2023-10-17T18:14:27.081 DEBUG:boto:Headers: {} 2023-10-17T18:14:27.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.081 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.082 DEBUG:boto:Params: {} 2023-10-17T18:14:27.082 DEBUG:boto:Token: None 2023-10-17T18:14:27.082 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/ 2023-10-17T18:14:27.083 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU= 2023-10-17T18:14:27.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU=', 'Content-Length': '0'} 2023-10-17T18:14:27.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000811d70d4e341aae7-00652ecf03-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.086 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:14:27.086 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:14:27.087 DEBUG:boto:Method: PUT 2023-10-17T18:14:27.087 DEBUG:boto:Path: /hbmuog-8/_myobj 2023-10-17T18:14:27.087 DEBUG:boto:Data: 2023-10-17T18:14:27.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:27.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.088 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.089 DEBUG:boto:Params: {} 2023-10-17T18:14:27.089 DEBUG:boto:Token: None 2023-10-17T18:14:27.090 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/_myobj 2023-10-17T18:14:27.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KYlEIARsTf6fgpR8VFCHoS0V1Es= 2023-10-17T18:14:27.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KYlEIARsTf6fgpR8VFCHoS0V1Es='} 2023-10-17T18:14:27.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e618b61128531211-00652ecf03-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.099 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:14:27.099 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:14:27.099 DEBUG:boto:Method: HEAD 2023-10-17T18:14:27.100 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:14:27.100 DEBUG:boto:Data: 2023-10-17T18:14:27.100 DEBUG:boto:Headers: {} 2023-10-17T18:14:27.101 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.101 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.101 DEBUG:boto:Params: {} 2023-10-17T18:14:27.102 DEBUG:boto:Token: None 2023-10-17T18:14:27.102 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/ 2023-10-17T18:14:27.103 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU= 2023-10-17T18:14:27.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU=', 'Content-Length': '0'} 2023-10-17T18:14:27.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000275d4f81a0a8eb9b-00652ecf03-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.106 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:14:27.106 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:14:27.107 DEBUG:boto:Method: PUT 2023-10-17T18:14:27.107 DEBUG:boto:Path: /hbmuog-8/%3A 2023-10-17T18:14:27.107 DEBUG:boto:Data: 2023-10-17T18:14:27.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:27.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.109 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.109 DEBUG:boto:Params: {} 2023-10-17T18:14:27.109 DEBUG:boto:Token: None 2023-10-17T18:14:27.110 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/%3A 2023-10-17T18:14:27.110 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/e3CzkAEew3ejjU5LN7bWccMK4= 2023-10-17T18:14:27.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/e3CzkAEew3ejjU5LN7bWccMK4='} 2023-10-17T18:14:27.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c3fa82add7654b5-00652ecf03-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.119 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:14:27.119 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:14:27.119 DEBUG:boto:Method: HEAD 2023-10-17T18:14:27.120 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:14:27.120 DEBUG:boto:Data: 2023-10-17T18:14:27.120 DEBUG:boto:Headers: {} 2023-10-17T18:14:27.121 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.121 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.121 DEBUG:boto:Params: {} 2023-10-17T18:14:27.122 DEBUG:boto:Token: None 2023-10-17T18:14:27.122 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/ 2023-10-17T18:14:27.123 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU= 2023-10-17T18:14:27.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:50G4m3glhyEiP5PZ9PMI4XpAWsU=', 'Content-Length': '0'} 2023-10-17T18:14:27.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082595881661fe988-00652ecf03-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.126 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:14:27.126 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:14:27.126 DEBUG:boto:Method: PUT 2023-10-17T18:14:27.127 DEBUG:boto:Path: /hbmuog-8/%26 2023-10-17T18:14:27.127 DEBUG:boto:Data: 2023-10-17T18:14:27.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:14:27.128 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:14:27.128 DEBUG:boto:Port: 8001 2023-10-17T18:14:27.129 DEBUG:boto:Params: {} 2023-10-17T18:14:27.129 DEBUG:boto:Token: None 2023-10-17T18:14:27.129 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:14:27 GMT /hbmuog-8/%26 2023-10-17T18:14:27.130 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:onBPjINeevvOKUTmZNh0a9weIww= 2023-10-17T18:14:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:14:27 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:onBPjINeevvOKUTmZNh0a9weIww='} 2023-10-17T18:14:27.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c45867b6fea3f9c9-00652ecf03-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:14:27.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:27.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:27.176 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:27.226 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:27.226 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:27.255 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:14:27.267 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:14:27.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:12:15.454808Z" 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.856132_108.1", 2023-10-17T18:14:27.268 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:26.856132Z" 2023-10-17T18:14:27.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.269 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:27.269 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:27.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.270 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:27.270 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:14:27.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.270 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:14:27.271 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566335.454808_63.1', 1: '1_1697566466.856132_108.1', 2: '', 3: ''} 2023-10-17T18:14:27.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:14:27.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:27.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:14:27.273 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:14:27.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:27.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:27.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:27.351+0000 7f3707653780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:27.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:27.352+0000 7f3707653780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:14:27.364 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:14:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:14:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:14:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:14:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:14:27.365 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:14:27.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:27.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.366 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:14:27.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.367 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:14:27.367 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:27.367 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:27.367 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566335.454808_63.1", 2023-10-17T18:14:27.367 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:27.368 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:14:27.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:27.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:12:15.454808Z", 2023-10-17T18:14:27.368 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:27.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.369 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:14:27.369 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:27.370 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:27.370 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566466.856132_108.1", 2023-10-17T18:14:27.370 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:27.370 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:27.371 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:27.371 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:26.856132Z", 2023-10-17T18:14:27.371 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:27.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.372 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:14:27.372 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:27.373 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:27.373 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:27.373 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:27.374 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:27.374 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:27.374 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:27.374 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:27.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:14:27.375 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:14:27.375 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:14:27.376 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:14:27.376 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:14:27.376 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:14:27.376 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:14:27.377 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:14:27.377 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:14:27.377 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:14:27.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:14:27.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:14:27.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.378 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:14:27.378 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:14:27.378 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:14:27.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:14:27.379 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:14:27Z" 2023-10-17T18:14:27.379 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:14:27.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566335.454808_63.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:12:15.454808Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566466.856132_108.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:26.856132Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:14:27Z" } 2023-10-17T18:14:27.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566335.454808_63.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:12:15.454808Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566466.856132_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:26.856132Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:14:27.381 DEBUG:rgw_multi.tests:log_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.856132_108.1', 2: '', 3: ''} 2023-10-17T18:14:27.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566335.454808_63.1', 1: '1_1697566466.856132_108.1', 2: '', 3: ''} 2023-10-17T18:14:27.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:14:27.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:14:27.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:14:27.383 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:14:27.441 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:14:27.441 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:14:27.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:27.473+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:32.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:32.474+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:37.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:37.476+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:42.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:42.476+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:47.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:47.476+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:52.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:52.477+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:14:57.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:14:57.477+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:02.478+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:07.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:07.479+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:12.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:12.480+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:17.481+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:22.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:22.480+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:27.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:27.481+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:32.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:32.481+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:37.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:37.482+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:42.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:42.483+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:47.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:47.484+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:52.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:52.485+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:15:57.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:15:57.486+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:02.486+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:07.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:07.486+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:12.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:12.487+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:17.488+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:22.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:22.489+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:27.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:27.490+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:32.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:32.491+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:37.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:37.490+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:42.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:42.491+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:47.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:47.492+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:52.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:52.493+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:16:57.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:16:57.493+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:02.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:02.495+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:07.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:07.495+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:12.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:12.495+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:17.495+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:22.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:22.496+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:27.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:27.497+0000 7f73a82b9780 1 waiting to reach incremental sync.. 2023-10-17T18:17:32.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:32.499+0000 7f73a82b9780 1 bucket sync caught up with source: 2023-10-17T18:17:32.501 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.3.6, , ] 2023-10-17T18:17:32.501 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.6.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.3.6, , ] 2023-10-17T18:17:32.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:32.499+0000 7f73a82b9780 0 bucket checkpoint complete 2023-10-17T18:17:32.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-7 zones={a2, a1} 2023-10-17T18:17:32.517 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.517 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.518 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.518 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:17:32.519 DEBUG:boto:Data: 2023-10-17T18:17:32.519 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.520 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.520 DEBUG:boto:Params: {} 2023-10-17T18:17:32.521 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:17:32.522 DEBUG:boto:Token: None 2023-10-17T18:17:32.522 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vfI+h/CQZn3fbQ9KykgRMJsKQYQ= 2023-10-17T18:17:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vfI+h/CQZn3fbQ9KykgRMJsKQYQ=', 'Content-Length': '0'} 2023-10-17T18:17:32.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d5528c5ae7c91fa-00652ecfbc-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.529 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.529 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.529 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.530 DEBUG:boto:Path: /hbmuog-7/ 2023-10-17T18:17:32.530 DEBUG:boto:Data: 2023-10-17T18:17:32.531 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.531 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.532 DEBUG:boto:Params: {} 2023-10-17T18:17:32.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:17:32.533 DEBUG:boto:Token: None 2023-10-17T18:17:32.533 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.534 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vfI+h/CQZn3fbQ9KykgRMJsKQYQ= 2023-10-17T18:17:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vfI+h/CQZn3fbQ9KykgRMJsKQYQ=', 'Content-Length': '0'} 2023-10-17T18:17:32.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcc4b8363b23f212-00652ecfbc-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.538 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:17:32.539 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.621 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.621 DEBUG:boto:path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.621 DEBUG:boto:auth_path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.622 DEBUG:boto:Method: GET 2023-10-17T18:17:32.622 DEBUG:boto:Path: /hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.623 DEBUG:boto:Data: 2023-10-17T18:17:32.623 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.624 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.624 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.625 DEBUG:boto:Params: {} 2023-10-17T18:17:32.625 DEBUG:boto:Token: None 2023-10-17T18:17:32.626 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/?versions 2023-10-17T18:17:32.626 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU= 2023-10-17T18:17:32.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU=', 'Content-Length': '0'} 2023-10-17T18:17:32.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f18359f00719728-00652ecfbc-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.630 DEBUG:boto:b'hbmuog-7999false&nulltrue2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.681 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-17T18:17:32.681 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-17T18:17:32.682 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-17T18:17:32.682 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:32.683 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:17:32.683 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.683 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.684 DEBUG:boto:path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.684 DEBUG:boto:auth_path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.685 DEBUG:boto:Method: GET 2023-10-17T18:17:32.685 DEBUG:boto:Path: /hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.685 DEBUG:boto:Data: 2023-10-17T18:17:32.686 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.687 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.687 DEBUG:boto:Params: {} 2023-10-17T18:17:32.688 DEBUG:boto:Token: None 2023-10-17T18:17:32.688 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/?versions 2023-10-17T18:17:32.689 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU= 2023-10-17T18:17:32.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU=', 'Content-Length': '0'} 2023-10-17T18:17:32.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b69d39123da9bf5a-00652ecfbc-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.692 DEBUG:boto:b'hbmuog-7999false&nulltrue2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.693 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-17T18:17:32.693 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-17T18:17:32.694 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-17T18:17:32.694 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:32.695 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.695 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.695 DEBUG:boto:path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.696 DEBUG:boto:auth_path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.696 DEBUG:boto:Method: GET 2023-10-17T18:17:32.697 DEBUG:boto:Path: /hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.697 DEBUG:boto:Data: 2023-10-17T18:17:32.697 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.698 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.699 DEBUG:boto:Params: {} 2023-10-17T18:17:32.699 DEBUG:boto:Token: None 2023-10-17T18:17:32.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/?versions 2023-10-17T18:17:32.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU= 2023-10-17T18:17:32.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU=', 'Content-Length': '0'} 2023-10-17T18:17:32.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfae52a0dce71f8-00652ecfbc-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.703 DEBUG:boto:b'hbmuog-7999false&nulltrue2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.704 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.705 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.705 DEBUG:boto:path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.705 DEBUG:boto:auth_path=/hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.706 DEBUG:boto:Method: GET 2023-10-17T18:17:32.706 DEBUG:boto:Path: /hbmuog-7/?versions&max-keys=999 2023-10-17T18:17:32.707 DEBUG:boto:Data: 2023-10-17T18:17:32.707 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.708 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.708 DEBUG:boto:Params: {} 2023-10-17T18:17:32.709 DEBUG:boto:Token: None 2023-10-17T18:17:32.709 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/?versions 2023-10-17T18:17:32.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU= 2023-10-17T18:17:32.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9QxADUe7iTf8G69cU9zHbA0ijtU=', 'Content-Length': '0'} 2023-10-17T18:17:32.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad98ee2f702c9fd-00652ecfbc-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.713 DEBUG:boto:b'hbmuog-7999false&nulltrue2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:32.714 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.715 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.715 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.716 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.716 DEBUG:boto:Method: GET 2023-10-17T18:17:32.716 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.717 DEBUG:boto:Data: 2023-10-17T18:17:32.717 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.718 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.718 DEBUG:boto:Params: {} 2023-10-17T18:17:32.719 DEBUG:boto:Token: None 2023-10-17T18:17:32.719 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI= 2023-10-17T18:17:32.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI=', 'Content-Length': '0'} 2023-10-17T18:17:32.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000123b1e8985ba986-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.723 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.724 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.724 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.724 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.725 DEBUG:boto:Method: GET 2023-10-17T18:17:32.725 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.726 DEBUG:boto:Data: 2023-10-17T18:17:32.726 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.727 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.727 DEBUG:boto:Params: {} 2023-10-17T18:17:32.728 DEBUG:boto:Token: None 2023-10-17T18:17:32.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.729 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI= 2023-10-17T18:17:32.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI=', 'Content-Length': '0'} 2023-10-17T18:17:32.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000913a28fabdcaaa10-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.732 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.732 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.733 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.733 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.733 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.734 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.734 DEBUG:boto:Data: 2023-10-17T18:17:32.735 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.735 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.735 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.736 DEBUG:boto:Params: {} 2023-10-17T18:17:32.736 DEBUG:boto:Token: None 2023-10-17T18:17:32.737 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.737 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfXZs8pajEzWWOAZka7Wd4uG250= 2023-10-17T18:17:32.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfXZs8pajEzWWOAZka7Wd4uG250=', 'Content-Length': '0'} 2023-10-17T18:17:32.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a86b36610bd414fb-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.740 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.740 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.741 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.741 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.742 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.742 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.742 DEBUG:boto:Data: 2023-10-17T18:17:32.743 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.744 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.744 DEBUG:boto:Params: {} 2023-10-17T18:17:32.745 DEBUG:boto:Token: None 2023-10-17T18:17:32.745 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfXZs8pajEzWWOAZka7Wd4uG250= 2023-10-17T18:17:32.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfXZs8pajEzWWOAZka7Wd4uG250=', 'Content-Length': '0'} 2023-10-17T18:17:32.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eae85da84f1bcda6-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:32.750 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.751 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.751 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.751 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.752 DEBUG:boto:Method: GET 2023-10-17T18:17:32.752 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.753 DEBUG:boto:Data: 2023-10-17T18:17:32.753 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.754 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.754 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.754 DEBUG:boto:Params: {} 2023-10-17T18:17:32.755 DEBUG:boto:Token: None 2023-10-17T18:17:32.755 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.756 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI= 2023-10-17T18:17:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI=', 'Content-Length': '0'} 2023-10-17T18:17:32.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096b84da5e104ca20-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.759 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.759 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.760 DEBUG:boto:path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.760 DEBUG:boto:auth_path=/hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.761 DEBUG:boto:Method: GET 2023-10-17T18:17:32.761 DEBUG:boto:Path: /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.761 DEBUG:boto:Data: 2023-10-17T18:17:32.762 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.762 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.763 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.763 DEBUG:boto:Params: {} 2023-10-17T18:17:32.764 DEBUG:boto:Token: None 2023-10-17T18:17:32.764 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26?versionId=null 2023-10-17T18:17:32.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI= 2023-10-17T18:17:32.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TxB1ZaBSVmvOX/e4wUN/91EJTkI=', 'Content-Length': '0'} 2023-10-17T18:17:32.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f52446e66cd17b0-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.768 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.768 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.769 DEBUG:boto:path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.769 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.770 DEBUG:boto:Method: GET 2023-10-17T18:17:32.770 DEBUG:boto:Path: /hbmuog-7/?prefix=%26 2023-10-17T18:17:32.770 DEBUG:boto:Data: 2023-10-17T18:17:32.771 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.772 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.772 DEBUG:boto:Params: {} 2023-10-17T18:17:32.773 DEBUG:boto:Token: None 2023-10-17T18:17:32.773 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6a645efc1e63c7-00652ecfbc-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.818 DEBUG:boto:b'hbmuog-7&1000false&2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.819 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.820 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.820 DEBUG:boto:path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.821 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.821 DEBUG:boto:Method: GET 2023-10-17T18:17:32.821 DEBUG:boto:Path: /hbmuog-7/?prefix=%26 2023-10-17T18:17:32.822 DEBUG:boto:Data: 2023-10-17T18:17:32.822 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.823 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.823 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.823 DEBUG:boto:Params: {} 2023-10-17T18:17:32.824 DEBUG:boto:Token: None 2023-10-17T18:17:32.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029dc089231fdde6f-00652ecfbc-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.829 DEBUG:boto:b'hbmuog-7&1000false&2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.829 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.830 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.830 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.831 DEBUG:boto:Path: /hbmuog-7/%26 2023-10-17T18:17:32.831 DEBUG:boto:Data: 2023-10-17T18:17:32.831 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.832 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.832 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.833 DEBUG:boto:Params: {} 2023-10-17T18:17:32.833 DEBUG:boto:Token: None 2023-10-17T18:17:32.833 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26 2023-10-17T18:17:32.834 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ep7fkeJWIbiCgD1VZ88sI/znDKU= 2023-10-17T18:17:32.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ep7fkeJWIbiCgD1VZ88sI/znDKU=', 'Content-Length': '0'} 2023-10-17T18:17:32.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a922b4bfa68c4df-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.837 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.837 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.838 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.838 DEBUG:boto:Path: /hbmuog-7/%26 2023-10-17T18:17:32.839 DEBUG:boto:Data: 2023-10-17T18:17:32.839 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.840 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.840 DEBUG:boto:Params: {} 2023-10-17T18:17:32.841 DEBUG:boto:Token: None 2023-10-17T18:17:32.841 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26 2023-10-17T18:17:32.842 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ep7fkeJWIbiCgD1VZ88sI/znDKU= 2023-10-17T18:17:32.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ep7fkeJWIbiCgD1VZ88sI/znDKU=', 'Content-Length': '0'} 2023-10-17T18:17:32.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f221a11534cb53-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:32.845 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.845 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.846 DEBUG:boto:Method: GET 2023-10-17T18:17:32.846 DEBUG:boto:Path: /hbmuog-7/%26 2023-10-17T18:17:32.847 DEBUG:boto:Data: 2023-10-17T18:17:32.847 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.848 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.848 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.849 DEBUG:boto:Params: {} 2023-10-17T18:17:32.849 DEBUG:boto:Token: None 2023-10-17T18:17:32.850 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26 2023-10-17T18:17:32.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N+s8sb2zozxObKltnAW/SYRFQcI= 2023-10-17T18:17:32.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N+s8sb2zozxObKltnAW/SYRFQcI=', 'Content-Length': '0'} 2023-10-17T18:17:32.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026543920ad94cf81-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.854 DEBUG:boto:path=/hbmuog-7/%26 2023-10-17T18:17:32.854 DEBUG:boto:auth_path=/hbmuog-7/%26 2023-10-17T18:17:32.855 DEBUG:boto:Method: GET 2023-10-17T18:17:32.855 DEBUG:boto:Path: /hbmuog-7/%26 2023-10-17T18:17:32.855 DEBUG:boto:Data: 2023-10-17T18:17:32.856 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.857 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.857 DEBUG:boto:Params: {} 2023-10-17T18:17:32.858 DEBUG:boto:Token: None 2023-10-17T18:17:32.858 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%26 2023-10-17T18:17:32.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N+s8sb2zozxObKltnAW/SYRFQcI= 2023-10-17T18:17:32.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N+s8sb2zozxObKltnAW/SYRFQcI=', 'Content-Length': '0'} 2023-10-17T18:17:32.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba40039398589104-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.862 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.862 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.863 DEBUG:boto:path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.863 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.863 DEBUG:boto:Method: GET 2023-10-17T18:17:32.864 DEBUG:boto:Path: /hbmuog-7/?prefix=%26 2023-10-17T18:17:32.864 DEBUG:boto:Data: 2023-10-17T18:17:32.865 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.866 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.866 DEBUG:boto:Params: {} 2023-10-17T18:17:32.866 DEBUG:boto:Token: None 2023-10-17T18:17:32.867 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a7baf54f2798a62-00652ecfbc-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.871 DEBUG:boto:b'hbmuog-7&1000false&2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.872 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.872 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.872 DEBUG:boto:path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.873 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%26 2023-10-17T18:17:32.873 DEBUG:boto:Method: GET 2023-10-17T18:17:32.874 DEBUG:boto:Path: /hbmuog-7/?prefix=%26 2023-10-17T18:17:32.874 DEBUG:boto:Data: 2023-10-17T18:17:32.875 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.875 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.876 DEBUG:boto:Params: {} 2023-10-17T18:17:32.876 DEBUG:boto:Token: None 2023-10-17T18:17:32.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.877 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec03432e6455fb2a-00652ecfbc-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.881 DEBUG:boto:b'hbmuog-7&1000false&2023-10-17T18:14:27.021Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:32.882 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.882 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.883 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.883 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.884 DEBUG:boto:Method: GET 2023-10-17T18:17:32.884 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.884 DEBUG:boto:Data: 2023-10-17T18:17:32.885 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.886 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.886 DEBUG:boto:Params: {} 2023-10-17T18:17:32.887 DEBUG:boto:Token: None 2023-10-17T18:17:32.887 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U= 2023-10-17T18:17:32.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U=', 'Content-Length': '0'} 2023-10-17T18:17:32.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a50df1d3d8c9d820-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.891 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.892 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.892 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.892 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.893 DEBUG:boto:Method: GET 2023-10-17T18:17:32.893 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.894 DEBUG:boto:Data: 2023-10-17T18:17:32.894 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.895 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.895 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.896 DEBUG:boto:Params: {} 2023-10-17T18:17:32.896 DEBUG:boto:Token: None 2023-10-17T18:17:32.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.897 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U= 2023-10-17T18:17:32.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U=', 'Content-Length': '0'} 2023-10-17T18:17:32.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009750e0b3205ef37e-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.900 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.900 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.901 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.901 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.902 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.902 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.903 DEBUG:boto:Data: 2023-10-17T18:17:32.903 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.903 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.904 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.904 DEBUG:boto:Params: {} 2023-10-17T18:17:32.905 DEBUG:boto:Token: None 2023-10-17T18:17:32.905 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.906 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GJbFSliRorKMBNEUWlZQT37agm8= 2023-10-17T18:17:32.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GJbFSliRorKMBNEUWlZQT37agm8=', 'Content-Length': '0'} 2023-10-17T18:17:32.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008c8e13f5554a3de-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.909 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.909 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.910 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.910 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.910 DEBUG:boto:Method: HEAD 2023-10-17T18:17:32.911 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.911 DEBUG:boto:Data: 2023-10-17T18:17:32.912 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.912 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.912 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.913 DEBUG:boto:Params: {} 2023-10-17T18:17:32.913 DEBUG:boto:Token: None 2023-10-17T18:17:32.914 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.914 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GJbFSliRorKMBNEUWlZQT37agm8= 2023-10-17T18:17:32.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GJbFSliRorKMBNEUWlZQT37agm8=', 'Content-Length': '0'} 2023-10-17T18:17:32.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aae384085256fec9-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:32.917 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.918 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.918 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.919 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.919 DEBUG:boto:Method: GET 2023-10-17T18:17:32.919 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.920 DEBUG:boto:Data: 2023-10-17T18:17:32.920 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.921 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.922 DEBUG:boto:Params: {} 2023-10-17T18:17:32.922 DEBUG:boto:Token: None 2023-10-17T18:17:32.923 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U= 2023-10-17T18:17:32.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U=', 'Content-Length': '0'} 2023-10-17T18:17:32.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f61a452695008c6-00652ecfbc-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.966 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:32.967 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:32.967 DEBUG:boto:path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.968 DEBUG:boto:auth_path=/hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.968 DEBUG:boto:Method: GET 2023-10-17T18:17:32.969 DEBUG:boto:Path: /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.969 DEBUG:boto:Data: 2023-10-17T18:17:32.970 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.971 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.971 DEBUG:boto:Params: {} 2023-10-17T18:17:32.971 DEBUG:boto:Token: None 2023-10-17T18:17:32.972 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/%3A?versionId=null 2023-10-17T18:17:32.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U= 2023-10-17T18:17:32.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZrtkTom1E1dXw6tTLTxfCeKfO3U=', 'Content-Length': '0'} 2023-10-17T18:17:32.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c73bea09401ecc9-00652ecfbc-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.976 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.976 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.976 DEBUG:boto:path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.977 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.977 DEBUG:boto:Method: GET 2023-10-17T18:17:32.978 DEBUG:boto:Path: /hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.978 DEBUG:boto:Data: 2023-10-17T18:17:32.979 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.979 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:32.980 DEBUG:boto:Port: 8001 2023-10-17T18:17:32.980 DEBUG:boto:Params: {} 2023-10-17T18:17:32.981 DEBUG:boto:Token: None 2023-10-17T18:17:32.981 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.982 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a591613d40d0ade-00652ecfbc-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:32.985 DEBUG:boto:b'hbmuog-7:1000false:2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:32.986 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:32.986 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:32.987 DEBUG:boto:path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.987 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.988 DEBUG:boto:Method: GET 2023-10-17T18:17:32.988 DEBUG:boto:Path: /hbmuog-7/?prefix=%3A 2023-10-17T18:17:32.989 DEBUG:boto:Data: 2023-10-17T18:17:32.989 DEBUG:boto:Headers: {} 2023-10-17T18:17:32.990 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:32.990 DEBUG:boto:Port: 8000 2023-10-17T18:17:32.991 DEBUG:boto:Params: {} 2023-10-17T18:17:32.991 DEBUG:boto:Token: None 2023-10-17T18:17:32.992 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:32 GMT /hbmuog-7/ 2023-10-17T18:17:32.992 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90= 2023-10-17T18:17:32.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gA7mnMVNSVRPOpjfqN/LzBRo90=', 'Content-Length': '0'} 2023-10-17T18:17:32.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511b3437dd9719fa-00652ecfbc-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.038 DEBUG:boto:b'hbmuog-7:1000false:2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.039 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:33.039 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:33.040 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.040 DEBUG:boto:Path: /hbmuog-7/%3A 2023-10-17T18:17:33.041 DEBUG:boto:Data: 2023-10-17T18:17:33.041 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.041 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.042 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.042 DEBUG:boto:Params: {} 2023-10-17T18:17:33.043 DEBUG:boto:Token: None 2023-10-17T18:17:33.043 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/%3A 2023-10-17T18:17:33.044 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2WR8tq5+vN0mzS6opC8Xpyd68+8= 2023-10-17T18:17:33.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2WR8tq5+vN0mzS6opC8Xpyd68+8=', 'Content-Length': '0'} 2023-10-17T18:17:33.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f513f9061051c1b-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.047 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:33.047 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:33.048 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.048 DEBUG:boto:Path: /hbmuog-7/%3A 2023-10-17T18:17:33.049 DEBUG:boto:Data: 2023-10-17T18:17:33.049 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.049 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.050 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.051 DEBUG:boto:Params: {} 2023-10-17T18:17:33.051 DEBUG:boto:Token: None 2023-10-17T18:17:33.052 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/%3A 2023-10-17T18:17:33.052 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2WR8tq5+vN0mzS6opC8Xpyd68+8= 2023-10-17T18:17:33.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2WR8tq5+vN0mzS6opC8Xpyd68+8=', 'Content-Length': '0'} 2023-10-17T18:17:33.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a49cfbebea5db5c-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:33.056 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:33.056 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:33.057 DEBUG:boto:Method: GET 2023-10-17T18:17:33.057 DEBUG:boto:Path: /hbmuog-7/%3A 2023-10-17T18:17:33.057 DEBUG:boto:Data: 2023-10-17T18:17:33.058 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.058 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.059 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.059 DEBUG:boto:Params: {} 2023-10-17T18:17:33.060 DEBUG:boto:Token: None 2023-10-17T18:17:33.060 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/%3A 2023-10-17T18:17:33.061 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZHZI8AwymQ3S/E9DsCPMBYaf57o= 2023-10-17T18:17:33.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZHZI8AwymQ3S/E9DsCPMBYaf57o=', 'Content-Length': '0'} 2023-10-17T18:17:33.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005058614e7358a533-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.064 DEBUG:boto:path=/hbmuog-7/%3A 2023-10-17T18:17:33.064 DEBUG:boto:auth_path=/hbmuog-7/%3A 2023-10-17T18:17:33.065 DEBUG:boto:Method: GET 2023-10-17T18:17:33.065 DEBUG:boto:Path: /hbmuog-7/%3A 2023-10-17T18:17:33.065 DEBUG:boto:Data: 2023-10-17T18:17:33.066 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.067 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.067 DEBUG:boto:Params: {} 2023-10-17T18:17:33.067 DEBUG:boto:Token: None 2023-10-17T18:17:33.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/%3A 2023-10-17T18:17:33.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZHZI8AwymQ3S/E9DsCPMBYaf57o= 2023-10-17T18:17:33.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZHZI8AwymQ3S/E9DsCPMBYaf57o=', 'Content-Length': '0'} 2023-10-17T18:17:33.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb79e78672ba840d-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.072 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.072 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.073 DEBUG:boto:path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.073 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.073 DEBUG:boto:Method: GET 2023-10-17T18:17:33.074 DEBUG:boto:Path: /hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.074 DEBUG:boto:Data: 2023-10-17T18:17:33.075 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.075 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.076 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.076 DEBUG:boto:Params: {} 2023-10-17T18:17:33.077 DEBUG:boto:Token: None 2023-10-17T18:17:33.077 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8c2aab3e5a334eb-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.122 DEBUG:boto:b'hbmuog-7:1000false:2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.123 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.124 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.124 DEBUG:boto:path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.125 DEBUG:boto:auth_path=/hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.125 DEBUG:boto:Method: GET 2023-10-17T18:17:33.126 DEBUG:boto:Path: /hbmuog-7/?prefix=%3A 2023-10-17T18:17:33.126 DEBUG:boto:Data: 2023-10-17T18:17:33.126 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.127 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.128 DEBUG:boto:Params: {} 2023-10-17T18:17:33.128 DEBUG:boto:Token: None 2023-10-17T18:17:33.129 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac0d50428c1ad8d-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.133 DEBUG:boto:b'hbmuog-7:1000false:2023-10-17T18:14:26.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:33.134 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.135 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.135 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.136 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.136 DEBUG:boto:Method: GET 2023-10-17T18:17:33.137 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.137 DEBUG:boto:Data: 2023-10-17T18:17:33.137 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.138 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.139 DEBUG:boto:Params: {} 2023-10-17T18:17:33.139 DEBUG:boto:Token: None 2023-10-17T18:17:33.140 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc= 2023-10-17T18:17:33.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc=', 'Content-Length': '0'} 2023-10-17T18:17:33.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eca0be9ea805fa2b-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.143 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.144 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.144 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.145 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.145 DEBUG:boto:Method: GET 2023-10-17T18:17:33.146 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.146 DEBUG:boto:Data: 2023-10-17T18:17:33.147 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.147 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.147 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.148 DEBUG:boto:Params: {} 2023-10-17T18:17:33.148 DEBUG:boto:Token: None 2023-10-17T18:17:33.149 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc= 2023-10-17T18:17:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc=', 'Content-Length': '0'} 2023-10-17T18:17:33.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006efb3d0fa4e04f0f-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.153 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.153 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.154 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.154 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.155 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.155 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.156 DEBUG:boto:Data: 2023-10-17T18:17:33.156 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.157 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.157 DEBUG:boto:Params: {} 2023-10-17T18:17:33.158 DEBUG:boto:Token: None 2023-10-17T18:17:33.158 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.159 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WsTdJnXWlWNbB2khsNbpxJDhr+o= 2023-10-17T18:17:33.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WsTdJnXWlWNbB2khsNbpxJDhr+o=', 'Content-Length': '0'} 2023-10-17T18:17:33.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017768d9e72bd6508-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.162 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.163 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.163 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.163 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.164 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.164 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.165 DEBUG:boto:Data: 2023-10-17T18:17:33.165 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.166 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.166 DEBUG:boto:Params: {} 2023-10-17T18:17:33.167 DEBUG:boto:Token: None 2023-10-17T18:17:33.167 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.168 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WsTdJnXWlWNbB2khsNbpxJDhr+o= 2023-10-17T18:17:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WsTdJnXWlWNbB2khsNbpxJDhr+o=', 'Content-Length': '0'} 2023-10-17T18:17:33.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000899ced1914b2acec-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:33.171 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.172 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.172 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.173 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.173 DEBUG:boto:Method: GET 2023-10-17T18:17:33.174 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.174 DEBUG:boto:Data: 2023-10-17T18:17:33.174 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.175 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.176 DEBUG:boto:Params: {} 2023-10-17T18:17:33.176 DEBUG:boto:Token: None 2023-10-17T18:17:33.177 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.177 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc= 2023-10-17T18:17:33.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc=', 'Content-Length': '0'} 2023-10-17T18:17:33.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003127581dd7778863-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.181 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.181 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.182 DEBUG:boto:path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.182 DEBUG:boto:auth_path=/hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.182 DEBUG:boto:Method: GET 2023-10-17T18:17:33.183 DEBUG:boto:Path: /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.183 DEBUG:boto:Data: 2023-10-17T18:17:33.184 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.184 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.185 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.185 DEBUG:boto:Params: {} 2023-10-17T18:17:33.186 DEBUG:boto:Token: None 2023-10-17T18:17:33.186 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj?versionId=null 2023-10-17T18:17:33.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc= 2023-10-17T18:17:33.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MN/vVvZcn4X+x6rczRey1TloeNc=', 'Content-Length': '0'} 2023-10-17T18:17:33.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000732164c9de3c3eba-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.230 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.231 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.231 DEBUG:boto:path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.232 DEBUG:boto:auth_path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.232 DEBUG:boto:Method: GET 2023-10-17T18:17:33.233 DEBUG:boto:Path: /hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.233 DEBUG:boto:Data: 2023-10-17T18:17:33.234 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.234 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.235 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.235 DEBUG:boto:Params: {} 2023-10-17T18:17:33.236 DEBUG:boto:Token: None 2023-10-17T18:17:33.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f082dc03fc6d6dfe-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.241 DEBUG:boto:b'hbmuog-7_myobj1000false_myobj2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.241 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.242 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.242 DEBUG:boto:path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.243 DEBUG:boto:auth_path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.243 DEBUG:boto:Method: GET 2023-10-17T18:17:33.244 DEBUG:boto:Path: /hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.244 DEBUG:boto:Data: 2023-10-17T18:17:33.245 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.246 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.246 DEBUG:boto:Params: {} 2023-10-17T18:17:33.247 DEBUG:boto:Token: None 2023-10-17T18:17:33.247 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf7057e3d3a200f-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.252 DEBUG:boto:b'hbmuog-7_myobj1000false_myobj2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.252 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.253 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.253 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.254 DEBUG:boto:Path: /hbmuog-7/_myobj 2023-10-17T18:17:33.254 DEBUG:boto:Data: 2023-10-17T18:17:33.255 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.255 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.256 DEBUG:boto:Params: {} 2023-10-17T18:17:33.256 DEBUG:boto:Token: None 2023-10-17T18:17:33.257 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj 2023-10-17T18:17:33.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U9bJ4aVn/lteXPaJzvHnQn1kyHc= 2023-10-17T18:17:33.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U9bJ4aVn/lteXPaJzvHnQn1kyHc=', 'Content-Length': '0'} 2023-10-17T18:17:33.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070ca70d6b2441d70-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.261 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.261 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.262 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.262 DEBUG:boto:Path: /hbmuog-7/_myobj 2023-10-17T18:17:33.262 DEBUG:boto:Data: 2023-10-17T18:17:33.263 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.264 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.264 DEBUG:boto:Params: {} 2023-10-17T18:17:33.265 DEBUG:boto:Token: None 2023-10-17T18:17:33.265 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj 2023-10-17T18:17:33.265 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U9bJ4aVn/lteXPaJzvHnQn1kyHc= 2023-10-17T18:17:33.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U9bJ4aVn/lteXPaJzvHnQn1kyHc=', 'Content-Length': '0'} 2023-10-17T18:17:33.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088fbc0c0ec6ad2d7-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:33.269 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.269 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.270 DEBUG:boto:Method: GET 2023-10-17T18:17:33.270 DEBUG:boto:Path: /hbmuog-7/_myobj 2023-10-17T18:17:33.271 DEBUG:boto:Data: 2023-10-17T18:17:33.271 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.272 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.272 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.273 DEBUG:boto:Params: {} 2023-10-17T18:17:33.273 DEBUG:boto:Token: None 2023-10-17T18:17:33.274 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj 2023-10-17T18:17:33.274 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PVS9950l9CNWpUolKjGyiyyobcw= 2023-10-17T18:17:33.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PVS9950l9CNWpUolKjGyiyyobcw=', 'Content-Length': '0'} 2023-10-17T18:17:33.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000532054e612cb9858-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.318 DEBUG:boto:path=/hbmuog-7/_myobj 2023-10-17T18:17:33.319 DEBUG:boto:auth_path=/hbmuog-7/_myobj 2023-10-17T18:17:33.319 DEBUG:boto:Method: GET 2023-10-17T18:17:33.320 DEBUG:boto:Path: /hbmuog-7/_myobj 2023-10-17T18:17:33.320 DEBUG:boto:Data: 2023-10-17T18:17:33.321 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.321 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.322 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.322 DEBUG:boto:Params: {} 2023-10-17T18:17:33.323 DEBUG:boto:Token: None 2023-10-17T18:17:33.323 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/_myobj 2023-10-17T18:17:33.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PVS9950l9CNWpUolKjGyiyyobcw= 2023-10-17T18:17:33.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PVS9950l9CNWpUolKjGyiyyobcw=', 'Content-Length': '0'} 2023-10-17T18:17:33.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004dbd252642b0855-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.327 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.327 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.328 DEBUG:boto:path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.328 DEBUG:boto:auth_path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.329 DEBUG:boto:Method: GET 2023-10-17T18:17:33.329 DEBUG:boto:Path: /hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.330 DEBUG:boto:Data: 2023-10-17T18:17:33.330 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.331 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.331 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.332 DEBUG:boto:Params: {} 2023-10-17T18:17:33.332 DEBUG:boto:Token: None 2023-10-17T18:17:33.333 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.333 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c4677dddbb6f77a-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.337 DEBUG:boto:b'hbmuog-7_myobj1000false_myobj2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.338 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.338 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.339 DEBUG:boto:path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.339 DEBUG:boto:auth_path=/hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.340 DEBUG:boto:Method: GET 2023-10-17T18:17:33.340 DEBUG:boto:Path: /hbmuog-7/?prefix=_myobj 2023-10-17T18:17:33.341 DEBUG:boto:Data: 2023-10-17T18:17:33.341 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.342 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.342 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.343 DEBUG:boto:Params: {} 2023-10-17T18:17:33.343 DEBUG:boto:Token: None 2023-10-17T18:17:33.344 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.344 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c8e4059ec0f03bb-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.390 DEBUG:boto:b'hbmuog-7_myobj1000false_myobj2023-10-17T18:14:26.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:33.392 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.392 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.392 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.393 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.393 DEBUG:boto:Method: GET 2023-10-17T18:17:33.394 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.394 DEBUG:boto:Data: 2023-10-17T18:17:33.395 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.395 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.396 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.396 DEBUG:boto:Params: {} 2023-10-17T18:17:33.397 DEBUG:boto:Token: None 2023-10-17T18:17:33.397 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.398 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw= 2023-10-17T18:17:33.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw=', 'Content-Length': '0'} 2023-10-17T18:17:33.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d460d4a4f0d15107-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.402 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.402 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.403 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.403 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.404 DEBUG:boto:Method: GET 2023-10-17T18:17:33.404 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.405 DEBUG:boto:Data: 2023-10-17T18:17:33.405 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.406 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.407 DEBUG:boto:Params: {} 2023-10-17T18:17:33.407 DEBUG:boto:Token: None 2023-10-17T18:17:33.408 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw= 2023-10-17T18:17:33.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw=', 'Content-Length': '0'} 2023-10-17T18:17:33.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080ca02c499d1a9f2-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.411 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.412 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.412 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.413 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.413 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.414 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.414 DEBUG:boto:Data: 2023-10-17T18:17:33.415 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.415 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.416 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.416 DEBUG:boto:Params: {} 2023-10-17T18:17:33.416 DEBUG:boto:Token: None 2023-10-17T18:17:33.417 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.418 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZLrIi1xJS6pV2tQaHnIcGHye1sA= 2023-10-17T18:17:33.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZLrIi1xJS6pV2tQaHnIcGHye1sA=', 'Content-Length': '0'} 2023-10-17T18:17:33.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010e740a838eefb8c-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.421 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.421 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.422 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.422 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.423 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.423 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.424 DEBUG:boto:Data: 2023-10-17T18:17:33.424 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.424 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.425 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.425 DEBUG:boto:Params: {} 2023-10-17T18:17:33.426 DEBUG:boto:Token: None 2023-10-17T18:17:33.426 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.427 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZLrIi1xJS6pV2tQaHnIcGHye1sA= 2023-10-17T18:17:33.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZLrIi1xJS6pV2tQaHnIcGHye1sA=', 'Content-Length': '0'} 2023-10-17T18:17:33.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8c3bc07f1cf2c9c-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:33.430 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.430 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.431 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.431 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.432 DEBUG:boto:Method: GET 2023-10-17T18:17:33.432 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.433 DEBUG:boto:Data: 2023-10-17T18:17:33.433 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.433 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.434 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.434 DEBUG:boto:Params: {} 2023-10-17T18:17:33.435 DEBUG:boto:Token: None 2023-10-17T18:17:33.435 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.436 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw= 2023-10-17T18:17:33.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw=', 'Content-Length': '0'} 2023-10-17T18:17:33.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4c244d969669e18-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.439 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.440 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.440 DEBUG:boto:path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.441 DEBUG:boto:auth_path=/hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.441 DEBUG:boto:Method: GET 2023-10-17T18:17:33.442 DEBUG:boto:Path: /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.442 DEBUG:boto:Data: 2023-10-17T18:17:33.443 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.443 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.443 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.444 DEBUG:boto:Params: {} 2023-10-17T18:17:33.444 DEBUG:boto:Token: None 2023-10-17T18:17:33.445 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj?versionId=null 2023-10-17T18:17:33.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw= 2023-10-17T18:17:33.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cXXGF+drsMwe9pAQCo6KlxNbWaw=', 'Content-Length': '0'} 2023-10-17T18:17:33.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d0946d143ed237f-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.448 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.449 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.449 DEBUG:boto:path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.450 DEBUG:boto:auth_path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.450 DEBUG:boto:Method: GET 2023-10-17T18:17:33.451 DEBUG:boto:Path: /hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.451 DEBUG:boto:Data: 2023-10-17T18:17:33.452 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.452 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.452 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.453 DEBUG:boto:Params: {} 2023-10-17T18:17:33.453 DEBUG:boto:Token: None 2023-10-17T18:17:33.454 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.454 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4feedb5a55853fd-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.458 DEBUG:boto:b'hbmuog-7myobj1000falsemyobj2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.459 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.459 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.460 DEBUG:boto:path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.460 DEBUG:boto:auth_path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.461 DEBUG:boto:Method: GET 2023-10-17T18:17:33.461 DEBUG:boto:Path: /hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.461 DEBUG:boto:Data: 2023-10-17T18:17:33.462 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.463 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.463 DEBUG:boto:Params: {} 2023-10-17T18:17:33.464 DEBUG:boto:Token: None 2023-10-17T18:17:33.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a49e52aa9a1fe84-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.468 DEBUG:boto:b'hbmuog-7myobj1000falsemyobj2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.469 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.469 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.470 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.470 DEBUG:boto:Path: /hbmuog-7/myobj 2023-10-17T18:17:33.471 DEBUG:boto:Data: 2023-10-17T18:17:33.471 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.472 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.472 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.472 DEBUG:boto:Params: {} 2023-10-17T18:17:33.473 DEBUG:boto:Token: None 2023-10-17T18:17:33.473 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj 2023-10-17T18:17:33.474 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TGDaDeZHGlSO3piiCmrGveclZE4= 2023-10-17T18:17:33.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TGDaDeZHGlSO3piiCmrGveclZE4=', 'Content-Length': '0'} 2023-10-17T18:17:33.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c88aef011079f77b-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.477 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.477 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.478 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.478 DEBUG:boto:Path: /hbmuog-7/myobj 2023-10-17T18:17:33.479 DEBUG:boto:Data: 2023-10-17T18:17:33.479 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.480 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.481 DEBUG:boto:Params: {} 2023-10-17T18:17:33.481 DEBUG:boto:Token: None 2023-10-17T18:17:33.481 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj 2023-10-17T18:17:33.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TGDaDeZHGlSO3piiCmrGveclZE4= 2023-10-17T18:17:33.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TGDaDeZHGlSO3piiCmrGveclZE4=', 'Content-Length': '0'} 2023-10-17T18:17:33.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009df3f558ff13e449-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:33.485 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.486 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.486 DEBUG:boto:Method: GET 2023-10-17T18:17:33.487 DEBUG:boto:Path: /hbmuog-7/myobj 2023-10-17T18:17:33.487 DEBUG:boto:Data: 2023-10-17T18:17:33.488 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.488 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.488 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.489 DEBUG:boto:Params: {} 2023-10-17T18:17:33.489 DEBUG:boto:Token: None 2023-10-17T18:17:33.490 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj 2023-10-17T18:17:33.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ABiHpCgLINJPKe9NWvbvzSERuRY= 2023-10-17T18:17:33.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ABiHpCgLINJPKe9NWvbvzSERuRY=', 'Content-Length': '0'} 2023-10-17T18:17:33.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2c876afe2b17583-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.494 DEBUG:boto:path=/hbmuog-7/myobj 2023-10-17T18:17:33.494 DEBUG:boto:auth_path=/hbmuog-7/myobj 2023-10-17T18:17:33.495 DEBUG:boto:Method: GET 2023-10-17T18:17:33.495 DEBUG:boto:Path: /hbmuog-7/myobj 2023-10-17T18:17:33.495 DEBUG:boto:Data: 2023-10-17T18:17:33.496 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.496 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.497 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.497 DEBUG:boto:Params: {} 2023-10-17T18:17:33.498 DEBUG:boto:Token: None 2023-10-17T18:17:33.498 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/myobj 2023-10-17T18:17:33.499 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ABiHpCgLINJPKe9NWvbvzSERuRY= 2023-10-17T18:17:33.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ABiHpCgLINJPKe9NWvbvzSERuRY=', 'Content-Length': '0'} 2023-10-17T18:17:33.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6134f7743428b1e-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.542 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.543 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.543 DEBUG:boto:path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.544 DEBUG:boto:auth_path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.544 DEBUG:boto:Method: GET 2023-10-17T18:17:33.544 DEBUG:boto:Path: /hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.545 DEBUG:boto:Data: 2023-10-17T18:17:33.545 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.546 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.546 DEBUG:boto:Params: {} 2023-10-17T18:17:33.547 DEBUG:boto:Token: None 2023-10-17T18:17:33.547 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9290f3408bf9ed-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.551 DEBUG:boto:b'hbmuog-7myobj1000falsemyobj2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.552 DEBUG:boto:path=/hbmuog-7/ 2023-10-17T18:17:33.553 DEBUG:boto:auth_path=/hbmuog-7/ 2023-10-17T18:17:33.553 DEBUG:boto:path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.554 DEBUG:boto:auth_path=/hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.554 DEBUG:boto:Method: GET 2023-10-17T18:17:33.555 DEBUG:boto:Path: /hbmuog-7/?prefix=myobj 2023-10-17T18:17:33.555 DEBUG:boto:Data: 2023-10-17T18:17:33.555 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.556 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.556 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.557 DEBUG:boto:Params: {} 2023-10-17T18:17:33.557 DEBUG:boto:Token: None 2023-10-17T18:17:33.558 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-7/ 2023-10-17T18:17:33.558 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0= 2023-10-17T18:17:33.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d53wvvKJKV6I6/+Ix5I7FKQgfD0=', 'Content-Length': '0'} 2023-10-17T18:17:33.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f286a16dff89beb3-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.562 DEBUG:boto:b'hbmuog-7myobj1000falsemyobj2023-10-17T18:14:26.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.565 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-7 zones={a2, a1} 2023-10-17T18:17:33.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:33.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:33.566 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:17:33.633 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:33.633 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:33.670+0000 7fe19c9eb780 1 bucket sync caught up with source: 2023-10-17T18:17:33.672 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.5.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.6.6, , ] 2023-10-17T18:17:33.672 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.5.6, , 00000000001.5.6, , , , 00000000001.7.6, 00000000001.6.6, , ] 2023-10-17T18:17:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:33.670+0000 7fe19c9eb780 0 bucket checkpoint complete 2023-10-17T18:17:33.684 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-8 zones={a1, a2} 2023-10-17T18:17:33.685 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.685 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.686 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.686 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:17:33.687 DEBUG:boto:Data: 2023-10-17T18:17:33.687 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.688 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.688 DEBUG:boto:Params: {} 2023-10-17T18:17:33.689 DEBUG:boto:Token: None 2023-10-17T18:17:33.690 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.691 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G5mPjQyNVG+OVHSQJYv1kgg8UJM= 2023-10-17T18:17:33.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G5mPjQyNVG+OVHSQJYv1kgg8UJM=', 'Content-Length': '0'} 2023-10-17T18:17:33.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006591f17b8ac3b0a0-00652ecfbd-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.694 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.695 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.695 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.695 DEBUG:boto:Path: /hbmuog-8/ 2023-10-17T18:17:33.696 DEBUG:boto:Data: 2023-10-17T18:17:33.696 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.697 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.698 DEBUG:boto:Params: {} 2023-10-17T18:17:33.698 DEBUG:boto:Token: None 2023-10-17T18:17:33.699 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G5mPjQyNVG+OVHSQJYv1kgg8UJM= 2023-10-17T18:17:33.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G5mPjQyNVG+OVHSQJYv1kgg8UJM=', 'Content-Length': '0'} 2023-10-17T18:17:33.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021c4a4a56318aca0-00652ecfbd-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.703 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:17:33.703 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.704 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.704 DEBUG:boto:path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.704 DEBUG:boto:auth_path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.705 DEBUG:boto:Method: GET 2023-10-17T18:17:33.705 DEBUG:boto:Path: /hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.706 DEBUG:boto:Data: 2023-10-17T18:17:33.706 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.707 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.707 DEBUG:boto:Params: {} 2023-10-17T18:17:33.708 DEBUG:boto:Token: None 2023-10-17T18:17:33.708 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/?versions 2023-10-17T18:17:33.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE= 2023-10-17T18:17:33.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE=', 'Content-Length': '0'} 2023-10-17T18:17:33.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf721ebabce0e27-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.712 DEBUG:boto:b'hbmuog-8999false&nulltrue2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.713 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-17T18:17:33.714 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-17T18:17:33.714 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-17T18:17:33.714 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:33.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:17:33.715 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.716 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.716 DEBUG:boto:path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.717 DEBUG:boto:auth_path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.717 DEBUG:boto:Method: GET 2023-10-17T18:17:33.717 DEBUG:boto:Path: /hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.718 DEBUG:boto:Data: 2023-10-17T18:17:33.718 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.719 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.720 DEBUG:boto:Params: {} 2023-10-17T18:17:33.720 DEBUG:boto:Token: None 2023-10-17T18:17:33.721 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/?versions 2023-10-17T18:17:33.721 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE= 2023-10-17T18:17:33.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE=', 'Content-Length': '0'} 2023-10-17T18:17:33.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e38266877e2f45a-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.724 DEBUG:boto:b'hbmuog-8999false&nulltrue2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.725 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-17T18:17:33.725 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-17T18:17:33.726 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-17T18:17:33.726 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:33.727 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.727 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.728 DEBUG:boto:path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.728 DEBUG:boto:auth_path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.728 DEBUG:boto:Method: GET 2023-10-17T18:17:33.729 DEBUG:boto:Path: /hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.729 DEBUG:boto:Data: 2023-10-17T18:17:33.730 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.730 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.731 DEBUG:boto:Params: {} 2023-10-17T18:17:33.731 DEBUG:boto:Token: None 2023-10-17T18:17:33.732 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/?versions 2023-10-17T18:17:33.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE= 2023-10-17T18:17:33.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE=', 'Content-Length': '0'} 2023-10-17T18:17:33.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df7a5e183789324d-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.736 DEBUG:boto:b'hbmuog-8999false&nulltrue2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.737 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.737 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.737 DEBUG:boto:path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.738 DEBUG:boto:auth_path=/hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.738 DEBUG:boto:Method: GET 2023-10-17T18:17:33.739 DEBUG:boto:Path: /hbmuog-8/?versions&max-keys=999 2023-10-17T18:17:33.739 DEBUG:boto:Data: 2023-10-17T18:17:33.740 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.740 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.740 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.741 DEBUG:boto:Params: {} 2023-10-17T18:17:33.741 DEBUG:boto:Token: None 2023-10-17T18:17:33.742 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/?versions 2023-10-17T18:17:33.742 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE= 2023-10-17T18:17:33.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IpA8ske4P/i72ievguPfxpA7JaE=', 'Content-Length': '0'} 2023-10-17T18:17:33.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2d5548d7c2746f-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.745 DEBUG:boto:b'hbmuog-8999false&nulltrue2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:33.747 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.747 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.747 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.748 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.748 DEBUG:boto:Method: GET 2023-10-17T18:17:33.749 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.749 DEBUG:boto:Data: 2023-10-17T18:17:33.750 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.750 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.751 DEBUG:boto:Params: {} 2023-10-17T18:17:33.751 DEBUG:boto:Token: None 2023-10-17T18:17:33.752 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso= 2023-10-17T18:17:33.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso=', 'Content-Length': '0'} 2023-10-17T18:17:33.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e26e6323f0a78d27-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.756 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.756 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.756 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.757 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.757 DEBUG:boto:Method: GET 2023-10-17T18:17:33.758 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.758 DEBUG:boto:Data: 2023-10-17T18:17:33.759 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.759 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.760 DEBUG:boto:Params: {} 2023-10-17T18:17:33.760 DEBUG:boto:Token: None 2023-10-17T18:17:33.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso= 2023-10-17T18:17:33.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso=', 'Content-Length': '0'} 2023-10-17T18:17:33.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fecee91e891e1777-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.806 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.807 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.807 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.808 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.808 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.808 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.809 DEBUG:boto:Data: 2023-10-17T18:17:33.809 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.810 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.810 DEBUG:boto:Params: {} 2023-10-17T18:17:33.811 DEBUG:boto:Token: None 2023-10-17T18:17:33.811 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CrsFKj1IdSjkWDaOYSR95qr1sxw= 2023-10-17T18:17:33.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CrsFKj1IdSjkWDaOYSR95qr1sxw=', 'Content-Length': '0'} 2023-10-17T18:17:33.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aacd2989db925c47-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.815 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.815 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.816 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.816 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.817 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.817 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.818 DEBUG:boto:Data: 2023-10-17T18:17:33.818 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.819 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.819 DEBUG:boto:Params: {} 2023-10-17T18:17:33.820 DEBUG:boto:Token: None 2023-10-17T18:17:33.820 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CrsFKj1IdSjkWDaOYSR95qr1sxw= 2023-10-17T18:17:33.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CrsFKj1IdSjkWDaOYSR95qr1sxw=', 'Content-Length': '0'} 2023-10-17T18:17:33.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2902c7da4747d3f-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:33.824 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.824 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.825 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.825 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.826 DEBUG:boto:Method: GET 2023-10-17T18:17:33.826 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.827 DEBUG:boto:Data: 2023-10-17T18:17:33.827 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.828 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.828 DEBUG:boto:Params: {} 2023-10-17T18:17:33.829 DEBUG:boto:Token: None 2023-10-17T18:17:33.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso= 2023-10-17T18:17:33.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso=', 'Content-Length': '0'} 2023-10-17T18:17:33.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af901efc7f86ebc2-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.874 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.875 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.875 DEBUG:boto:path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.876 DEBUG:boto:auth_path=/hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.876 DEBUG:boto:Method: GET 2023-10-17T18:17:33.877 DEBUG:boto:Path: /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.877 DEBUG:boto:Data: 2023-10-17T18:17:33.877 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.878 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.879 DEBUG:boto:Params: {} 2023-10-17T18:17:33.879 DEBUG:boto:Token: None 2023-10-17T18:17:33.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26?versionId=null 2023-10-17T18:17:33.880 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso= 2023-10-17T18:17:33.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2TPQFA23kDQXkK1kqB6nx6PKeso=', 'Content-Length': '0'} 2023-10-17T18:17:33.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a2d1d4e270c0de1-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.883 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.884 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.884 DEBUG:boto:path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.884 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.885 DEBUG:boto:Method: GET 2023-10-17T18:17:33.885 DEBUG:boto:Path: /hbmuog-8/?prefix=%26 2023-10-17T18:17:33.886 DEBUG:boto:Data: 2023-10-17T18:17:33.886 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.887 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.887 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.887 DEBUG:boto:Params: {} 2023-10-17T18:17:33.888 DEBUG:boto:Token: None 2023-10-17T18:17:33.889 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.889 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA= 2023-10-17T18:17:33.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA=', 'Content-Length': '0'} 2023-10-17T18:17:33.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005c55a03a81c1714-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.892 DEBUG:boto:b'hbmuog-8&1000false&2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.893 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.894 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.894 DEBUG:boto:path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.894 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.895 DEBUG:boto:Method: GET 2023-10-17T18:17:33.895 DEBUG:boto:Path: /hbmuog-8/?prefix=%26 2023-10-17T18:17:33.896 DEBUG:boto:Data: 2023-10-17T18:17:33.896 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.896 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.897 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.897 DEBUG:boto:Params: {} 2023-10-17T18:17:33.898 DEBUG:boto:Token: None 2023-10-17T18:17:33.898 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.899 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA= 2023-10-17T18:17:33.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA=', 'Content-Length': '0'} 2023-10-17T18:17:33.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000194ab6a88b14f8e-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.902 DEBUG:boto:b'hbmuog-8&1000false&2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.903 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.904 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.904 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.904 DEBUG:boto:Path: /hbmuog-8/%26 2023-10-17T18:17:33.905 DEBUG:boto:Data: 2023-10-17T18:17:33.905 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.906 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.907 DEBUG:boto:Params: {} 2023-10-17T18:17:33.907 DEBUG:boto:Token: None 2023-10-17T18:17:33.907 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26 2023-10-17T18:17:33.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/W0K9vDbQWA4VSbOCW9PS2bDVBE= 2023-10-17T18:17:33.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/W0K9vDbQWA4VSbOCW9PS2bDVBE=', 'Content-Length': '0'} 2023-10-17T18:17:33.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac1040388d923783-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.911 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.911 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.912 DEBUG:boto:Method: HEAD 2023-10-17T18:17:33.912 DEBUG:boto:Path: /hbmuog-8/%26 2023-10-17T18:17:33.912 DEBUG:boto:Data: 2023-10-17T18:17:33.913 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.914 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.914 DEBUG:boto:Params: {} 2023-10-17T18:17:33.915 DEBUG:boto:Token: None 2023-10-17T18:17:33.915 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26 2023-10-17T18:17:33.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/W0K9vDbQWA4VSbOCW9PS2bDVBE= 2023-10-17T18:17:33.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/W0K9vDbQWA4VSbOCW9PS2bDVBE=', 'Content-Length': '0'} 2023-10-17T18:17:33.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3523bd3ea669300-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-17T18:17:33.919 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.919 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.920 DEBUG:boto:Method: GET 2023-10-17T18:17:33.920 DEBUG:boto:Path: /hbmuog-8/%26 2023-10-17T18:17:33.920 DEBUG:boto:Data: 2023-10-17T18:17:33.921 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.922 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.922 DEBUG:boto:Params: {} 2023-10-17T18:17:33.923 DEBUG:boto:Token: None 2023-10-17T18:17:33.923 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26 2023-10-17T18:17:33.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ejwt46KP5Ccz+ewqzbzxeD3bKHw= 2023-10-17T18:17:33.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ejwt46KP5Ccz+ewqzbzxeD3bKHw=', 'Content-Length': '0'} 2023-10-17T18:17:33.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c94540751e03dfc2-00652ecfbd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.926 DEBUG:boto:path=/hbmuog-8/%26 2023-10-17T18:17:33.927 DEBUG:boto:auth_path=/hbmuog-8/%26 2023-10-17T18:17:33.927 DEBUG:boto:Method: GET 2023-10-17T18:17:33.928 DEBUG:boto:Path: /hbmuog-8/%26 2023-10-17T18:17:33.928 DEBUG:boto:Data: 2023-10-17T18:17:33.929 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.930 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.930 DEBUG:boto:Params: {} 2023-10-17T18:17:33.931 DEBUG:boto:Token: None 2023-10-17T18:17:33.931 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/%26 2023-10-17T18:17:33.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ejwt46KP5Ccz+ewqzbzxeD3bKHw= 2023-10-17T18:17:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ejwt46KP5Ccz+ewqzbzxeD3bKHw=', 'Content-Length': '0'} 2023-10-17T18:17:33.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1c183b6a862544-00652ecfbd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.935 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.935 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.936 DEBUG:boto:path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.936 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.937 DEBUG:boto:Method: GET 2023-10-17T18:17:33.937 DEBUG:boto:Path: /hbmuog-8/?prefix=%26 2023-10-17T18:17:33.938 DEBUG:boto:Data: 2023-10-17T18:17:33.938 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:33.939 DEBUG:boto:Port: 8000 2023-10-17T18:17:33.939 DEBUG:boto:Params: {} 2023-10-17T18:17:33.940 DEBUG:boto:Token: None 2023-10-17T18:17:33.940 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA= 2023-10-17T18:17:33.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA=', 'Content-Length': '0'} 2023-10-17T18:17:33.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d257d35bbea4a29-00652ecfbd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.944 DEBUG:boto:b'hbmuog-8&1000false&2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.945 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:33.945 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:33.946 DEBUG:boto:path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.946 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%26 2023-10-17T18:17:33.946 DEBUG:boto:Method: GET 2023-10-17T18:17:33.947 DEBUG:boto:Path: /hbmuog-8/?prefix=%26 2023-10-17T18:17:33.947 DEBUG:boto:Data: 2023-10-17T18:17:33.948 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.948 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:33.949 DEBUG:boto:Port: 8001 2023-10-17T18:17:33.949 DEBUG:boto:Params: {} 2023-10-17T18:17:33.949 DEBUG:boto:Token: None 2023-10-17T18:17:33.950 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:33 GMT /hbmuog-8/ 2023-10-17T18:17:33.950 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA= 2023-10-17T18:17:33.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSj0pI/kHT8ePY3vNhGDbGZirLA=', 'Content-Length': '0'} 2023-10-17T18:17:33.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3c42d2bfa11edb-00652ecfbd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:33.994 DEBUG:boto:b'hbmuog-8&1000false&2023-10-17T18:14:27.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:33.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:33.996 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:33.996 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:33.997 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:33.997 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:33.997 DEBUG:boto:Method: GET 2023-10-17T18:17:33.998 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:33.998 DEBUG:boto:Data: 2023-10-17T18:17:33.999 DEBUG:boto:Headers: {} 2023-10-17T18:17:33.999 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.000 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.000 DEBUG:boto:Params: {} 2023-10-17T18:17:34.001 DEBUG:boto:Token: None 2023-10-17T18:17:34.001 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.002 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0= 2023-10-17T18:17:34.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0=', 'Content-Length': '0'} 2023-10-17T18:17:34.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da6aec4fd9d8afa7-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.005 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.005 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.006 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.006 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.007 DEBUG:boto:Method: GET 2023-10-17T18:17:34.007 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.008 DEBUG:boto:Data: 2023-10-17T18:17:34.008 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.009 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.010 DEBUG:boto:Params: {} 2023-10-17T18:17:34.010 DEBUG:boto:Token: None 2023-10-17T18:17:34.011 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0= 2023-10-17T18:17:34.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0=', 'Content-Length': '0'} 2023-10-17T18:17:34.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2a28e0731ae5da0-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.014 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.015 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.015 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.016 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.016 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.016 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.017 DEBUG:boto:Data: 2023-10-17T18:17:34.017 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.018 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.019 DEBUG:boto:Params: {} 2023-10-17T18:17:34.019 DEBUG:boto:Token: None 2023-10-17T18:17:34.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fDnukfTqlG31mdEBRWEq/ujcI7s= 2023-10-17T18:17:34.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fDnukfTqlG31mdEBRWEq/ujcI7s=', 'Content-Length': '0'} 2023-10-17T18:17:34.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000258d126e8b368941-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.023 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.023 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.024 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.024 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.025 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.025 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.025 DEBUG:boto:Data: 2023-10-17T18:17:34.026 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.027 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.027 DEBUG:boto:Params: {} 2023-10-17T18:17:34.027 DEBUG:boto:Token: None 2023-10-17T18:17:34.028 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.028 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fDnukfTqlG31mdEBRWEq/ujcI7s= 2023-10-17T18:17:34.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fDnukfTqlG31mdEBRWEq/ujcI7s=', 'Content-Length': '0'} 2023-10-17T18:17:34.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1e498afa6395903-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:34.031 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.032 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.032 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.033 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.033 DEBUG:boto:Method: GET 2023-10-17T18:17:34.034 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.034 DEBUG:boto:Data: 2023-10-17T18:17:34.034 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.035 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.036 DEBUG:boto:Params: {} 2023-10-17T18:17:34.036 DEBUG:boto:Token: None 2023-10-17T18:17:34.037 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0= 2023-10-17T18:17:34.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0=', 'Content-Length': '0'} 2023-10-17T18:17:34.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045bae4594ae0dad8-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.040 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.041 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.041 DEBUG:boto:path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.041 DEBUG:boto:auth_path=/hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.042 DEBUG:boto:Method: GET 2023-10-17T18:17:34.042 DEBUG:boto:Path: /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.043 DEBUG:boto:Data: 2023-10-17T18:17:34.043 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.044 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.044 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.044 DEBUG:boto:Params: {} 2023-10-17T18:17:34.045 DEBUG:boto:Token: None 2023-10-17T18:17:34.045 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A?versionId=null 2023-10-17T18:17:34.046 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0= 2023-10-17T18:17:34.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvCyjwgiNX721H8tNoBk7KXj1E0=', 'Content-Length': '0'} 2023-10-17T18:17:34.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4efc69d5ff57d58-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.049 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.049 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.050 DEBUG:boto:path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.050 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.051 DEBUG:boto:Method: GET 2023-10-17T18:17:34.051 DEBUG:boto:Path: /hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.052 DEBUG:boto:Data: 2023-10-17T18:17:34.052 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.052 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.053 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.053 DEBUG:boto:Params: {} 2023-10-17T18:17:34.054 DEBUG:boto:Token: None 2023-10-17T18:17:34.054 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.055 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0415146df8ed12-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.058 DEBUG:boto:b'hbmuog-8:1000false:2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.059 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.059 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.060 DEBUG:boto:path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.060 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.061 DEBUG:boto:Method: GET 2023-10-17T18:17:34.061 DEBUG:boto:Path: /hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.061 DEBUG:boto:Data: 2023-10-17T18:17:34.062 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.062 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.063 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.063 DEBUG:boto:Params: {} 2023-10-17T18:17:34.064 DEBUG:boto:Token: None 2023-10-17T18:17:34.064 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c49824bfaecbf7ff-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.068 DEBUG:boto:b'hbmuog-8:1000false:2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.069 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.069 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.070 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.070 DEBUG:boto:Path: /hbmuog-8/%3A 2023-10-17T18:17:34.070 DEBUG:boto:Data: 2023-10-17T18:17:34.071 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.071 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.072 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.072 DEBUG:boto:Params: {} 2023-10-17T18:17:34.072 DEBUG:boto:Token: None 2023-10-17T18:17:34.073 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A 2023-10-17T18:17:34.073 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7pdCBby9lXfeDJSeNmJx+fC1jDI= 2023-10-17T18:17:34.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7pdCBby9lXfeDJSeNmJx+fC1jDI=', 'Content-Length': '0'} 2023-10-17T18:17:34.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa41a427edfdbe65-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.076 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.077 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.077 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.077 DEBUG:boto:Path: /hbmuog-8/%3A 2023-10-17T18:17:34.078 DEBUG:boto:Data: 2023-10-17T18:17:34.078 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.079 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.080 DEBUG:boto:Params: {} 2023-10-17T18:17:34.080 DEBUG:boto:Token: None 2023-10-17T18:17:34.081 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A 2023-10-17T18:17:34.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7pdCBby9lXfeDJSeNmJx+fC1jDI= 2023-10-17T18:17:34.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7pdCBby9lXfeDJSeNmJx+fC1jDI=', 'Content-Length': '0'} 2023-10-17T18:17:34.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2b8ed57aaeb15e4-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-17T18:17:34.084 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.085 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.085 DEBUG:boto:Method: GET 2023-10-17T18:17:34.086 DEBUG:boto:Path: /hbmuog-8/%3A 2023-10-17T18:17:34.086 DEBUG:boto:Data: 2023-10-17T18:17:34.087 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.087 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.088 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.088 DEBUG:boto:Params: {} 2023-10-17T18:17:34.089 DEBUG:boto:Token: None 2023-10-17T18:17:34.089 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A 2023-10-17T18:17:34.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4GZw6dqkEInf0HccZqKGbEpXEz8= 2023-10-17T18:17:34.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4GZw6dqkEInf0HccZqKGbEpXEz8=', 'Content-Length': '0'} 2023-10-17T18:17:34.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d4d06cbe2c0c7dc-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.094 DEBUG:boto:path=/hbmuog-8/%3A 2023-10-17T18:17:34.094 DEBUG:boto:auth_path=/hbmuog-8/%3A 2023-10-17T18:17:34.095 DEBUG:boto:Method: GET 2023-10-17T18:17:34.095 DEBUG:boto:Path: /hbmuog-8/%3A 2023-10-17T18:17:34.095 DEBUG:boto:Data: 2023-10-17T18:17:34.096 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.097 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.097 DEBUG:boto:Params: {} 2023-10-17T18:17:34.098 DEBUG:boto:Token: None 2023-10-17T18:17:34.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/%3A 2023-10-17T18:17:34.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4GZw6dqkEInf0HccZqKGbEpXEz8= 2023-10-17T18:17:34.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4GZw6dqkEInf0HccZqKGbEpXEz8=', 'Content-Length': '0'} 2023-10-17T18:17:34.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036152e0378abd248-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.146 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.147 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.147 DEBUG:boto:path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.148 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.148 DEBUG:boto:Method: GET 2023-10-17T18:17:34.148 DEBUG:boto:Path: /hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.149 DEBUG:boto:Data: 2023-10-17T18:17:34.149 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.150 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.150 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.151 DEBUG:boto:Params: {} 2023-10-17T18:17:34.151 DEBUG:boto:Token: None 2023-10-17T18:17:34.152 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.152 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f791658e770316c4-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.156 DEBUG:boto:b'hbmuog-8:1000false:2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.156 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.157 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.157 DEBUG:boto:path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.158 DEBUG:boto:auth_path=/hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.158 DEBUG:boto:Method: GET 2023-10-17T18:17:34.158 DEBUG:boto:Path: /hbmuog-8/?prefix=%3A 2023-10-17T18:17:34.159 DEBUG:boto:Data: 2023-10-17T18:17:34.159 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.160 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.161 DEBUG:boto:Params: {} 2023-10-17T18:17:34.161 DEBUG:boto:Token: None 2023-10-17T18:17:34.161 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.162 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab0c042a97b3ffd-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.165 DEBUG:boto:b'hbmuog-8:1000false:2023-10-17T18:14:27.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:34.166 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.167 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.167 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.168 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.168 DEBUG:boto:Method: GET 2023-10-17T18:17:34.168 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.169 DEBUG:boto:Data: 2023-10-17T18:17:34.169 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.170 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.171 DEBUG:boto:Params: {} 2023-10-17T18:17:34.171 DEBUG:boto:Token: None 2023-10-17T18:17:34.171 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0= 2023-10-17T18:17:34.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0=', 'Content-Length': '0'} 2023-10-17T18:17:34.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc641c002f49354-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.218 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.219 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.219 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.220 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.220 DEBUG:boto:Method: GET 2023-10-17T18:17:34.220 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.221 DEBUG:boto:Data: 2023-10-17T18:17:34.221 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.222 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.222 DEBUG:boto:Params: {} 2023-10-17T18:17:34.223 DEBUG:boto:Token: None 2023-10-17T18:17:34.224 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0= 2023-10-17T18:17:34.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0=', 'Content-Length': '0'} 2023-10-17T18:17:34.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c89cedd24b237fc-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.227 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.228 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.228 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.228 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.229 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.229 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.230 DEBUG:boto:Data: 2023-10-17T18:17:34.230 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.231 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.231 DEBUG:boto:Params: {} 2023-10-17T18:17:34.232 DEBUG:boto:Token: None 2023-10-17T18:17:34.232 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.233 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CV8PSjxR0nmxy4W7CZMDBzh/YPQ= 2023-10-17T18:17:34.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CV8PSjxR0nmxy4W7CZMDBzh/YPQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000206c661ad795b7b6-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.236 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.236 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.236 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.237 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.237 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.238 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.238 DEBUG:boto:Data: 2023-10-17T18:17:34.238 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.239 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.240 DEBUG:boto:Params: {} 2023-10-17T18:17:34.240 DEBUG:boto:Token: None 2023-10-17T18:17:34.241 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CV8PSjxR0nmxy4W7CZMDBzh/YPQ= 2023-10-17T18:17:34.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CV8PSjxR0nmxy4W7CZMDBzh/YPQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9194b6372e68050-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:34.244 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.245 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.245 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.246 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.246 DEBUG:boto:Method: GET 2023-10-17T18:17:34.246 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.247 DEBUG:boto:Data: 2023-10-17T18:17:34.247 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.248 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.248 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.249 DEBUG:boto:Params: {} 2023-10-17T18:17:34.249 DEBUG:boto:Token: None 2023-10-17T18:17:34.250 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.250 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0= 2023-10-17T18:17:34.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0=', 'Content-Length': '0'} 2023-10-17T18:17:34.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db0df5b7f6c9a05e-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.253 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.254 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.254 DEBUG:boto:path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.255 DEBUG:boto:auth_path=/hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.255 DEBUG:boto:Method: GET 2023-10-17T18:17:34.255 DEBUG:boto:Path: /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.256 DEBUG:boto:Data: 2023-10-17T18:17:34.256 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.257 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.257 DEBUG:boto:Params: {} 2023-10-17T18:17:34.258 DEBUG:boto:Token: None 2023-10-17T18:17:34.258 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj?versionId=null 2023-10-17T18:17:34.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0= 2023-10-17T18:17:34.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NSsKkMX6DzSXzZT53r0bQ1zGCY0=', 'Content-Length': '0'} 2023-10-17T18:17:34.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000256e22af2d963d46-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.262 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.262 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.263 DEBUG:boto:path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.263 DEBUG:boto:auth_path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.264 DEBUG:boto:Method: GET 2023-10-17T18:17:34.264 DEBUG:boto:Path: /hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.265 DEBUG:boto:Data: 2023-10-17T18:17:34.265 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.266 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.266 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.266 DEBUG:boto:Params: {} 2023-10-17T18:17:34.267 DEBUG:boto:Token: None 2023-10-17T18:17:34.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.268 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d65f8da0e11a6d-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.271 DEBUG:boto:b'hbmuog-8_myobj1000false_myobj2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.272 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.272 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.273 DEBUG:boto:path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.273 DEBUG:boto:auth_path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.274 DEBUG:boto:Method: GET 2023-10-17T18:17:34.274 DEBUG:boto:Path: /hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.275 DEBUG:boto:Data: 2023-10-17T18:17:34.276 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.276 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.277 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.277 DEBUG:boto:Params: {} 2023-10-17T18:17:34.277 DEBUG:boto:Token: None 2023-10-17T18:17:34.278 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.278 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6cde21a523a05ec-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.282 DEBUG:boto:b'hbmuog-8_myobj1000false_myobj2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.282 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.283 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.283 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.284 DEBUG:boto:Path: /hbmuog-8/_myobj 2023-10-17T18:17:34.284 DEBUG:boto:Data: 2023-10-17T18:17:34.285 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.285 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.285 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.286 DEBUG:boto:Params: {} 2023-10-17T18:17:34.286 DEBUG:boto:Token: None 2023-10-17T18:17:34.287 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj 2023-10-17T18:17:34.287 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aFO0Ekvc6j5gXewTXmkijrOwDCQ= 2023-10-17T18:17:34.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aFO0Ekvc6j5gXewTXmkijrOwDCQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ecab8b1a31ee7e0-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.290 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.290 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.291 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.291 DEBUG:boto:Path: /hbmuog-8/_myobj 2023-10-17T18:17:34.292 DEBUG:boto:Data: 2023-10-17T18:17:34.292 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.292 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.293 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.293 DEBUG:boto:Params: {} 2023-10-17T18:17:34.294 DEBUG:boto:Token: None 2023-10-17T18:17:34.294 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj 2023-10-17T18:17:34.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aFO0Ekvc6j5gXewTXmkijrOwDCQ= 2023-10-17T18:17:34.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aFO0Ekvc6j5gXewTXmkijrOwDCQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045c0381306433ca2-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-17T18:17:34.298 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.298 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.299 DEBUG:boto:Method: GET 2023-10-17T18:17:34.299 DEBUG:boto:Path: /hbmuog-8/_myobj 2023-10-17T18:17:34.300 DEBUG:boto:Data: 2023-10-17T18:17:34.300 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.301 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.301 DEBUG:boto:Params: {} 2023-10-17T18:17:34.302 DEBUG:boto:Token: None 2023-10-17T18:17:34.302 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj 2023-10-17T18:17:34.303 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cwcjJtqLIG3HUHxqpJyf5i8vTi8= 2023-10-17T18:17:34.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cwcjJtqLIG3HUHxqpJyf5i8vTi8=', 'Content-Length': '0'} 2023-10-17T18:17:34.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098f0aac25cf54c24-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.350 DEBUG:boto:path=/hbmuog-8/_myobj 2023-10-17T18:17:34.351 DEBUG:boto:auth_path=/hbmuog-8/_myobj 2023-10-17T18:17:34.351 DEBUG:boto:Method: GET 2023-10-17T18:17:34.352 DEBUG:boto:Path: /hbmuog-8/_myobj 2023-10-17T18:17:34.352 DEBUG:boto:Data: 2023-10-17T18:17:34.353 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.353 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.354 DEBUG:boto:Params: {} 2023-10-17T18:17:34.354 DEBUG:boto:Token: None 2023-10-17T18:17:34.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/_myobj 2023-10-17T18:17:34.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cwcjJtqLIG3HUHxqpJyf5i8vTi8= 2023-10-17T18:17:34.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cwcjJtqLIG3HUHxqpJyf5i8vTi8=', 'Content-Length': '0'} 2023-10-17T18:17:34.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8d2852626fffb66-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.359 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.359 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.360 DEBUG:boto:path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.360 DEBUG:boto:auth_path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.360 DEBUG:boto:Method: GET 2023-10-17T18:17:34.361 DEBUG:boto:Path: /hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.361 DEBUG:boto:Data: 2023-10-17T18:17:34.362 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.363 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.363 DEBUG:boto:Params: {} 2023-10-17T18:17:34.363 DEBUG:boto:Token: None 2023-10-17T18:17:34.364 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c9a3e2632912bb-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.368 DEBUG:boto:b'hbmuog-8_myobj1000false_myobj2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.369 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.369 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.370 DEBUG:boto:path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.370 DEBUG:boto:auth_path=/hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.370 DEBUG:boto:Method: GET 2023-10-17T18:17:34.371 DEBUG:boto:Path: /hbmuog-8/?prefix=_myobj 2023-10-17T18:17:34.371 DEBUG:boto:Data: 2023-10-17T18:17:34.372 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.373 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.373 DEBUG:boto:Params: {} 2023-10-17T18:17:34.374 DEBUG:boto:Token: None 2023-10-17T18:17:34.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013369929fe29197f-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.378 DEBUG:boto:b'hbmuog-8_myobj1000false_myobj2023-10-17T18:14:27.093Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:34.379 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.380 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.380 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.381 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.381 DEBUG:boto:Method: GET 2023-10-17T18:17:34.381 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.382 DEBUG:boto:Data: 2023-10-17T18:17:34.382 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.383 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.384 DEBUG:boto:Params: {} 2023-10-17T18:17:34.384 DEBUG:boto:Token: None 2023-10-17T18:17:34.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k= 2023-10-17T18:17:34.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k=', 'Content-Length': '0'} 2023-10-17T18:17:34.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005538b6f2a804579b-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.388 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.389 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.389 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.389 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.390 DEBUG:boto:Method: GET 2023-10-17T18:17:34.390 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.391 DEBUG:boto:Data: 2023-10-17T18:17:34.391 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.392 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.392 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.392 DEBUG:boto:Params: {} 2023-10-17T18:17:34.393 DEBUG:boto:Token: None 2023-10-17T18:17:34.393 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.394 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k= 2023-10-17T18:17:34.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k=', 'Content-Length': '0'} 2023-10-17T18:17:34.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5da7be39e90a61b-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.397 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.397 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.398 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.398 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.399 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.399 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.399 DEBUG:boto:Data: 2023-10-17T18:17:34.400 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.401 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.401 DEBUG:boto:Params: {} 2023-10-17T18:17:34.401 DEBUG:boto:Token: None 2023-10-17T18:17:34.402 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tXPMZrRWEaPexmkdRwI8TNckkW8= 2023-10-17T18:17:34.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tXPMZrRWEaPexmkdRwI8TNckkW8=', 'Content-Length': '0'} 2023-10-17T18:17:34.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044b5ba67daec257d-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.406 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.406 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.406 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.407 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.407 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.408 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.408 DEBUG:boto:Data: 2023-10-17T18:17:34.408 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.409 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.410 DEBUG:boto:Params: {} 2023-10-17T18:17:34.410 DEBUG:boto:Token: None 2023-10-17T18:17:34.411 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tXPMZrRWEaPexmkdRwI8TNckkW8= 2023-10-17T18:17:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tXPMZrRWEaPexmkdRwI8TNckkW8=', 'Content-Length': '0'} 2023-10-17T18:17:34.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4267781424895e9-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:34.414 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.415 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.415 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.416 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.416 DEBUG:boto:Method: GET 2023-10-17T18:17:34.417 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.417 DEBUG:boto:Data: 2023-10-17T18:17:34.417 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.418 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.419 DEBUG:boto:Params: {} 2023-10-17T18:17:34.419 DEBUG:boto:Token: None 2023-10-17T18:17:34.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k= 2023-10-17T18:17:34.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k=', 'Content-Length': '0'} 2023-10-17T18:17:34.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063db6edba72a0c9e-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.423 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.424 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.424 DEBUG:boto:path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.424 DEBUG:boto:auth_path=/hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.425 DEBUG:boto:Method: GET 2023-10-17T18:17:34.425 DEBUG:boto:Path: /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.425 DEBUG:boto:Data: 2023-10-17T18:17:34.426 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.427 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.427 DEBUG:boto:Params: {} 2023-10-17T18:17:34.428 DEBUG:boto:Token: None 2023-10-17T18:17:34.428 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj?versionId=null 2023-10-17T18:17:34.429 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k= 2023-10-17T18:17:34.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8MrCAuK46Eiagx5b4MCOYI4kx0k=', 'Content-Length': '0'} 2023-10-17T18:17:34.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3a647596496e65e-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.432 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.432 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.433 DEBUG:boto:path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.433 DEBUG:boto:auth_path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.433 DEBUG:boto:Method: GET 2023-10-17T18:17:34.434 DEBUG:boto:Path: /hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.434 DEBUG:boto:Data: 2023-10-17T18:17:34.435 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.435 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.435 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.436 DEBUG:boto:Params: {} 2023-10-17T18:17:34.436 DEBUG:boto:Token: None 2023-10-17T18:17:34.437 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e621c005cb081047-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.482 DEBUG:boto:b'hbmuog-8myobj1000falsemyobj2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.483 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.483 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.484 DEBUG:boto:path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.484 DEBUG:boto:auth_path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.485 DEBUG:boto:Method: GET 2023-10-17T18:17:34.485 DEBUG:boto:Path: /hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.485 DEBUG:boto:Data: 2023-10-17T18:17:34.486 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.487 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.487 DEBUG:boto:Params: {} 2023-10-17T18:17:34.488 DEBUG:boto:Token: None 2023-10-17T18:17:34.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045909c9f9bf4328e-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.492 DEBUG:boto:b'hbmuog-8myobj1000falsemyobj2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.493 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.493 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.494 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.494 DEBUG:boto:Path: /hbmuog-8/myobj 2023-10-17T18:17:34.495 DEBUG:boto:Data: 2023-10-17T18:17:34.495 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.496 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.496 DEBUG:boto:Params: {} 2023-10-17T18:17:34.497 DEBUG:boto:Token: None 2023-10-17T18:17:34.497 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj 2023-10-17T18:17:34.498 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JFoIXtXHLy8DE8RoqIwc+SmeHlM= 2023-10-17T18:17:34.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JFoIXtXHLy8DE8RoqIwc+SmeHlM=', 'Content-Length': '0'} 2023-10-17T18:17:34.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be4c1154eede51f4-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.500 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.501 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.501 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.502 DEBUG:boto:Path: /hbmuog-8/myobj 2023-10-17T18:17:34.502 DEBUG:boto:Data: 2023-10-17T18:17:34.503 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.503 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.503 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.504 DEBUG:boto:Params: {} 2023-10-17T18:17:34.504 DEBUG:boto:Token: None 2023-10-17T18:17:34.505 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj 2023-10-17T18:17:34.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JFoIXtXHLy8DE8RoqIwc+SmeHlM= 2023-10-17T18:17:34.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JFoIXtXHLy8DE8RoqIwc+SmeHlM=', 'Content-Length': '0'} 2023-10-17T18:17:34.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e231d9e364ebdec4-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:34.508 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.509 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.509 DEBUG:boto:Method: GET 2023-10-17T18:17:34.510 DEBUG:boto:Path: /hbmuog-8/myobj 2023-10-17T18:17:34.510 DEBUG:boto:Data: 2023-10-17T18:17:34.511 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.511 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.512 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.512 DEBUG:boto:Params: {} 2023-10-17T18:17:34.513 DEBUG:boto:Token: None 2023-10-17T18:17:34.513 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj 2023-10-17T18:17:34.513 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SuBDxj0kNGU9yhjw+IeRACwgBcw= 2023-10-17T18:17:34.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SuBDxj0kNGU9yhjw+IeRACwgBcw=', 'Content-Length': '0'} 2023-10-17T18:17:34.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2f9ef6581cd9d6-00652ecfbe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.517 DEBUG:boto:path=/hbmuog-8/myobj 2023-10-17T18:17:34.517 DEBUG:boto:auth_path=/hbmuog-8/myobj 2023-10-17T18:17:34.517 DEBUG:boto:Method: GET 2023-10-17T18:17:34.518 DEBUG:boto:Path: /hbmuog-8/myobj 2023-10-17T18:17:34.518 DEBUG:boto:Data: 2023-10-17T18:17:34.519 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.519 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.520 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.520 DEBUG:boto:Params: {} 2023-10-17T18:17:34.521 DEBUG:boto:Token: None 2023-10-17T18:17:34.521 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/myobj 2023-10-17T18:17:34.521 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SuBDxj0kNGU9yhjw+IeRACwgBcw= 2023-10-17T18:17:34.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SuBDxj0kNGU9yhjw+IeRACwgBcw=', 'Content-Length': '0'} 2023-10-17T18:17:34.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e499cdacc8dc5cc-00652ecfbe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.566 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.567 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.567 DEBUG:boto:path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.568 DEBUG:boto:auth_path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.568 DEBUG:boto:Method: GET 2023-10-17T18:17:34.569 DEBUG:boto:Path: /hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.569 DEBUG:boto:Data: 2023-10-17T18:17:34.569 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.570 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.571 DEBUG:boto:Params: {} 2023-10-17T18:17:34.571 DEBUG:boto:Token: None 2023-10-17T18:17:34.572 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d702a62281cb1a81-00652ecfbe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.576 DEBUG:boto:b'hbmuog-8myobj1000falsemyobj2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.576 DEBUG:boto:path=/hbmuog-8/ 2023-10-17T18:17:34.577 DEBUG:boto:auth_path=/hbmuog-8/ 2023-10-17T18:17:34.577 DEBUG:boto:path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.578 DEBUG:boto:auth_path=/hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.578 DEBUG:boto:Method: GET 2023-10-17T18:17:34.579 DEBUG:boto:Path: /hbmuog-8/?prefix=myobj 2023-10-17T18:17:34.579 DEBUG:boto:Data: 2023-10-17T18:17:34.580 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.580 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.580 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.581 DEBUG:boto:Params: {} 2023-10-17T18:17:34.581 DEBUG:boto:Token: None 2023-10-17T18:17:34.582 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-8/ 2023-10-17T18:17:34.582 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ= 2023-10-17T18:17:34.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FCcBqjlS//Fj4MJZvNDwB1Rz4jQ=', 'Content-Length': '0'} 2023-10-17T18:17:34.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce5fe553ba1568cf-00652ecfbe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.586 DEBUG:boto:b'hbmuog-8myobj1000falsemyobj2023-10-17T18:14:27.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:34.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-8 zones={a1, a2} 2023-10-17T18:17:34.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-10-17T18:17:34.588 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-9 2023-10-17T18:17:34.589 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:34.589 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:34.589 DEBUG:boto:Method: PUT 2023-10-17T18:17:34.590 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:17:34.590 DEBUG:boto:Data: 2023-10-17T18:17:34.591 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.591 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.592 DEBUG:boto:Params: {} 2023-10-17T18:17:34.592 DEBUG:boto:Token: None 2023-10-17T18:17:34.593 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-9/ 2023-10-17T18:17:34.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VXRwHRs7LMs/fXh+8IuUZ+c7QOw= 2023-10-17T18:17:34.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VXRwHRs7LMs/fXh+8IuUZ+c7QOw=', 'Content-Length': '0'} 2023-10-17T18:17:34.608 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c9a3b4f30ef3278-00652ecfbe-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.609 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-10 2023-10-17T18:17:34.609 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:34.609 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:34.610 DEBUG:boto:Method: PUT 2023-10-17T18:17:34.610 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:17:34.611 DEBUG:boto:Data: 2023-10-17T18:17:34.611 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.612 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.612 DEBUG:boto:Params: {} 2023-10-17T18:17:34.613 DEBUG:boto:Token: None 2023-10-17T18:17:34.613 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-10/ 2023-10-17T18:17:34.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ej5rhLtpNAEH9M8VeDGHYqWmm70= 2023-10-17T18:17:34.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ej5rhLtpNAEH9M8VeDGHYqWmm70=', 'Content-Length': '0'} 2023-10-17T18:17:34.637 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009bba51145b5b58b4-00652ecfbe-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.637 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:34.638 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:34.638 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.639 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:17:34.639 DEBUG:boto:Data: 2023-10-17T18:17:34.640 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.640 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.641 DEBUG:boto:Params: {} 2023-10-17T18:17:34.641 DEBUG:boto:Token: None 2023-10-17T18:17:34.642 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-9/ 2023-10-17T18:17:34.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s2NOUoWR3uNtRIh4X3xh55oSgV0= 2023-10-17T18:17:34.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s2NOUoWR3uNtRIh4X3xh55oSgV0=', 'Content-Length': '0'} 2023-10-17T18:17:34.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6ec450827a2cb50-00652ecfbe-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.646 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:34.647 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:34.647 DEBUG:boto:Method: PUT 2023-10-17T18:17:34.647 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:34.648 DEBUG:boto:Data: 2023-10-17T18:17:34.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:17:34.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:34.649 DEBUG:boto:Port: 8000 2023-10-17T18:17:34.649 DEBUG:boto:Params: {} 2023-10-17T18:17:34.650 DEBUG:boto:Token: None 2023-10-17T18:17:34.650 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-9/myobj 2023-10-17T18:17:34.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n1gHbJWNNpOvyqDvTB+moG7KIZY= 2023-10-17T18:17:34.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n1gHbJWNNpOvyqDvTB+moG7KIZY='} 2023-10-17T18:17:34.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fca78abdc4712a37-00652ecfbe-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.695 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:34.696 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:34.696 DEBUG:boto:Method: HEAD 2023-10-17T18:17:34.696 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:17:34.697 DEBUG:boto:Data: 2023-10-17T18:17:34.697 DEBUG:boto:Headers: {} 2023-10-17T18:17:34.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.698 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.698 DEBUG:boto:Params: {} 2023-10-17T18:17:34.699 DEBUG:boto:Token: None 2023-10-17T18:17:34.699 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-10/ 2023-10-17T18:17:34.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fYMcecSNMch38G1X7ky0NrqOmcM= 2023-10-17T18:17:34.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fYMcecSNMch38G1X7ky0NrqOmcM=', 'Content-Length': '0'} 2023-10-17T18:17:34.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f94a54b718d9f3bd-00652ecfbe-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.703 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:34.704 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:34.704 DEBUG:boto:Method: PUT 2023-10-17T18:17:34.705 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:17:34.705 DEBUG:boto:Data: 2023-10-17T18:17:34.705 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:17:34.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:34.706 DEBUG:boto:Port: 8001 2023-10-17T18:17:34.707 DEBUG:boto:Params: {} 2023-10-17T18:17:34.707 DEBUG:boto:Token: None 2023-10-17T18:17:34.707 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:17:34 GMT /hbmuog-10/myobj 2023-10-17T18:17:34.708 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XhMOWZzdAPkNKpErsJAV2Ua5ULM= 2023-10-17T18:17:34.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:17:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XhMOWZzdAPkNKpErsJAV2Ua5ULM='} 2023-10-17T18:17:34.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cabec79a3b4ce0f7-00652ecfbe-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:34.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:17:34.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:17:34.719 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:17:34.771 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:34.772 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:17:34.815 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:17:34.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.816 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.627624_134.1", 2023-10-17T18:17:34.816 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.627624Z" 2023-10-17T18:17:34.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.816 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:17:34.817 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:17:34.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.817 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:17:34.817 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:17:34.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.818 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:17:34.818 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:17:34.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.819 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:17:34.820 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566654.627624_134.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:17:34.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:17:34.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:17:34.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:17:34.822 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:17:34.874 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:34.874 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:34.902 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:34.901+0000 7fe71200b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:17:34.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:34.902+0000 7fe71200b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:17:34.915 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:17:34.915 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:17:34.915 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:17:34.916 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:17:34.916 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:17:34.916 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:17:34.916 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:17:34.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.917 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:17:34.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.917 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:17:34.917 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:17:34.917 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:17:34.918 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.627624_134.1", 2023-10-17T18:17:34.918 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:17:34.918 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:17:34.918 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:17:34.919 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.627624Z", 2023-10-17T18:17:34.919 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:17:34.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.920 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:17:34.920 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:17:34.920 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:17:34.920 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:17:34.920 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:17:34.921 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:17:34.921 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:17:34.921 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:17:34.921 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:17:34.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.922 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:17:34.922 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:17:34.922 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:17:34.923 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:17:34.923 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:17:34.923 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:17:34.923 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:17:34.923 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:17:34.924 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:17:34.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:17:34.924 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:17:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:17:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:17:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:17:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:17:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:17:34.927 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:17:34.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.927 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:17:34.927 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:17:34.927 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:17:34.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:17:34.928 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:17:34Z" 2023-10-17T18:17:34.928 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:17:34.929 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566654.627624_134.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:17:34.627624Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:17:34Z" } 2023-10-17T18:17:34.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566654.627624_134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.627624Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:17:34.930 DEBUG:rgw_multi.tests:log_status={0: '1_1697566654.627624_134.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:17:34.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566654.627624_134.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:17:34.931 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:17:34.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:34.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:34.932 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:17:34.990 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:34.990 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:35.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:35.022+0000 7f8162bb3780 1 waiting to reach incremental sync.. 2023-10-17T18:17:40.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:40.024+0000 7f8162bb3780 1 waiting to reach incremental sync.. 2023-10-17T18:17:45.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:45.024+0000 7f8162bb3780 1 waiting to reach incremental sync.. 2023-10-17T18:17:50.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:50.025+0000 7f8162bb3780 1 bucket sync caught up with source: 2023-10-17T18:17:50.027 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:17:50.028 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:17:50.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:50.025+0000 7f8162bb3780 0 bucket checkpoint complete 2023-10-17T18:17:50.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-9 zones={a2, a1} 2023-10-17T18:17:50.043 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.044 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.044 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.045 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:17:50.045 DEBUG:boto:Data: 2023-10-17T18:17:50.045 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.046 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.047 DEBUG:boto:Params: {} 2023-10-17T18:17:50.047 DEBUG:boto:Token: None 2023-10-17T18:17:50.048 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4= 2023-10-17T18:17:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4=', 'Content-Length': '0'} 2023-10-17T18:17:50.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b294879da06b4f5-00652ecfce-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.052 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.052 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.053 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.053 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:17:50.053 DEBUG:boto:Data: 2023-10-17T18:17:50.054 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.055 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.055 DEBUG:boto:Params: {} 2023-10-17T18:17:50.056 DEBUG:boto:Token: None 2023-10-17T18:17:50.056 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4= 2023-10-17T18:17:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4=', 'Content-Length': '0'} 2023-10-17T18:17:50.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e0df478b422a465-00652ecfce-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:17:50.060 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.061 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.061 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.061 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.062 DEBUG:boto:Method: GET 2023-10-17T18:17:50.062 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.063 DEBUG:boto:Data: 2023-10-17T18:17:50.063 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.064 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.064 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.064 DEBUG:boto:Params: {} 2023-10-17T18:17:50.065 DEBUG:boto:Token: None 2023-10-17T18:17:50.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/?versions 2023-10-17T18:17:50.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8= 2023-10-17T18:17:50.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8=', 'Content-Length': '0'} 2023-10-17T18:17:50.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ca34fe1d92a5e9-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.111 DEBUG:boto:b'hbmuog-9999falsemyobjnulltrue2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.112 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:50.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:17:50.113 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.114 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.114 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.114 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.115 DEBUG:boto:Method: GET 2023-10-17T18:17:50.115 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.116 DEBUG:boto:Data: 2023-10-17T18:17:50.116 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.117 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.117 DEBUG:boto:Params: {} 2023-10-17T18:17:50.118 DEBUG:boto:Token: None 2023-10-17T18:17:50.118 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/?versions 2023-10-17T18:17:50.119 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8= 2023-10-17T18:17:50.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8=', 'Content-Length': '0'} 2023-10-17T18:17:50.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516b7f650b44dc4e-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.123 DEBUG:boto:b'hbmuog-9999falsemyobjnulltrue2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.123 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:50.124 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.124 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.124 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.125 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.125 DEBUG:boto:Method: GET 2023-10-17T18:17:50.126 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.126 DEBUG:boto:Data: 2023-10-17T18:17:50.127 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.127 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.128 DEBUG:boto:Params: {} 2023-10-17T18:17:50.128 DEBUG:boto:Token: None 2023-10-17T18:17:50.129 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/?versions 2023-10-17T18:17:50.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8= 2023-10-17T18:17:50.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8=', 'Content-Length': '0'} 2023-10-17T18:17:50.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000296d0f69c86a1ec8-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.133 DEBUG:boto:b'hbmuog-9999falsemyobjnulltrue2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.133 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.134 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.134 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.135 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.135 DEBUG:boto:Method: GET 2023-10-17T18:17:50.136 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:17:50.136 DEBUG:boto:Data: 2023-10-17T18:17:50.137 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.137 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.138 DEBUG:boto:Params: {} 2023-10-17T18:17:50.138 DEBUG:boto:Token: None 2023-10-17T18:17:50.139 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/?versions 2023-10-17T18:17:50.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8= 2023-10-17T18:17:50.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZU0dD4bPvVsNSbolhxPKwu/awG8=', 'Content-Length': '0'} 2023-10-17T18:17:50.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae575211146188d-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.143 DEBUG:boto:b'hbmuog-9999falsemyobjnulltrue2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.144 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.144 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.145 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.145 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.146 DEBUG:boto:Method: GET 2023-10-17T18:17:50.146 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.147 DEBUG:boto:Data: 2023-10-17T18:17:50.147 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.148 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.148 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.148 DEBUG:boto:Params: {} 2023-10-17T18:17:50.149 DEBUG:boto:Token: None 2023-10-17T18:17:50.149 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.150 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk= 2023-10-17T18:17:50.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk=', 'Content-Length': '0'} 2023-10-17T18:17:50.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073c3e457347a86c8-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.154 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.154 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.154 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.155 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.155 DEBUG:boto:Method: GET 2023-10-17T18:17:50.156 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.156 DEBUG:boto:Data: 2023-10-17T18:17:50.157 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.157 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.157 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.158 DEBUG:boto:Params: {} 2023-10-17T18:17:50.158 DEBUG:boto:Token: None 2023-10-17T18:17:50.159 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.160 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk= 2023-10-17T18:17:50.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk=', 'Content-Length': '0'} 2023-10-17T18:17:50.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c35cee2306bc17ff-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.163 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.163 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.164 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.164 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.165 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.165 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.165 DEBUG:boto:Data: 2023-10-17T18:17:50.166 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.167 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.167 DEBUG:boto:Params: {} 2023-10-17T18:17:50.167 DEBUG:boto:Token: None 2023-10-17T18:17:50.168 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.168 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pajAXx0hLrnhO/sO0uL0ZtYAHiY= 2023-10-17T18:17:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pajAXx0hLrnhO/sO0uL0ZtYAHiY=', 'Content-Length': '0'} 2023-10-17T18:17:50.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8cf429e5b02d603-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.172 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.172 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.173 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.173 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.173 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.174 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.174 DEBUG:boto:Data: 2023-10-17T18:17:50.175 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.176 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.176 DEBUG:boto:Params: {} 2023-10-17T18:17:50.177 DEBUG:boto:Token: None 2023-10-17T18:17:50.177 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.177 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pajAXx0hLrnhO/sO0uL0ZtYAHiY= 2023-10-17T18:17:50.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pajAXx0hLrnhO/sO0uL0ZtYAHiY=', 'Content-Length': '0'} 2023-10-17T18:17:50.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6c2e0402d18eedd-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.181 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.181 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.182 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.182 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.183 DEBUG:boto:Method: GET 2023-10-17T18:17:50.183 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.184 DEBUG:boto:Data: 2023-10-17T18:17:50.184 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.185 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.186 DEBUG:boto:Params: {} 2023-10-17T18:17:50.186 DEBUG:boto:Token: None 2023-10-17T18:17:50.187 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk= 2023-10-17T18:17:50.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk=', 'Content-Length': '0'} 2023-10-17T18:17:50.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020a423546d86d70e-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.190 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.191 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.191 DEBUG:boto:path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.192 DEBUG:boto:auth_path=/hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.192 DEBUG:boto:Method: GET 2023-10-17T18:17:50.193 DEBUG:boto:Path: /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.193 DEBUG:boto:Data: 2023-10-17T18:17:50.194 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.194 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.194 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.195 DEBUG:boto:Params: {} 2023-10-17T18:17:50.195 DEBUG:boto:Token: None 2023-10-17T18:17:50.196 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj?versionId=null 2023-10-17T18:17:50.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk= 2023-10-17T18:17:50.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cuu2NSdKOI+85TXeC/6jvAa6UTk=', 'Content-Length': '0'} 2023-10-17T18:17:50.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee8b158b203e21ff-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.199 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.200 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.200 DEBUG:boto:path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.201 DEBUG:boto:auth_path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.201 DEBUG:boto:Method: GET 2023-10-17T18:17:50.202 DEBUG:boto:Path: /hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.202 DEBUG:boto:Data: 2023-10-17T18:17:50.203 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.203 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.203 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.204 DEBUG:boto:Params: {} 2023-10-17T18:17:50.204 DEBUG:boto:Token: None 2023-10-17T18:17:50.205 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI= 2023-10-17T18:17:50.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI=', 'Content-Length': '0'} 2023-10-17T18:17:50.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e46bfb3ec8cb5800-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.209 DEBUG:boto:b'hbmuog-9myobj1000falsemyobj2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.209 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.210 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.210 DEBUG:boto:path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.211 DEBUG:boto:auth_path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.211 DEBUG:boto:Method: GET 2023-10-17T18:17:50.212 DEBUG:boto:Path: /hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.212 DEBUG:boto:Data: 2023-10-17T18:17:50.213 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.214 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.214 DEBUG:boto:Params: {} 2023-10-17T18:17:50.214 DEBUG:boto:Token: None 2023-10-17T18:17:50.215 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI= 2023-10-17T18:17:50.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI=', 'Content-Length': '0'} 2023-10-17T18:17:50.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9374072b007637-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.219 DEBUG:boto:b'hbmuog-9myobj1000falsemyobj2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.220 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.220 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.220 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.221 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.221 DEBUG:boto:Data: 2023-10-17T18:17:50.222 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.223 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.223 DEBUG:boto:Params: {} 2023-10-17T18:17:50.223 DEBUG:boto:Token: None 2023-10-17T18:17:50.224 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w= 2023-10-17T18:17:50.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w=', 'Content-Length': '0'} 2023-10-17T18:17:50.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000954f19ff232d6567-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.228 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.228 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.229 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.229 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.230 DEBUG:boto:Data: 2023-10-17T18:17:50.230 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.231 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.231 DEBUG:boto:Params: {} 2023-10-17T18:17:50.232 DEBUG:boto:Token: None 2023-10-17T18:17:50.232 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.233 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w= 2023-10-17T18:17:50.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w=', 'Content-Length': '0'} 2023-10-17T18:17:50.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a5dcacf1de631c5-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.236 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.236 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.236 DEBUG:boto:Method: GET 2023-10-17T18:17:50.237 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.237 DEBUG:boto:Data: 2023-10-17T18:17:50.238 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.238 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.239 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.239 DEBUG:boto:Params: {} 2023-10-17T18:17:50.240 DEBUG:boto:Token: None 2023-10-17T18:17:50.240 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Wl40qaXK0RQ62dANCKT5TUK943A= 2023-10-17T18:17:50.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Wl40qaXK0RQ62dANCKT5TUK943A=', 'Content-Length': '0'} 2023-10-17T18:17:50.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005b6d1e2b43fda7b-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.286 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.287 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.287 DEBUG:boto:Method: GET 2023-10-17T18:17:50.288 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.288 DEBUG:boto:Data: 2023-10-17T18:17:50.289 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.290 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.290 DEBUG:boto:Params: {} 2023-10-17T18:17:50.291 DEBUG:boto:Token: None 2023-10-17T18:17:50.291 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.292 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Wl40qaXK0RQ62dANCKT5TUK943A= 2023-10-17T18:17:50.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Wl40qaXK0RQ62dANCKT5TUK943A=', 'Content-Length': '0'} 2023-10-17T18:17:50.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004216bf84be95742f-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.295 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.296 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.296 DEBUG:boto:path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.297 DEBUG:boto:auth_path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.297 DEBUG:boto:Method: GET 2023-10-17T18:17:50.297 DEBUG:boto:Path: /hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.298 DEBUG:boto:Data: 2023-10-17T18:17:50.298 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.299 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.299 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.300 DEBUG:boto:Params: {} 2023-10-17T18:17:50.300 DEBUG:boto:Token: None 2023-10-17T18:17:50.301 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.301 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI= 2023-10-17T18:17:50.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI=', 'Content-Length': '0'} 2023-10-17T18:17:50.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000507c95fe84c5f038-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.305 DEBUG:boto:b'hbmuog-9myobj1000falsemyobj2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.306 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.306 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.307 DEBUG:boto:path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.307 DEBUG:boto:auth_path=/hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.308 DEBUG:boto:Method: GET 2023-10-17T18:17:50.308 DEBUG:boto:Path: /hbmuog-9/?prefix=myobj 2023-10-17T18:17:50.309 DEBUG:boto:Data: 2023-10-17T18:17:50.309 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.310 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.310 DEBUG:boto:Params: {} 2023-10-17T18:17:50.311 DEBUG:boto:Token: None 2023-10-17T18:17:50.311 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI= 2023-10-17T18:17:50.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NEqSJQf30VrSN0tEsIuW9WwIgqI=', 'Content-Length': '0'} 2023-10-17T18:17:50.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f3cb80259f0ef4-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.315 DEBUG:boto:b'hbmuog-9myobj1000falsemyobj2023-10-17T18:17:34.654Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-9 zones={a2, a1} 2023-10-17T18:17:50.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:50.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:50.317 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:17:50.384 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:50.384 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:50.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:50.421+0000 7f2078bcb780 1 bucket sync caught up with source: 2023-10-17T18:17:50.422 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:17:50.423 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:17:50.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:50.421+0000 7f2078bcb780 0 bucket checkpoint complete 2023-10-17T18:17:50.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-10 zones={a1, a2} 2023-10-17T18:17:50.436 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.436 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.437 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.437 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:17:50.438 DEBUG:boto:Data: 2023-10-17T18:17:50.438 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.439 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.440 DEBUG:boto:Params: {} 2023-10-17T18:17:50.440 DEBUG:boto:Token: None 2023-10-17T18:17:50.440 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gopnXRkISnXttrmWDxPXR/VTU3M= 2023-10-17T18:17:50.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gopnXRkISnXttrmWDxPXR/VTU3M=', 'Content-Length': '0'} 2023-10-17T18:17:50.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009eb48d6c4491d53-00652ecfce-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.445 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.445 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.446 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.446 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:17:50.447 DEBUG:boto:Data: 2023-10-17T18:17:50.447 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.447 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.448 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.448 DEBUG:boto:Params: {} 2023-10-17T18:17:50.449 DEBUG:boto:Token: None 2023-10-17T18:17:50.449 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.450 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gopnXRkISnXttrmWDxPXR/VTU3M= 2023-10-17T18:17:50.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gopnXRkISnXttrmWDxPXR/VTU3M=', 'Content-Length': '0'} 2023-10-17T18:17:50.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027700654b99fbdac-00652ecfce-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:17:50.454 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.454 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.455 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.455 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.455 DEBUG:boto:Method: GET 2023-10-17T18:17:50.456 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.456 DEBUG:boto:Data: 2023-10-17T18:17:50.457 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.457 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.458 DEBUG:boto:Params: {} 2023-10-17T18:17:50.458 DEBUG:boto:Token: None 2023-10-17T18:17:50.459 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/?versions 2023-10-17T18:17:50.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk= 2023-10-17T18:17:50.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk=', 'Content-Length': '0'} 2023-10-17T18:17:50.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062030b73e3c01d4-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.463 DEBUG:boto:b'hbmuog-10999falsemyobjnulltrue2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.464 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:50.464 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:17:50.465 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.465 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.466 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.466 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.466 DEBUG:boto:Method: GET 2023-10-17T18:17:50.467 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.467 DEBUG:boto:Data: 2023-10-17T18:17:50.468 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.469 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.469 DEBUG:boto:Params: {} 2023-10-17T18:17:50.469 DEBUG:boto:Token: None 2023-10-17T18:17:50.470 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/?versions 2023-10-17T18:17:50.470 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk= 2023-10-17T18:17:50.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk=', 'Content-Length': '0'} 2023-10-17T18:17:50.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e2ab37d9233ca34-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.473 DEBUG:boto:b'hbmuog-10999falsemyobjnulltrue2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.474 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-17T18:17:50.475 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.475 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.475 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.476 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.476 DEBUG:boto:Method: GET 2023-10-17T18:17:50.477 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.477 DEBUG:boto:Data: 2023-10-17T18:17:50.478 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.479 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.479 DEBUG:boto:Params: {} 2023-10-17T18:17:50.480 DEBUG:boto:Token: None 2023-10-17T18:17:50.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/?versions 2023-10-17T18:17:50.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk= 2023-10-17T18:17:50.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk=', 'Content-Length': '0'} 2023-10-17T18:17:50.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b92677d43493bd6-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.484 DEBUG:boto:b'hbmuog-10999falsemyobjnulltrue2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.484 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.485 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.485 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.486 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.486 DEBUG:boto:Method: GET 2023-10-17T18:17:50.487 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:17:50.487 DEBUG:boto:Data: 2023-10-17T18:17:50.487 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.488 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.488 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.489 DEBUG:boto:Params: {} 2023-10-17T18:17:50.489 DEBUG:boto:Token: None 2023-10-17T18:17:50.490 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/?versions 2023-10-17T18:17:50.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk= 2023-10-17T18:17:50.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3YEPqT3FIuKBZwqbcBRE35WODsk=', 'Content-Length': '0'} 2023-10-17T18:17:50.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095571e22bf9d03d8-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.494 DEBUG:boto:b'hbmuog-10999falsemyobjnulltrue2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.495 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.495 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.496 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.496 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.497 DEBUG:boto:Method: GET 2023-10-17T18:17:50.497 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.498 DEBUG:boto:Data: 2023-10-17T18:17:50.498 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.499 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.499 DEBUG:boto:Params: {} 2023-10-17T18:17:50.500 DEBUG:boto:Token: None 2023-10-17T18:17:50.501 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0= 2023-10-17T18:17:50.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0=', 'Content-Length': '0'} 2023-10-17T18:17:50.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e34778a14566fccd-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.546 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.547 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.547 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.548 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.548 DEBUG:boto:Method: GET 2023-10-17T18:17:50.549 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.549 DEBUG:boto:Data: 2023-10-17T18:17:50.550 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.550 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.550 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.551 DEBUG:boto:Params: {} 2023-10-17T18:17:50.551 DEBUG:boto:Token: None 2023-10-17T18:17:50.552 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0= 2023-10-17T18:17:50.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0=', 'Content-Length': '0'} 2023-10-17T18:17:50.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a16c81242f75732c-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.556 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.556 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.556 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.557 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.557 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.558 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.558 DEBUG:boto:Data: 2023-10-17T18:17:50.558 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.559 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.560 DEBUG:boto:Params: {} 2023-10-17T18:17:50.560 DEBUG:boto:Token: None 2023-10-17T18:17:50.561 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iEi5HaN75FTweVXuV50s2dAnfuo= 2023-10-17T18:17:50.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iEi5HaN75FTweVXuV50s2dAnfuo=', 'Content-Length': '0'} 2023-10-17T18:17:50.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015f6dc612c21f0dd-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.564 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.564 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.565 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.565 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.565 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.566 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.566 DEBUG:boto:Data: 2023-10-17T18:17:50.566 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.567 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.568 DEBUG:boto:Params: {} 2023-10-17T18:17:50.568 DEBUG:boto:Token: None 2023-10-17T18:17:50.568 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iEi5HaN75FTweVXuV50s2dAnfuo= 2023-10-17T18:17:50.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iEi5HaN75FTweVXuV50s2dAnfuo=', 'Content-Length': '0'} 2023-10-17T18:17:50.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5a29c941724bc76-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.572 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.573 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.573 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.574 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.574 DEBUG:boto:Method: GET 2023-10-17T18:17:50.575 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.575 DEBUG:boto:Data: 2023-10-17T18:17:50.575 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.576 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.577 DEBUG:boto:Params: {} 2023-10-17T18:17:50.577 DEBUG:boto:Token: None 2023-10-17T18:17:50.578 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0= 2023-10-17T18:17:50.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0=', 'Content-Length': '0'} 2023-10-17T18:17:50.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b0dd708270a41d4-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.581 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.582 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.582 DEBUG:boto:path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.583 DEBUG:boto:auth_path=/hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.583 DEBUG:boto:Method: GET 2023-10-17T18:17:50.583 DEBUG:boto:Path: /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.584 DEBUG:boto:Data: 2023-10-17T18:17:50.584 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.585 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.586 DEBUG:boto:Params: {} 2023-10-17T18:17:50.586 DEBUG:boto:Token: None 2023-10-17T18:17:50.586 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj?versionId=null 2023-10-17T18:17:50.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0= 2023-10-17T18:17:50.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whOU9jQUfiPjlWCOFK0RJi/Sah0=', 'Content-Length': '0'} 2023-10-17T18:17:50.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f30c5e8815f6b6c0-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.590 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.590 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.591 DEBUG:boto:path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.591 DEBUG:boto:auth_path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.592 DEBUG:boto:Method: GET 2023-10-17T18:17:50.592 DEBUG:boto:Path: /hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.592 DEBUG:boto:Data: 2023-10-17T18:17:50.593 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.593 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.594 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.594 DEBUG:boto:Params: {} 2023-10-17T18:17:50.595 DEBUG:boto:Token: None 2023-10-17T18:17:50.595 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.596 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek= 2023-10-17T18:17:50.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek=', 'Content-Length': '0'} 2023-10-17T18:17:50.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da18e1adc6b0b67f-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.599 DEBUG:boto:b'hbmuog-10myobj1000falsemyobj2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.600 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.600 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.600 DEBUG:boto:path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.601 DEBUG:boto:auth_path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.601 DEBUG:boto:Method: GET 2023-10-17T18:17:50.602 DEBUG:boto:Path: /hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.602 DEBUG:boto:Data: 2023-10-17T18:17:50.603 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.603 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.604 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.604 DEBUG:boto:Params: {} 2023-10-17T18:17:50.605 DEBUG:boto:Token: None 2023-10-17T18:17:50.605 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.606 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek= 2023-10-17T18:17:50.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek=', 'Content-Length': '0'} 2023-10-17T18:17:50.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007074e5d2d3672396-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.609 DEBUG:boto:b'hbmuog-10myobj1000falsemyobj2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.610 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.610 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.610 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.611 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:17:50.611 DEBUG:boto:Data: 2023-10-17T18:17:50.612 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.613 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.613 DEBUG:boto:Params: {} 2023-10-17T18:17:50.613 DEBUG:boto:Token: None 2023-10-17T18:17:50.614 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj 2023-10-17T18:17:50.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/H318ncfPWQj4T6KKH+Qodw2Wo8= 2023-10-17T18:17:50.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/H318ncfPWQj4T6KKH+Qodw2Wo8=', 'Content-Length': '0'} 2023-10-17T18:17:50.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092a43d78eef111ef-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.617 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.618 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.618 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.619 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:17:50.619 DEBUG:boto:Data: 2023-10-17T18:17:50.619 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.620 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.621 DEBUG:boto:Params: {} 2023-10-17T18:17:50.621 DEBUG:boto:Token: None 2023-10-17T18:17:50.622 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj 2023-10-17T18:17:50.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/H318ncfPWQj4T6KKH+Qodw2Wo8= 2023-10-17T18:17:50.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/H318ncfPWQj4T6KKH+Qodw2Wo8=', 'Content-Length': '0'} 2023-10-17T18:17:50.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000345a799eee643df5-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-17T18:17:50.625 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.626 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.626 DEBUG:boto:Method: GET 2023-10-17T18:17:50.627 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:17:50.627 DEBUG:boto:Data: 2023-10-17T18:17:50.628 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.628 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.629 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.629 DEBUG:boto:Params: {} 2023-10-17T18:17:50.629 DEBUG:boto:Token: None 2023-10-17T18:17:50.630 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj 2023-10-17T18:17:50.630 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1RgzCmPKqYqSe3wR/XZbOwi11eM= 2023-10-17T18:17:50.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1RgzCmPKqYqSe3wR/XZbOwi11eM=', 'Content-Length': '0'} 2023-10-17T18:17:50.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ca60e4f01535157-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.634 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:17:50.634 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:17:50.635 DEBUG:boto:Method: GET 2023-10-17T18:17:50.635 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:17:50.635 DEBUG:boto:Data: 2023-10-17T18:17:50.636 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.636 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.637 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.637 DEBUG:boto:Params: {} 2023-10-17T18:17:50.638 DEBUG:boto:Token: None 2023-10-17T18:17:50.638 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/myobj 2023-10-17T18:17:50.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1RgzCmPKqYqSe3wR/XZbOwi11eM= 2023-10-17T18:17:50.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1RgzCmPKqYqSe3wR/XZbOwi11eM=', 'Content-Length': '0'} 2023-10-17T18:17:50.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002529101f411f2f5f-00652ecfce-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.641 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.642 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.642 DEBUG:boto:path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.643 DEBUG:boto:auth_path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.643 DEBUG:boto:Method: GET 2023-10-17T18:17:50.644 DEBUG:boto:Path: /hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.644 DEBUG:boto:Data: 2023-10-17T18:17:50.644 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.645 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.646 DEBUG:boto:Params: {} 2023-10-17T18:17:50.646 DEBUG:boto:Token: None 2023-10-17T18:17:50.647 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek= 2023-10-17T18:17:50.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek=', 'Content-Length': '0'} 2023-10-17T18:17:50.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067bace9413b7256c-00652ecfce-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.651 DEBUG:boto:b'hbmuog-10myobj1000falsemyobj2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.651 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:17:50.652 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:17:50.652 DEBUG:boto:path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.653 DEBUG:boto:auth_path=/hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.653 DEBUG:boto:Method: GET 2023-10-17T18:17:50.654 DEBUG:boto:Path: /hbmuog-10/?prefix=myobj 2023-10-17T18:17:50.654 DEBUG:boto:Data: 2023-10-17T18:17:50.655 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.655 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:17:50.655 DEBUG:boto:Port: 8001 2023-10-17T18:17:50.656 DEBUG:boto:Params: {} 2023-10-17T18:17:50.656 DEBUG:boto:Token: None 2023-10-17T18:17:50.657 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-10/ 2023-10-17T18:17:50.657 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek= 2023-10-17T18:17:50.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wckvN4b5B/lmI+3LAwPRdUdTLek=', 'Content-Length': '0'} 2023-10-17T18:17:50.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f721f31acf3ae66f-00652ecfce-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.661 DEBUG:boto:b'hbmuog-10myobj1000falsemyobj2023-10-17T18:17:34.711Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:17:50.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-10 zones={a1, a2} 2023-10-17T18:17:50.662 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:17:50.662 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:17:50.662 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.663 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:17:50.663 DEBUG:boto:Data: 2023-10-17T18:17:50.664 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.664 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.664 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.665 DEBUG:boto:Params: {} 2023-10-17T18:17:50.665 DEBUG:boto:Token: None 2023-10-17T18:17:50.666 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/ 2023-10-17T18:17:50.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4= 2023-10-17T18:17:50.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FbaxhZIYgwauppNKuzrxF13m+E4=', 'Content-Length': '0'} 2023-10-17T18:17:50.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bfca4b49f5202be-00652ecfce-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.669 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.670 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.670 DEBUG:boto:Method: HEAD 2023-10-17T18:17:50.671 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.671 DEBUG:boto:Data: 2023-10-17T18:17:50.671 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.672 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.672 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.673 DEBUG:boto:Params: {} 2023-10-17T18:17:50.673 DEBUG:boto:Token: None 2023-10-17T18:17:50.674 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.674 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w= 2023-10-17T18:17:50.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dj9YX9s8JENWIfPSGeyCt8p+S4w=', 'Content-Length': '0'} 2023-10-17T18:17:50.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb5633036308ea92-00652ecfce-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.677 DEBUG:boto:path=/hbmuog-9/myobj 2023-10-17T18:17:50.678 DEBUG:boto:auth_path=/hbmuog-9/myobj 2023-10-17T18:17:50.678 DEBUG:boto:Method: DELETE 2023-10-17T18:17:50.678 DEBUG:boto:Path: /hbmuog-9/myobj 2023-10-17T18:17:50.679 DEBUG:boto:Data: 2023-10-17T18:17:50.679 DEBUG:boto:Headers: {} 2023-10-17T18:17:50.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:17:50.680 DEBUG:boto:Port: 8000 2023-10-17T18:17:50.680 DEBUG:boto:Params: {} 2023-10-17T18:17:50.681 DEBUG:boto:Token: None 2023-10-17T18:17:50.681 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:17:50 GMT /hbmuog-9/myobj 2023-10-17T18:17:50.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YOuJTZQXOrtQZ5MHS0YomF4kVIM= 2023-10-17T18:17:50.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YOuJTZQXOrtQZ5MHS0YomF4kVIM=', 'Content-Length': '0'} 2023-10-17T18:17:50.686 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ee20adcd8e6feae-00652ecfce-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:17:50.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:50.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:17:50.688 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:17:50.747 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:17:50.747 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:17:50.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:50.780+0000 7f4dbae1e780 1 waiting for incremental sync to catch up: 2023-10-17T18:17:50.782 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:17:50.782 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:17:55.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:17:55.781+0000 7f4dbae1e780 1 waiting for incremental sync to catch up: 2023-10-17T18:17:55.783 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:17:55.783 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:18:00.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:00.783+0000 7f4dbae1e780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:00.786 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:18:00.787 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:18:05.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:05.784+0000 7f4dbae1e780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:05.785 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.6.6, , , , , , , , , ] 2023-10-17T18:18:05.786 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:18:10.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:10.785+0000 7f4dbae1e780 1 bucket sync caught up with source: 2023-10-17T18:18:10.787 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:18:10.787 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.8.6, , , , , , , , , ] 2023-10-17T18:18:10.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:10.785+0000 7f4dbae1e780 0 bucket checkpoint complete 2023-10-17T18:18:10.804 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-9 zones={a2, a1} 2023-10-17T18:18:10.805 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.806 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.806 DEBUG:boto:Method: HEAD 2023-10-17T18:18:10.807 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:18:10.807 DEBUG:boto:Data: 2023-10-17T18:18:10.807 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.808 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.808 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.808 DEBUG:boto:Params: {} 2023-10-17T18:18:10.809 DEBUG:boto:Token: None 2023-10-17T18:18:10.810 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/ 2023-10-17T18:18:10.810 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T/Q6E5DgsrJCmo5V8xys+7F5QgY= 2023-10-17T18:18:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T/Q6E5DgsrJCmo5V8xys+7F5QgY=', 'Content-Length': '0'} 2023-10-17T18:18:10.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b946dcbed198c18-00652ecfe2-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.814 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.814 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.814 DEBUG:boto:Method: HEAD 2023-10-17T18:18:10.815 DEBUG:boto:Path: /hbmuog-9/ 2023-10-17T18:18:10.815 DEBUG:boto:Data: 2023-10-17T18:18:10.816 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.816 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:10.816 DEBUG:boto:Port: 8000 2023-10-17T18:18:10.817 DEBUG:boto:Params: {} 2023-10-17T18:18:10.817 DEBUG:boto:Token: None 2023-10-17T18:18:10.818 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/ 2023-10-17T18:18:10.818 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T/Q6E5DgsrJCmo5V8xys+7F5QgY= 2023-10-17T18:18:10.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T/Q6E5DgsrJCmo5V8xys+7F5QgY=', 'Content-Length': '0'} 2023-10-17T18:18:10.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001269364fa934b603-00652ecfe2-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:18:10.822 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.822 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.823 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.823 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.823 DEBUG:boto:Method: GET 2023-10-17T18:18:10.824 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.824 DEBUG:boto:Data: 2023-10-17T18:18:10.825 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.825 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.825 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.826 DEBUG:boto:Params: {} 2023-10-17T18:18:10.826 DEBUG:boto:Token: None 2023-10-17T18:18:10.827 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/?versions 2023-10-17T18:18:10.827 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ= 2023-10-17T18:18:10.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ=', 'Content-Length': '0'} 2023-10-17T18:18:10.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f285102a34fe0d-00652ecfe2-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.831 DEBUG:boto:b'hbmuog-9999false' 2023-10-17T18:18:10.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:18:10.832 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.832 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.833 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.833 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.833 DEBUG:boto:Method: GET 2023-10-17T18:18:10.834 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.834 DEBUG:boto:Data: 2023-10-17T18:18:10.835 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:10.836 DEBUG:boto:Port: 8000 2023-10-17T18:18:10.836 DEBUG:boto:Params: {} 2023-10-17T18:18:10.836 DEBUG:boto:Token: None 2023-10-17T18:18:10.837 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/?versions 2023-10-17T18:18:10.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ= 2023-10-17T18:18:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ=', 'Content-Length': '0'} 2023-10-17T18:18:10.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d2b9ed8210af73-00652ecfe2-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.841 DEBUG:boto:b'hbmuog-9999false' 2023-10-17T18:18:10.841 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.842 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.842 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.843 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.843 DEBUG:boto:Method: GET 2023-10-17T18:18:10.843 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.844 DEBUG:boto:Data: 2023-10-17T18:18:10.844 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.844 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.845 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.845 DEBUG:boto:Params: {} 2023-10-17T18:18:10.846 DEBUG:boto:Token: None 2023-10-17T18:18:10.846 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/?versions 2023-10-17T18:18:10.846 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ= 2023-10-17T18:18:10.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ=', 'Content-Length': '0'} 2023-10-17T18:18:10.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f256e5bc868ea2-00652ecfe2-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.890 DEBUG:boto:b'hbmuog-9999false' 2023-10-17T18:18:10.891 DEBUG:boto:path=/hbmuog-9/ 2023-10-17T18:18:10.892 DEBUG:boto:auth_path=/hbmuog-9/ 2023-10-17T18:18:10.892 DEBUG:boto:path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.892 DEBUG:boto:auth_path=/hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.893 DEBUG:boto:Method: GET 2023-10-17T18:18:10.893 DEBUG:boto:Path: /hbmuog-9/?versions&max-keys=999 2023-10-17T18:18:10.894 DEBUG:boto:Data: 2023-10-17T18:18:10.894 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:10.895 DEBUG:boto:Port: 8000 2023-10-17T18:18:10.895 DEBUG:boto:Params: {} 2023-10-17T18:18:10.896 DEBUG:boto:Token: None 2023-10-17T18:18:10.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-9/?versions 2023-10-17T18:18:10.897 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ= 2023-10-17T18:18:10.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fSQiTxiUdldduo6yp7Wba+epHHQ=', 'Content-Length': '0'} 2023-10-17T18:18:10.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d96bcfeb04ee15-00652ecfe2-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.900 DEBUG:boto:b'hbmuog-9999false' 2023-10-17T18:18:10.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-9 zones={a2, a1} 2023-10-17T18:18:10.901 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:10.902 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:10.902 DEBUG:boto:Method: HEAD 2023-10-17T18:18:10.902 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:18:10.903 DEBUG:boto:Data: 2023-10-17T18:18:10.903 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.904 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.904 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.904 DEBUG:boto:Params: {} 2023-10-17T18:18:10.905 DEBUG:boto:Token: None 2023-10-17T18:18:10.905 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-10/ 2023-10-17T18:18:10.905 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oUodWvVgYx3KxO4VwDz3E3VFSGk= 2023-10-17T18:18:10.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oUodWvVgYx3KxO4VwDz3E3VFSGk=', 'Content-Length': '0'} 2023-10-17T18:18:10.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afaa9267103bc826-00652ecfe2-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.909 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:18:10.910 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:18:10.910 DEBUG:boto:Method: HEAD 2023-10-17T18:18:10.910 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:18:10.911 DEBUG:boto:Data: 2023-10-17T18:18:10.911 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.911 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.912 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.912 DEBUG:boto:Params: {} 2023-10-17T18:18:10.913 DEBUG:boto:Token: None 2023-10-17T18:18:10.913 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-10/myobj 2023-10-17T18:18:10.913 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oqVHHhXq9WoQukiUYmP3FZtYXt0= 2023-10-17T18:18:10.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oqVHHhXq9WoQukiUYmP3FZtYXt0=', 'Content-Length': '0'} 2023-10-17T18:18:10.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:17:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3ccf2d5d984797a-00652ecfe2-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.916 DEBUG:boto:path=/hbmuog-10/myobj 2023-10-17T18:18:10.917 DEBUG:boto:auth_path=/hbmuog-10/myobj 2023-10-17T18:18:10.917 DEBUG:boto:Method: DELETE 2023-10-17T18:18:10.918 DEBUG:boto:Path: /hbmuog-10/myobj 2023-10-17T18:18:10.918 DEBUG:boto:Data: 2023-10-17T18:18:10.918 DEBUG:boto:Headers: {} 2023-10-17T18:18:10.919 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:10.919 DEBUG:boto:Port: 8001 2023-10-17T18:18:10.920 DEBUG:boto:Params: {} 2023-10-17T18:18:10.920 DEBUG:boto:Token: None 2023-10-17T18:18:10.920 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:18:10 GMT /hbmuog-10/myobj 2023-10-17T18:18:10.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/U0P4SGBtXFhJr2C1NIa4D0ge58= 2023-10-17T18:18:10.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/U0P4SGBtXFhJr2C1NIa4D0ge58=', 'Content-Length': '0'} 2023-10-17T18:18:10.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008be3052a9b4698d0-00652ecfe2-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:18:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:10.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:10.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:10.929 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:18:10.994 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:10.994 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:11.030+0000 7f1d59c83780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:11.031 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:18:11.032 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:16.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:16.033+0000 7f1d59c83780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:16.033 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:18:16.034 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:21.034+0000 7f1d59c83780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:21.035 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:18:21.035 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:26.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:26.034+0000 7f1d59c83780 1 waiting for incremental sync to catch up: 2023-10-17T18:18:26.036 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2023-10-17T18:18:26.036 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:31.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:31.036+0000 7f1d59c83780 1 bucket sync caught up with source: 2023-10-17T18:18:31.038 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:31.038 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.14.6, , , , , , , , , ] 2023-10-17T18:18:31.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:31.036+0000 7f1d59c83780 0 bucket checkpoint complete 2023-10-17T18:18:31.053 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-10 zones={a1, a2} 2023-10-17T18:18:31.054 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.054 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.055 DEBUG:boto:Method: HEAD 2023-10-17T18:18:31.055 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:18:31.056 DEBUG:boto:Data: 2023-10-17T18:18:31.056 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:31.057 DEBUG:boto:Port: 8000 2023-10-17T18:18:31.057 DEBUG:boto:Params: {} 2023-10-17T18:18:31.058 DEBUG:boto:Token: None 2023-10-17T18:18:31.058 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/ 2023-10-17T18:18:31.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TF+2TNZpFZ/jx0xrONks2bbh9co= 2023-10-17T18:18:31.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TF+2TNZpFZ/jx0xrONks2bbh9co=', 'Content-Length': '0'} 2023-10-17T18:18:31.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afec79035c4a8727-00652ecff7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.063 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.063 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.064 DEBUG:boto:Method: HEAD 2023-10-17T18:18:31.064 DEBUG:boto:Path: /hbmuog-10/ 2023-10-17T18:18:31.065 DEBUG:boto:Data: 2023-10-17T18:18:31.065 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:31.066 DEBUG:boto:Port: 8001 2023-10-17T18:18:31.066 DEBUG:boto:Params: {} 2023-10-17T18:18:31.067 DEBUG:boto:Token: None 2023-10-17T18:18:31.067 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/ 2023-10-17T18:18:31.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TF+2TNZpFZ/jx0xrONks2bbh9co= 2023-10-17T18:18:31.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TF+2TNZpFZ/jx0xrONks2bbh9co=', 'Content-Length': '0'} 2023-10-17T18:18:31.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000506e59053774a38a-00652ecff7-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.071 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:18:31.072 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.072 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.072 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.073 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.073 DEBUG:boto:Method: GET 2023-10-17T18:18:31.074 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.075 DEBUG:boto:Data: 2023-10-17T18:18:31.075 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:31.076 DEBUG:boto:Port: 8000 2023-10-17T18:18:31.077 DEBUG:boto:Params: {} 2023-10-17T18:18:31.077 DEBUG:boto:Token: None 2023-10-17T18:18:31.078 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/?versions 2023-10-17T18:18:31.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw= 2023-10-17T18:18:31.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw=', 'Content-Length': '0'} 2023-10-17T18:18:31.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7faabcad462081e-00652ecff7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.081 DEBUG:boto:b'hbmuog-10999false' 2023-10-17T18:18:31.085 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:18:31.086 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.086 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.087 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.087 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.088 DEBUG:boto:Method: GET 2023-10-17T18:18:31.088 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.089 DEBUG:boto:Data: 2023-10-17T18:18:31.089 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.089 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:31.090 DEBUG:boto:Port: 8001 2023-10-17T18:18:31.090 DEBUG:boto:Params: {} 2023-10-17T18:18:31.091 DEBUG:boto:Token: None 2023-10-17T18:18:31.091 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/?versions 2023-10-17T18:18:31.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw= 2023-10-17T18:18:31.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw=', 'Content-Length': '0'} 2023-10-17T18:18:31.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043dc3ba3d02f33f6-00652ecff7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.095 DEBUG:boto:b'hbmuog-10999false' 2023-10-17T18:18:31.096 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.097 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.097 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.097 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.098 DEBUG:boto:Method: GET 2023-10-17T18:18:31.098 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.099 DEBUG:boto:Data: 2023-10-17T18:18:31.099 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:31.100 DEBUG:boto:Port: 8000 2023-10-17T18:18:31.100 DEBUG:boto:Params: {} 2023-10-17T18:18:31.101 DEBUG:boto:Token: None 2023-10-17T18:18:31.101 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/?versions 2023-10-17T18:18:31.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw= 2023-10-17T18:18:31.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw=', 'Content-Length': '0'} 2023-10-17T18:18:31.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a419fb57bb23234e-00652ecff7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.106 DEBUG:boto:b'hbmuog-10999false' 2023-10-17T18:18:31.106 DEBUG:boto:path=/hbmuog-10/ 2023-10-17T18:18:31.107 DEBUG:boto:auth_path=/hbmuog-10/ 2023-10-17T18:18:31.107 DEBUG:boto:path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.107 DEBUG:boto:auth_path=/hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.108 DEBUG:boto:Method: GET 2023-10-17T18:18:31.108 DEBUG:boto:Path: /hbmuog-10/?versions&max-keys=999 2023-10-17T18:18:31.109 DEBUG:boto:Data: 2023-10-17T18:18:31.109 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:31.110 DEBUG:boto:Port: 8001 2023-10-17T18:18:31.110 DEBUG:boto:Params: {} 2023-10-17T18:18:31.111 DEBUG:boto:Token: None 2023-10-17T18:18:31.111 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-10/?versions 2023-10-17T18:18:31.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw= 2023-10-17T18:18:31.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vOiGCJVLKtNZsMMNbCsJ70gwMiw=', 'Content-Length': '0'} 2023-10-17T18:18:31.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6e7a71d49b2f99-00652ecff7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.158 DEBUG:boto:b'hbmuog-10999false' 2023-10-17T18:18:31.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-10 zones={a1, a2} 2023-10-17T18:18:31.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-10-17T18:18:31.161 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-11 2023-10-17T18:18:31.161 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:18:31.162 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:18:31.162 DEBUG:boto:Method: PUT 2023-10-17T18:18:31.163 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:18:31.163 DEBUG:boto:Data: 2023-10-17T18:18:31.164 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:31.164 DEBUG:boto:Port: 8000 2023-10-17T18:18:31.165 DEBUG:boto:Params: {} 2023-10-17T18:18:31.165 DEBUG:boto:Token: None 2023-10-17T18:18:31.166 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-11/ 2023-10-17T18:18:31.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gEizRTKBUy8iI7xokDVq4AN996E= 2023-10-17T18:18:31.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gEizRTKBUy8iI7xokDVq4AN996E=', 'Content-Length': '0'} 2023-10-17T18:18:31.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0f8fe807238c9e5-00652ecff7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.182 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-12 2023-10-17T18:18:31.182 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:18:31.183 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:18:31.183 DEBUG:boto:Method: PUT 2023-10-17T18:18:31.184 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:18:31.184 DEBUG:boto:Data: 2023-10-17T18:18:31.184 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:31.185 DEBUG:boto:Port: 8001 2023-10-17T18:18:31.186 DEBUG:boto:Params: {} 2023-10-17T18:18:31.186 DEBUG:boto:Token: None 2023-10-17T18:18:31.187 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-12/ 2023-10-17T18:18:31.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ckEW03cyUWhKa3XDbbVnR/nihNI= 2023-10-17T18:18:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ckEW03cyUWhKa3XDbbVnR/nihNI=', 'Content-Length': '0'} 2023-10-17T18:18:31.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026eb9f38d2969b0c-00652ecff7-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.212 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:18:31.213 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:18:31.213 DEBUG:boto:path=/hbmuog-11/?versioning 2023-10-17T18:18:31.214 DEBUG:boto:auth_path=/hbmuog-11/?versioning 2023-10-17T18:18:31.214 DEBUG:boto:Method: PUT 2023-10-17T18:18:31.215 DEBUG:boto:Path: /hbmuog-11/?versioning 2023-10-17T18:18:31.215 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:18:31.216 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.216 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:31.216 DEBUG:boto:Port: 8000 2023-10-17T18:18:31.217 DEBUG:boto:Params: {} 2023-10-17T18:18:31.217 DEBUG:boto:Token: None 2023-10-17T18:18:31.218 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-11/?versioning 2023-10-17T18:18:31.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ktWHUjFG2Knvz37UtQU5i+o5eR0= 2023-10-17T18:18:31.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ktWHUjFG2Knvz37UtQU5i+o5eR0=', 'Content-Length': '228'} 2023-10-17T18:18:31.224 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000856a8db197511e33-00652ecff7-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.225 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:18:31.225 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:18:31.226 DEBUG:boto:path=/hbmuog-12/?versioning 2023-10-17T18:18:31.226 DEBUG:boto:auth_path=/hbmuog-12/?versioning 2023-10-17T18:18:31.226 DEBUG:boto:Method: PUT 2023-10-17T18:18:31.227 DEBUG:boto:Path: /hbmuog-12/?versioning 2023-10-17T18:18:31.227 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:18:31.228 DEBUG:boto:Headers: {} 2023-10-17T18:18:31.228 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:31.229 DEBUG:boto:Port: 8001 2023-10-17T18:18:31.229 DEBUG:boto:Params: {} 2023-10-17T18:18:31.229 DEBUG:boto:Token: None 2023-10-17T18:18:31.230 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:18:31 GMT /hbmuog-12/?versioning 2023-10-17T18:18:31.230 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c0eKfSJ33/thuldLVPe4Lk1O6pE= 2023-10-17T18:18:31.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c0eKfSJ33/thuldLVPe4Lk1O6pE=', 'Content-Length': '228'} 2023-10-17T18:18:31.240 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c0c0b332538b0ef-00652ecff7-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:31.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:31.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:31.242 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:18:31.294 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:31.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:31.326 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:18:31.338 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:18:31.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.338 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566711.236903_150.1", 2023-10-17T18:18:31.338 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:18:31.236903Z" 2023-10-17T18:18:31.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.339 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:18:31.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:18:31.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.340 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:31.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:18:31.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.341 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:31.341 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:18:31.341 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.342 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:18:31.343 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566711.236903_150.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:18:31.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:18:31.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:31.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:31.345 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:18:31.396 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:31.396 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:31.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:31.423+0000 7f5399f21780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:18:31.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:31.424+0000 7f5399f21780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:18:31.441 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:18:31.441 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:18:31.441 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:18:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:18:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:18:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:18:31.442 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:18:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.443 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:18:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566711.173496_139.1", 2023-10-17T18:18:31.444 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:18:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:18:31.173496Z", 2023-10-17T18:18:31.445 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:31.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.446 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:18:31.447 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:31.447 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:31.447 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:18:31.447 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:31.448 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:31.448 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:31.448 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:18:31.448 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:31.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.449 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:18:31.450 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:31.450 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:31.450 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:31.450 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:31.450 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:31.451 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:31.451 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:18:31.451 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:31.451 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:31.452 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:18:31.452 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:31.453 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:31.453 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:31.453 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:31.453 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:31.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:31.454 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:18:31.454 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:31.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:31.455 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:18:31.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.455 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:18:31.455 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:18:31.456 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:18:31.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:31.456 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:18:31Z" 2023-10-17T18:18:31.456 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:18:31.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566711.173496_139.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:18:31.173496Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:18:31Z" } 2023-10-17T18:18:31.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566711.173496_139.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:18:31.173496Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:18:31.459 DEBUG:rgw_multi.tests:log_status={0: '1_1697566711.236903_150.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:18:31.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566711.173496_139.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:18:31.460 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697566711.236903_150.1 target=1_1697566711.173496_139.1 2023-10-17T18:18:36.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:36.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:18:36.462 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:18:36.522 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:36.522 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:36.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:36.553+0000 7f072fe75780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:18:36.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:36.554+0000 7f072fe75780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:18:36.568 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:18:36.568 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:18:36.568 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:18:36.568 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:18:36.569 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:18:36.569 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:18:36.569 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:36.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.569 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:18:36.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:36.570 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:18:36.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:36.570 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:36.571 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566711.236903_150.1", 2023-10-17T18:18:36.571 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:36.571 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:18:36.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:36.571 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:18:31.236903Z", 2023-10-17T18:18:36.572 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:36.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:36.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:36.573 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:18:36.573 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:36.573 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:36.573 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:18:36.573 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:36.574 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:36.574 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:36.574 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:18:36.574 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:36.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:36.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:36.575 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:18:36.575 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:36.576 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:36.576 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:36.576 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:36.576 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:36.577 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:36.577 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:18:36.577 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:36.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:36.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:18:36.578 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:18:36.578 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:18:36.578 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:18:36.578 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:18:36.579 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:18:36.579 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:18:36.579 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:18:36.579 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:18:36.579 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:18:36.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:36.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:18:36.580 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:18:36.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.581 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:18:36.581 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:18:36.581 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:18:36.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:18:36.581 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:18:36Z" 2023-10-17T18:18:36.582 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:18:36.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566711.236903_150.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:18:31.236903Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:18:36Z" } 2023-10-17T18:18:36.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566711.236903_150.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:18:31.236903Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:18:36.584 DEBUG:rgw_multi.tests:log_status={0: '1_1697566711.236903_150.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:18:36.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566711.236903_150.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:18:36.585 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:18:36.585 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:18:36.585 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:18:36.586 DEBUG:boto:Method: HEAD 2023-10-17T18:18:36.586 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:18:36.587 DEBUG:boto:Data: 2023-10-17T18:18:36.587 DEBUG:boto:Headers: {} 2023-10-17T18:18:36.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:36.588 DEBUG:boto:Port: 8000 2023-10-17T18:18:36.588 DEBUG:boto:Params: {} 2023-10-17T18:18:36.589 DEBUG:boto:Token: None 2023-10-17T18:18:36.589 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:36 GMT /hbmuog-11/ 2023-10-17T18:18:36.590 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:74s7vv4EWYxjEldwXpuBk5cZyFY= 2023-10-17T18:18:36.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:74s7vv4EWYxjEldwXpuBk5cZyFY=', 'Content-Length': '0'} 2023-10-17T18:18:36.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f67417aa25aea412-00652ecffc-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:36.593 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:18:36.594 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:18:36.594 DEBUG:boto:Method: PUT 2023-10-17T18:18:36.595 DEBUG:boto:Path: /hbmuog-11/dummy 2023-10-17T18:18:36.595 DEBUG:boto:Data: 2023-10-17T18:18:36.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:18:36.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:18:36.596 DEBUG:boto:Port: 8000 2023-10-17T18:18:36.597 DEBUG:boto:Params: {} 2023-10-17T18:18:36.597 DEBUG:boto:Token: None 2023-10-17T18:18:36.598 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:18:36 GMT /hbmuog-11/dummy 2023-10-17T18:18:36.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:apkze3itBv3BQ6Fo119yjXGS/aY= 2023-10-17T18:18:36.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:18:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:apkze3itBv3BQ6Fo119yjXGS/aY='} 2023-10-17T18:18:36.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-amz-request-id', 'tx000000e5931664a100b7f-00652ecffc-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:36.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:36.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:36.611 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:18:36.672 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:36.672 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:36.705 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:36.704+0000 7ff13645b780 1 waiting to reach incremental sync.. 2023-10-17T18:18:41.706 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:41.705+0000 7ff13645b780 1 waiting to reach incremental sync.. 2023-10-17T18:18:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:46.706+0000 7ff13645b780 1 waiting to reach incremental sync.. 2023-10-17T18:18:51.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:51.708+0000 7ff13645b780 1 bucket sync caught up with source: 2023-10-17T18:18:51.709 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:18:51.709 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:18:51.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:51.708+0000 7ff13645b780 0 bucket checkpoint complete 2023-10-17T18:18:51.724 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:18:51.725 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:18:51.725 DEBUG:boto:Method: HEAD 2023-10-17T18:18:51.726 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:18:51.726 DEBUG:boto:Data: 2023-10-17T18:18:51.727 DEBUG:boto:Headers: {} 2023-10-17T18:18:51.727 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:51.727 DEBUG:boto:Port: 8001 2023-10-17T18:18:51.728 DEBUG:boto:Params: {} 2023-10-17T18:18:51.728 DEBUG:boto:Token: None 2023-10-17T18:18:51.729 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:18:51 GMT /hbmuog-12/ 2023-10-17T18:18:51.729 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6dcLEm4hQfkQWbtgkMIIM+foa/M= 2023-10-17T18:18:51.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:18:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6dcLEm4hQfkQWbtgkMIIM+foa/M=', 'Content-Length': '0'} 2023-10-17T18:18:51.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f3de7503544770f-00652ed00b-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:51.733 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:18:51.734 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:18:51.734 DEBUG:boto:Method: PUT 2023-10-17T18:18:51.734 DEBUG:boto:Path: /hbmuog-12/dummy 2023-10-17T18:18:51.735 DEBUG:boto:Data: 2023-10-17T18:18:51.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:18:51.736 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:18:51.736 DEBUG:boto:Port: 8001 2023-10-17T18:18:51.736 DEBUG:boto:Params: {} 2023-10-17T18:18:51.737 DEBUG:boto:Token: None 2023-10-17T18:18:51.737 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:18:51 GMT /hbmuog-12/dummy 2023-10-17T18:18:51.738 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tQVCOMDzTN+8HiQX8dgKAToT+j0= 2023-10-17T18:18:51.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:18:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tQVCOMDzTN+8HiQX8dgKAToT+j0='} 2023-10-17T18:18:51.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-amz-request-id', 'tx000002af85908015983af-00652ed00b-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:18:51.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:51.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:18:51.752 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:18:51.817 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:18:51.817 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:18:51.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:51.853+0000 7f79f3832780 1 waiting to reach incremental sync.. 2023-10-17T18:18:56.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:18:56.854+0000 7f79f3832780 1 waiting to reach incremental sync.. 2023-10-17T18:19:01.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:01.853+0000 7f79f3832780 1 waiting to reach incremental sync.. 2023-10-17T18:19:06.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:06.854+0000 7f79f3832780 1 waiting to reach incremental sync.. 2023-10-17T18:19:11.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:11.856+0000 7f79f3832780 1 bucket sync caught up with source: 2023-10-17T18:19:11.857 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.39.11, , , , , ] 2023-10-17T18:19:11.858 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.39.11, , , , , ] 2023-10-17T18:19:11.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:11.856+0000 7f79f3832780 0 bucket checkpoint complete 2023-10-17T18:19:11.873 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:11.874 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:11.874 DEBUG:boto:Method: HEAD 2023-10-17T18:19:11.875 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:11.875 DEBUG:boto:Data: 2023-10-17T18:19:11.876 DEBUG:boto:Headers: {} 2023-10-17T18:19:11.876 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.876 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.877 DEBUG:boto:Params: {} 2023-10-17T18:19:11.877 DEBUG:boto:Token: None 2023-10-17T18:19:11.878 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/ 2023-10-17T18:19:11.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OF0LnmTVDGXsy3Kd9JuZgxxVrdo= 2023-10-17T18:19:11.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OF0LnmTVDGXsy3Kd9JuZgxxVrdo=', 'Content-Length': '0'} 2023-10-17T18:19:11.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004959cafda6d5b79c-00652ed01f-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:11.882 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.883 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.883 DEBUG:boto:Method: PUT 2023-10-17T18:19:11.883 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:19:11.884 DEBUG:boto:Data: 2023-10-17T18:19:11.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:11.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.885 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.885 DEBUG:boto:Params: {} 2023-10-17T18:19:11.886 DEBUG:boto:Token: None 2023-10-17T18:19:11.886 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/obj-a1 2023-10-17T18:19:11.887 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6Ccl+889vvLJ1/S1rs2qOpHCL2A= 2023-10-17T18:19:11.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6Ccl+889vvLJ1/S1rs2qOpHCL2A='} 2023-10-17T18:19:11.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-amz-request-id', 'tx00000ad546178c847954f-00652ed01f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:11.900 DEBUG:rgw_multi.tests:version1 id=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:11.900 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.901 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.901 DEBUG:boto:Method: PUT 2023-10-17T18:19:11.902 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:19:11.902 DEBUG:boto:Data: 2023-10-17T18:19:11.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:11.903 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.904 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.904 DEBUG:boto:Params: {} 2023-10-17T18:19:11.905 DEBUG:boto:Token: None 2023-10-17T18:19:11.905 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/obj-a1 2023-10-17T18:19:11.906 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M34oCpp+QK+Wy9ktqPTw/A3PZoE= 2023-10-17T18:19:11.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M34oCpp+QK+Wy9ktqPTw/A3PZoE='} 2023-10-17T18:19:11.916 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73'), ('x-amz-request-id', 'tx00000f99b76f034d14530-00652ed01f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:11.917 DEBUG:rgw_multi.tests:version2 id=Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73 2023-10-17T18:19:11.917 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.917 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.918 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73 2023-10-17T18:19:11.919 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73 2023-10-17T18:19:11.919 DEBUG:boto:Method: DELETE 2023-10-17T18:19:11.919 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73 2023-10-17T18:19:11.920 DEBUG:boto:Data: 2023-10-17T18:19:11.920 DEBUG:boto:Headers: {} 2023-10-17T18:19:11.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.921 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.921 DEBUG:boto:Params: {} 2023-10-17T18:19:11.922 DEBUG:boto:Token: None 2023-10-17T18:19:11.922 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/obj-a1?versionId=Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73 2023-10-17T18:19:11.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6f3vjEIoe4d1uCQCW8irenTwH6w= 2023-10-17T18:19:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6f3vjEIoe4d1uCQCW8irenTwH6w=', 'Content-Length': '0'} 2023-10-17T18:19:11.934 DEBUG:boto:Response headers: [('x-amz-version-id', 'Ij0z62ZZoTyxDyrXyp9D1.DIWgJmN73'), ('x-amz-request-id', 'tx0000089894b55033e9e30-00652ed01f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:11.934 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.935 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.935 DEBUG:boto:Method: PUT 2023-10-17T18:19:11.936 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:19:11.936 DEBUG:boto:Data: 2023-10-17T18:19:11.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:11.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.937 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.938 DEBUG:boto:Params: {} 2023-10-17T18:19:11.938 DEBUG:boto:Token: None 2023-10-17T18:19:11.939 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/obj-a1 2023-10-17T18:19:11.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6jE1SY2jVmxMoTyxs+zx1I7rDrQ= 2023-10-17T18:19:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6jE1SY2jVmxMoTyxs+zx1I7rDrQ='} 2023-10-17T18:19:11.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5'), ('x-amz-request-id', 'tx000006c398716b8c34741-00652ed01f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:11.983 DEBUG:rgw_multi.tests:version3 id=Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5 2023-10-17T18:19:11.983 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.984 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:11.984 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5 2023-10-17T18:19:11.985 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5 2023-10-17T18:19:11.985 DEBUG:boto:Method: DELETE 2023-10-17T18:19:11.985 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5 2023-10-17T18:19:11.986 DEBUG:boto:Data: 2023-10-17T18:19:11.986 DEBUG:boto:Headers: {} 2023-10-17T18:19:11.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:11.987 DEBUG:boto:Port: 8000 2023-10-17T18:19:11.987 DEBUG:boto:Params: {} 2023-10-17T18:19:11.988 DEBUG:boto:Token: None 2023-10-17T18:19:11.988 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:11 GMT /hbmuog-11/obj-a1?versionId=Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5 2023-10-17T18:19:11.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qdmee5wpbSenigxoZWuj1Yo8RA0= 2023-10-17T18:19:11.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qdmee5wpbSenigxoZWuj1Yo8RA0=', 'Content-Length': '0'} 2023-10-17T18:19:12.000 DEBUG:boto:Response headers: [('x-amz-version-id', 'Kq7fy5bzcFiMCQq8XTh31AtWS91gBk5'), ('x-amz-request-id', 'tx00000e97d83cfb8bf0a1d-00652ed01f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.000 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:12.001 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:12.001 DEBUG:boto:Method: HEAD 2023-10-17T18:19:12.001 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:12.002 DEBUG:boto:Data: 2023-10-17T18:19:12.002 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.003 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.003 DEBUG:boto:Params: {} 2023-10-17T18:19:12.004 DEBUG:boto:Token: None 2023-10-17T18:19:12.004 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/ 2023-10-17T18:19:12.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IlXVToJ0DV0hHyzssVwtVav87PY= 2023-10-17T18:19:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IlXVToJ0DV0hHyzssVwtVav87PY=', 'Content-Length': '0'} 2023-10-17T18:19:12.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000887c6eb2209f6cfe-00652ed020-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.008 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.009 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.009 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.010 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:19:12.010 DEBUG:boto:Data: 2023-10-17T18:19:12.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.011 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.012 DEBUG:boto:Params: {} 2023-10-17T18:19:12.012 DEBUG:boto:Token: None 2023-10-17T18:19:12.013 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/obj-a2 2023-10-17T18:19:12.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lYjM7iNmMeWLlOSwdE/Eb6teLYw= 2023-10-17T18:19:12.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lYjM7iNmMeWLlOSwdE/Eb6teLYw='} 2023-10-17T18:19:12.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-amz-request-id', 'tx00000e8ecf671169ed0b8-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.027 DEBUG:rgw_multi.tests:version1 id=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:12.027 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.028 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.028 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.029 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:19:12.029 DEBUG:boto:Data: 2023-10-17T18:19:12.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.031 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.031 DEBUG:boto:Params: {} 2023-10-17T18:19:12.031 DEBUG:boto:Token: None 2023-10-17T18:19:12.032 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/obj-a2 2023-10-17T18:19:12.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uc1GwvyVmIoWaQBkwbdFH3i5yy4= 2023-10-17T18:19:12.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uc1GwvyVmIoWaQBkwbdFH3i5yy4='} 2023-10-17T18:19:12.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv'), ('x-amz-request-id', 'tx000008f6aa01d84c3bfad-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.075 DEBUG:rgw_multi.tests:version2 id=NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv 2023-10-17T18:19:12.076 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.076 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.076 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv 2023-10-17T18:19:12.077 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv 2023-10-17T18:19:12.077 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.078 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv 2023-10-17T18:19:12.078 DEBUG:boto:Data: 2023-10-17T18:19:12.078 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.079 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.080 DEBUG:boto:Params: {} 2023-10-17T18:19:12.080 DEBUG:boto:Token: None 2023-10-17T18:19:12.081 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/obj-a2?versionId=NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv 2023-10-17T18:19:12.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hWRIBjswMY83s8qQ3fr+/13CDF8= 2023-10-17T18:19:12.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hWRIBjswMY83s8qQ3fr+/13CDF8=', 'Content-Length': '0'} 2023-10-17T18:19:12.092 DEBUG:boto:Response headers: [('x-amz-version-id', 'NgBoYjpOX4XjPzoiEyezYzJsQqZm8Cv'), ('x-amz-request-id', 'tx00000db04869fa0363b65-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.092 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.093 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.093 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.094 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:19:12.094 DEBUG:boto:Data: 2023-10-17T18:19:12.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.095 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.095 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.096 DEBUG:boto:Params: {} 2023-10-17T18:19:12.096 DEBUG:boto:Token: None 2023-10-17T18:19:12.097 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/obj-a2 2023-10-17T18:19:12.097 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VIDQnuKTDSuRZAfJejtltdbnlZI= 2023-10-17T18:19:12.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VIDQnuKTDSuRZAfJejtltdbnlZI='} 2023-10-17T18:19:12.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y'), ('x-amz-request-id', 'tx0000045c1d22fd8965e60-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.107 DEBUG:rgw_multi.tests:version3 id=aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y 2023-10-17T18:19:12.108 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.108 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:12.109 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y 2023-10-17T18:19:12.109 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y 2023-10-17T18:19:12.109 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.110 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y 2023-10-17T18:19:12.110 DEBUG:boto:Data: 2023-10-17T18:19:12.111 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.111 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.112 DEBUG:boto:Params: {} 2023-10-17T18:19:12.112 DEBUG:boto:Token: None 2023-10-17T18:19:12.113 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-11/obj-a2?versionId=aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y 2023-10-17T18:19:12.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mAo1F+l+NBm8ih0/s4uclODrfP0= 2023-10-17T18:19:12.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mAo1F+l+NBm8ih0/s4uclODrfP0=', 'Content-Length': '0'} 2023-10-17T18:19:12.123 DEBUG:boto:Response headers: [('x-amz-version-id', 'aQkH7feLgdh0NB4Sx59PVGbCBw1WZ0y'), ('x-amz-request-id', 'tx000003d5ffef1dd364a34-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.124 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:12.124 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:12.125 DEBUG:boto:Method: HEAD 2023-10-17T18:19:12.125 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:12.125 DEBUG:boto:Data: 2023-10-17T18:19:12.126 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.127 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.127 DEBUG:boto:Params: {} 2023-10-17T18:19:12.128 DEBUG:boto:Token: None 2023-10-17T18:19:12.128 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/ 2023-10-17T18:19:12.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgkHMuoJMjsYWtwnWdgLfwWjmt4= 2023-10-17T18:19:12.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgkHMuoJMjsYWtwnWdgLfwWjmt4=', 'Content-Length': '0'} 2023-10-17T18:19:12.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1042e06aec65a52-00652ed020-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.132 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.132 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.133 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.133 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:19:12.133 DEBUG:boto:Data: 2023-10-17T18:19:12.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.135 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.135 DEBUG:boto:Params: {} 2023-10-17T18:19:12.135 DEBUG:boto:Token: None 2023-10-17T18:19:12.136 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a1 2023-10-17T18:19:12.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wdvcKuH77lt4nphO7Fczcg18400= 2023-10-17T18:19:12.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wdvcKuH77lt4nphO7Fczcg18400='} 2023-10-17T18:19:12.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-amz-request-id', 'tx000007365663dbea30688-00652ed020-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.147 DEBUG:rgw_multi.tests:version1 id=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:12.148 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.148 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.149 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.149 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:19:12.149 DEBUG:boto:Data: 2023-10-17T18:19:12.150 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.150 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.151 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.151 DEBUG:boto:Params: {} 2023-10-17T18:19:12.151 DEBUG:boto:Token: None 2023-10-17T18:19:12.152 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a1 2023-10-17T18:19:12.152 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LSg6kJQkEj8+da3M5ZNGE9td2zQ= 2023-10-17T18:19:12.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LSg6kJQkEj8+da3M5ZNGE9td2zQ='} 2023-10-17T18:19:12.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DUq58acqpqm36YkAl0wfQIeDXh2ex4m'), ('x-amz-request-id', 'tx0000006964235cb50cc5e-00652ed020-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.195 DEBUG:rgw_multi.tests:version2 id=DUq58acqpqm36YkAl0wfQIeDXh2ex4m 2023-10-17T18:19:12.195 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.196 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.196 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=DUq58acqpqm36YkAl0wfQIeDXh2ex4m 2023-10-17T18:19:12.197 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=DUq58acqpqm36YkAl0wfQIeDXh2ex4m 2023-10-17T18:19:12.197 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.197 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=DUq58acqpqm36YkAl0wfQIeDXh2ex4m 2023-10-17T18:19:12.198 DEBUG:boto:Data: 2023-10-17T18:19:12.198 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.199 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.199 DEBUG:boto:Params: {} 2023-10-17T18:19:12.200 DEBUG:boto:Token: None 2023-10-17T18:19:12.200 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a1?versionId=DUq58acqpqm36YkAl0wfQIeDXh2ex4m 2023-10-17T18:19:12.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EE2w3Hmy71a8G3N60zBVzqm0Myo= 2023-10-17T18:19:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EE2w3Hmy71a8G3N60zBVzqm0Myo=', 'Content-Length': '0'} 2023-10-17T18:19:12.211 DEBUG:boto:Response headers: [('x-amz-version-id', 'DUq58acqpqm36YkAl0wfQIeDXh2ex4m'), ('x-amz-request-id', 'tx00000bf4c38354cce7160-00652ed020-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.211 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.212 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.212 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.213 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:19:12.213 DEBUG:boto:Data: 2023-10-17T18:19:12.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.214 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.214 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.215 DEBUG:boto:Params: {} 2023-10-17T18:19:12.215 DEBUG:boto:Token: None 2023-10-17T18:19:12.216 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a1 2023-10-17T18:19:12.216 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G40BOcwtqCK96mEAYTxrXFnIviw= 2023-10-17T18:19:12.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G40BOcwtqCK96mEAYTxrXFnIviw='} 2023-10-17T18:19:12.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '28sroRFaJbMkEuV9DfAS7tAmcas0SYL'), ('x-amz-request-id', 'tx00000a2a72a058ed5aa1e-00652ed020-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.226 DEBUG:rgw_multi.tests:version3 id=28sroRFaJbMkEuV9DfAS7tAmcas0SYL 2023-10-17T18:19:12.226 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.227 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:12.227 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=28sroRFaJbMkEuV9DfAS7tAmcas0SYL 2023-10-17T18:19:12.228 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=28sroRFaJbMkEuV9DfAS7tAmcas0SYL 2023-10-17T18:19:12.228 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.228 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=28sroRFaJbMkEuV9DfAS7tAmcas0SYL 2023-10-17T18:19:12.229 DEBUG:boto:Data: 2023-10-17T18:19:12.229 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.230 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:12.230 DEBUG:boto:Port: 8000 2023-10-17T18:19:12.231 DEBUG:boto:Params: {} 2023-10-17T18:19:12.231 DEBUG:boto:Token: None 2023-10-17T18:19:12.232 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a1?versionId=28sroRFaJbMkEuV9DfAS7tAmcas0SYL 2023-10-17T18:19:12.232 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yrGz8WXIIprYnkaFl+UfM9NZiJ8= 2023-10-17T18:19:12.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yrGz8WXIIprYnkaFl+UfM9NZiJ8=', 'Content-Length': '0'} 2023-10-17T18:19:12.243 DEBUG:boto:Response headers: [('x-amz-version-id', '28sroRFaJbMkEuV9DfAS7tAmcas0SYL'), ('x-amz-request-id', 'tx00000b0b2055dac3727f5-00652ed020-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.243 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:12.244 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:12.244 DEBUG:boto:Method: HEAD 2023-10-17T18:19:12.245 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:12.245 DEBUG:boto:Data: 2023-10-17T18:19:12.245 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.246 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.247 DEBUG:boto:Params: {} 2023-10-17T18:19:12.247 DEBUG:boto:Token: None 2023-10-17T18:19:12.248 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/ 2023-10-17T18:19:12.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgkHMuoJMjsYWtwnWdgLfwWjmt4= 2023-10-17T18:19:12.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgkHMuoJMjsYWtwnWdgLfwWjmt4=', 'Content-Length': '0'} 2023-10-17T18:19:12.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4c852a64435de41-00652ed020-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.251 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.252 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.252 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.253 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:19:12.253 DEBUG:boto:Data: 2023-10-17T18:19:12.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.254 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.255 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.255 DEBUG:boto:Params: {} 2023-10-17T18:19:12.256 DEBUG:boto:Token: None 2023-10-17T18:19:12.256 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a2 2023-10-17T18:19:12.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I5quQrAjmsS9OdanzfvOHb32ijE= 2023-10-17T18:19:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I5quQrAjmsS9OdanzfvOHb32ijE='} 2023-10-17T18:19:12.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-amz-request-id', 'tx000007336867034902c71-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.267 DEBUG:rgw_multi.tests:version1 id=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:12.268 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.268 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.268 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.269 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:19:12.269 DEBUG:boto:Data: 2023-10-17T18:19:12.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.271 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.271 DEBUG:boto:Params: {} 2023-10-17T18:19:12.272 DEBUG:boto:Token: None 2023-10-17T18:19:12.272 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a2 2023-10-17T18:19:12.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EZV5SX6KWUPK4RHnAUE7Udk+Vl0= 2023-10-17T18:19:12.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EZV5SX6KWUPK4RHnAUE7Udk+Vl0='} 2023-10-17T18:19:12.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UiHxKyUCs34SlO-9X58l8.fABwn7w4V'), ('x-amz-request-id', 'tx00000f098446a770a0f35-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.282 DEBUG:rgw_multi.tests:version2 id=UiHxKyUCs34SlO-9X58l8.fABwn7w4V 2023-10-17T18:19:12.282 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.282 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.283 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=UiHxKyUCs34SlO-9X58l8.fABwn7w4V 2023-10-17T18:19:12.283 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=UiHxKyUCs34SlO-9X58l8.fABwn7w4V 2023-10-17T18:19:12.284 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.284 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=UiHxKyUCs34SlO-9X58l8.fABwn7w4V 2023-10-17T18:19:12.285 DEBUG:boto:Data: 2023-10-17T18:19:12.285 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.285 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.286 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.286 DEBUG:boto:Params: {} 2023-10-17T18:19:12.287 DEBUG:boto:Token: None 2023-10-17T18:19:12.287 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a2?versionId=UiHxKyUCs34SlO-9X58l8.fABwn7w4V 2023-10-17T18:19:12.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e2cYmPXLxdMzq8tDD3UMtGFLQyY= 2023-10-17T18:19:12.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e2cYmPXLxdMzq8tDD3UMtGFLQyY=', 'Content-Length': '0'} 2023-10-17T18:19:12.297 DEBUG:boto:Response headers: [('x-amz-version-id', 'UiHxKyUCs34SlO-9X58l8.fABwn7w4V'), ('x-amz-request-id', 'tx000006cdd9da0dfbaab73-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.297 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.298 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.298 DEBUG:boto:Method: PUT 2023-10-17T18:19:12.299 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:19:12.299 DEBUG:boto:Data: 2023-10-17T18:19:12.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:19:12.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.300 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.301 DEBUG:boto:Params: {} 2023-10-17T18:19:12.301 DEBUG:boto:Token: None 2023-10-17T18:19:12.302 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a2 2023-10-17T18:19:12.302 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hJ8LrQetPFL9cdJGGsda41h0enI= 2023-10-17T18:19:12.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hJ8LrQetPFL9cdJGGsda41h0enI='} 2023-10-17T18:19:12.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eba.z86hKQ.prI4Xvvac5Unus5CxGHN'), ('x-amz-request-id', 'tx00000afe8639644258ea1-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.347 DEBUG:rgw_multi.tests:version3 id=eba.z86hKQ.prI4Xvvac5Unus5CxGHN 2023-10-17T18:19:12.348 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.348 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:12.349 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=eba.z86hKQ.prI4Xvvac5Unus5CxGHN 2023-10-17T18:19:12.349 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=eba.z86hKQ.prI4Xvvac5Unus5CxGHN 2023-10-17T18:19:12.349 DEBUG:boto:Method: DELETE 2023-10-17T18:19:12.350 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=eba.z86hKQ.prI4Xvvac5Unus5CxGHN 2023-10-17T18:19:12.350 DEBUG:boto:Data: 2023-10-17T18:19:12.351 DEBUG:boto:Headers: {} 2023-10-17T18:19:12.351 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:12.351 DEBUG:boto:Port: 8001 2023-10-17T18:19:12.352 DEBUG:boto:Params: {} 2023-10-17T18:19:12.352 DEBUG:boto:Token: None 2023-10-17T18:19:12.353 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:19:12 GMT /hbmuog-12/obj-a2?versionId=eba.z86hKQ.prI4Xvvac5Unus5CxGHN 2023-10-17T18:19:12.353 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EjF3ZPv22dv6k7qOiGSJOWm+ubM= 2023-10-17T18:19:12.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EjF3ZPv22dv6k7qOiGSJOWm+ubM=', 'Content-Length': '0'} 2023-10-17T18:19:12.363 DEBUG:boto:Response headers: [('x-amz-version-id', 'eba.z86hKQ.prI4Xvvac5Unus5CxGHN'), ('x-amz-request-id', 'tx000004289fdc858c1fea8-00652ed020-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:12.363 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-11 2023-10-17T18:19:12.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:12.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:12.365 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:19:12.423 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:19:12.424 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:19:12.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:12.457+0000 7f3f0d4fd780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:12.459 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:19:12.459 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:17.459+0000 7f3f0d4fd780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:17.460 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:19:17.461 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:22.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:22.460+0000 7f3f0d4fd780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:22.462 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:19:22.462 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:27.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:27.462+0000 7f3f0d4fd780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:27.463 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.4.11, , , , , ] 2023-10-17T18:19:27.463 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:32.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:32.464+0000 7f3f0d4fd780 1 bucket sync caught up with source: 2023-10-17T18:19:32.464 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:32.464 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , , 00000000010.20.6, ] 2023-10-17T18:19:32.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:32.464+0000 7f3f0d4fd780 0 bucket checkpoint complete 2023-10-17T18:19:32.480 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-11 2023-10-17T18:19:32.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:32.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:32.482 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:19:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:19:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:19:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:32.585+0000 7fce1aa77780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:32.585 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000010.40.6, , ] 2023-10-17T18:19:32.586 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:37.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:37.586+0000 7fce1aa77780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:37.587 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000010.40.6, , ] 2023-10-17T18:19:37.587 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:42.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:42.586+0000 7fce1aa77780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:42.588 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000010.40.6, , ] 2023-10-17T18:19:42.588 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:47.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:47.588+0000 7fce1aa77780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:47.589 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000010.40.6, , ] 2023-10-17T18:19:47.589 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:52.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:52.589+0000 7fce1aa77780 1 bucket sync caught up with source: 2023-10-17T18:19:52.590 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:52.591 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000010.40.6, 00000000002.16.11, ] 2023-10-17T18:19:52.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:52.589+0000 7fce1aa77780 0 bucket checkpoint complete 2023-10-17T18:19:52.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-11 zones={a2, a1} 2023-10-17T18:19:52.607 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.607 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.608 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.608 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:52.608 DEBUG:boto:Data: 2023-10-17T18:19:52.609 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.610 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.610 DEBUG:boto:Params: {} 2023-10-17T18:19:52.611 DEBUG:boto:Token: None 2023-10-17T18:19:52.611 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5RS+ocAWgCHGXWNa6glmkPLF5SU= 2023-10-17T18:19:52.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5RS+ocAWgCHGXWNa6glmkPLF5SU=', 'Content-Length': '0'} 2023-10-17T18:19:52.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f943b7dbb4ac8b69-00652ed048-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.616 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.616 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.617 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.617 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:52.617 DEBUG:boto:Data: 2023-10-17T18:19:52.618 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.619 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.619 DEBUG:boto:Params: {} 2023-10-17T18:19:52.620 DEBUG:boto:Token: None 2023-10-17T18:19:52.620 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5RS+ocAWgCHGXWNa6glmkPLF5SU= 2023-10-17T18:19:52.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5RS+ocAWgCHGXWNa6glmkPLF5SU=', 'Content-Length': '0'} 2023-10-17T18:19:52.624 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf3f5cc634cd363d-00652ed048-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.624 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:19:52.625 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.625 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.626 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.626 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.627 DEBUG:boto:Method: GET 2023-10-17T18:19:52.627 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.628 DEBUG:boto:Data: 2023-10-17T18:19:52.628 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.628 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.629 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.629 DEBUG:boto:Params: {} 2023-10-17T18:19:52.630 DEBUG:boto:Token: None 2023-10-17T18:19:52.630 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/?versions 2023-10-17T18:19:52.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco= 2023-10-17T18:19:52.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco=', 'Content-Length': '0'} 2023-10-17T18:19:52.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e84431920bcf82-00652ed048-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.635 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.636 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:19:52.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:19:52.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:19:52.637 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:19:52.637 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.638 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.638 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.639 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.639 DEBUG:boto:Method: GET 2023-10-17T18:19:52.639 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.640 DEBUG:boto:Data: 2023-10-17T18:19:52.640 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.641 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.642 DEBUG:boto:Params: {} 2023-10-17T18:19:52.642 DEBUG:boto:Token: None 2023-10-17T18:19:52.643 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/?versions 2023-10-17T18:19:52.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco= 2023-10-17T18:19:52.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco=', 'Content-Length': '0'} 2023-10-17T18:19:52.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ec671ea31d75cf-00652ed048-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.647 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.648 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:19:52.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:19:52.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:19:52.649 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.649 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.650 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.650 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.651 DEBUG:boto:Method: GET 2023-10-17T18:19:52.651 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.652 DEBUG:boto:Data: 2023-10-17T18:19:52.652 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.652 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.653 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.653 DEBUG:boto:Params: {} 2023-10-17T18:19:52.654 DEBUG:boto:Token: None 2023-10-17T18:19:52.654 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/?versions 2023-10-17T18:19:52.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco= 2023-10-17T18:19:52.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco=', 'Content-Length': '0'} 2023-10-17T18:19:52.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8fed9c2e31d8ce8-00652ed048-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.658 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.659 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.660 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.660 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.660 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.661 DEBUG:boto:Method: GET 2023-10-17T18:19:52.661 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:19:52.662 DEBUG:boto:Data: 2023-10-17T18:19:52.662 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.663 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.663 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.663 DEBUG:boto:Params: {} 2023-10-17T18:19:52.664 DEBUG:boto:Token: None 2023-10-17T18:19:52.664 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/?versions 2023-10-17T18:19:52.665 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco= 2023-10-17T18:19:52.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J5Kts2/wWzNqdlhFKwoCdUF5Pco=', 'Content-Length': '0'} 2023-10-17T18:19:52.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4de05b9c55631c-00652ed048-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.668 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:52.669 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.670 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.670 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.671 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.671 DEBUG:boto:Method: GET 2023-10-17T18:19:52.671 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.672 DEBUG:boto:Data: 2023-10-17T18:19:52.672 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.673 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.674 DEBUG:boto:Params: {} 2023-10-17T18:19:52.674 DEBUG:boto:Token: None 2023-10-17T18:19:52.675 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044edede3ad837b8f-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.678 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.679 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.679 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.679 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.680 DEBUG:boto:Method: GET 2023-10-17T18:19:52.680 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.681 DEBUG:boto:Data: 2023-10-17T18:19:52.681 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.682 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.682 DEBUG:boto:Params: {} 2023-10-17T18:19:52.683 DEBUG:boto:Token: None 2023-10-17T18:19:52.683 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048efd1e3cdbb67ab-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.687 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.687 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.688 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.688 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.688 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.689 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.689 DEBUG:boto:Data: 2023-10-17T18:19:52.690 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.690 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.691 DEBUG:boto:Params: {} 2023-10-17T18:19:52.691 DEBUG:boto:Token: None 2023-10-17T18:19:52.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Em4SOHYMtih3M1Cpkrj0v4d61VY= 2023-10-17T18:19:52.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Em4SOHYMtih3M1Cpkrj0v4d61VY=', 'Content-Length': '0'} 2023-10-17T18:19:52.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000342b250a286d8bb2-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.695 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.696 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.696 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.696 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.697 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.697 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.698 DEBUG:boto:Data: 2023-10-17T18:19:52.698 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.699 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.700 DEBUG:boto:Params: {} 2023-10-17T18:19:52.700 DEBUG:boto:Token: None 2023-10-17T18:19:52.701 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Em4SOHYMtih3M1Cpkrj0v4d61VY= 2023-10-17T18:19:52.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Em4SOHYMtih3M1Cpkrj0v4d61VY=', 'Content-Length': '0'} 2023-10-17T18:19:52.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006eba32f949ea2271-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:52.704 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.705 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.705 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.706 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.706 DEBUG:boto:Method: GET 2023-10-17T18:19:52.706 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.707 DEBUG:boto:Data: 2023-10-17T18:19:52.707 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.708 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.708 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.709 DEBUG:boto:Params: {} 2023-10-17T18:19:52.709 DEBUG:boto:Token: None 2023-10-17T18:19:52.710 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c50268f72e6c16f7-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.713 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.714 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.714 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.714 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.715 DEBUG:boto:Method: GET 2023-10-17T18:19:52.715 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.716 DEBUG:boto:Data: 2023-10-17T18:19:52.716 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.717 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.718 DEBUG:boto:Params: {} 2023-10-17T18:19:52.718 DEBUG:boto:Token: None 2023-10-17T18:19:52.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000acf7b40cc0beee1-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.722 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.723 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.723 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.723 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.724 DEBUG:boto:Method: GET 2023-10-17T18:19:52.724 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.725 DEBUG:boto:Data: 2023-10-17T18:19:52.725 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.725 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.726 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.726 DEBUG:boto:Params: {} 2023-10-17T18:19:52.727 DEBUG:boto:Token: None 2023-10-17T18:19:52.727 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029c841de0d985253-00652ed048-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.778 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.779 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.780 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.780 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.781 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.781 DEBUG:boto:Method: GET 2023-10-17T18:19:52.782 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.782 DEBUG:boto:Data: 2023-10-17T18:19:52.782 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.783 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.784 DEBUG:boto:Params: {} 2023-10-17T18:19:52.784 DEBUG:boto:Token: None 2023-10-17T18:19:52.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e3feec1d5e874c-00652ed048-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.789 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.789 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.790 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.790 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.791 DEBUG:boto:Path: /hbmuog-11/dummy 2023-10-17T18:19:52.791 DEBUG:boto:Data: 2023-10-17T18:19:52.791 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.792 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.793 DEBUG:boto:Params: {} 2023-10-17T18:19:52.793 DEBUG:boto:Token: None 2023-10-17T18:19:52.793 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy 2023-10-17T18:19:52.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:daalmnluFWLtq52go+vNV/0oCzg= 2023-10-17T18:19:52.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:daalmnluFWLtq52go+vNV/0oCzg=', 'Content-Length': '0'} 2023-10-17T18:19:52.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000016331af29192ca8-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.797 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.798 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.798 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.799 DEBUG:boto:Path: /hbmuog-11/dummy 2023-10-17T18:19:52.799 DEBUG:boto:Data: 2023-10-17T18:19:52.800 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.800 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.801 DEBUG:boto:Params: {} 2023-10-17T18:19:52.801 DEBUG:boto:Token: None 2023-10-17T18:19:52.802 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy 2023-10-17T18:19:52.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:daalmnluFWLtq52go+vNV/0oCzg= 2023-10-17T18:19:52.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:daalmnluFWLtq52go+vNV/0oCzg=', 'Content-Length': '0'} 2023-10-17T18:19:52.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f082523b35c3bee3-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:52.806 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.806 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.807 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.807 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.808 DEBUG:boto:Method: GET 2023-10-17T18:19:52.808 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.808 DEBUG:boto:Data: 2023-10-17T18:19:52.809 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.810 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.810 DEBUG:boto:Params: {} 2023-10-17T18:19:52.811 DEBUG:boto:Token: None 2023-10-17T18:19:52.811 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061e0ed34e4681b50-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.815 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:19:52.815 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:19:52.816 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.816 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.816 DEBUG:boto:Method: GET 2023-10-17T18:19:52.817 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.817 DEBUG:boto:Data: 2023-10-17T18:19:52.818 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.819 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.819 DEBUG:boto:Params: {} 2023-10-17T18:19:52.819 DEBUG:boto:Token: None 2023-10-17T18:19:52.820 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:19:52.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI= 2023-10-17T18:19:52.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KTl1bK/ge/dDLmihKINk/5ohxyI=', 'Content-Length': '0'} 2023-10-17T18:19:52.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a297be5c2648ce8f-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.823 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.824 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.824 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.825 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.825 DEBUG:boto:Method: GET 2023-10-17T18:19:52.826 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.826 DEBUG:boto:Data: 2023-10-17T18:19:52.827 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.827 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.828 DEBUG:boto:Params: {} 2023-10-17T18:19:52.828 DEBUG:boto:Token: None 2023-10-17T18:19:52.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.829 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000456c464c7946c56b-00652ed048-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.833 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.834 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.834 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.834 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.835 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.835 DEBUG:boto:Method: GET 2023-10-17T18:19:52.836 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:19:52.836 DEBUG:boto:Data: 2023-10-17T18:19:52.837 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.837 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.838 DEBUG:boto:Params: {} 2023-10-17T18:19:52.838 DEBUG:boto:Token: None 2023-10-17T18:19:52.839 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d91b9c17e2e0600-00652ed048-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.843 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:52.844 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.844 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.845 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.845 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.845 DEBUG:boto:Method: GET 2023-10-17T18:19:52.846 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.846 DEBUG:boto:Data: 2023-10-17T18:19:52.847 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.847 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.848 DEBUG:boto:Params: {} 2023-10-17T18:19:52.848 DEBUG:boto:Token: None 2023-10-17T18:19:52.849 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE= 2023-10-17T18:19:52.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE=', 'Content-Length': '0'} 2023-10-17T18:19:52.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000037b1ee40265df20-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.853 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.853 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.853 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.854 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.854 DEBUG:boto:Method: GET 2023-10-17T18:19:52.855 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.855 DEBUG:boto:Data: 2023-10-17T18:19:52.856 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.856 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.857 DEBUG:boto:Params: {} 2023-10-17T18:19:52.857 DEBUG:boto:Token: None 2023-10-17T18:19:52.858 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE= 2023-10-17T18:19:52.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE=', 'Content-Length': '0'} 2023-10-17T18:19:52.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e3135eb68d27fc9-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.862 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.862 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.863 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.863 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.863 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.864 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.864 DEBUG:boto:Data: 2023-10-17T18:19:52.865 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.865 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.866 DEBUG:boto:Params: {} 2023-10-17T18:19:52.866 DEBUG:boto:Token: None 2023-10-17T18:19:52.867 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ObRJaeRVer4JDN6hoAi7oz/jugQ= 2023-10-17T18:19:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ObRJaeRVer4JDN6hoAi7oz/jugQ=', 'Content-Length': '0'} 2023-10-17T18:19:52.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d164e201a6d38034-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.870 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.871 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.871 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.872 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.872 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.872 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.873 DEBUG:boto:Data: 2023-10-17T18:19:52.873 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.874 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.874 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.874 DEBUG:boto:Params: {} 2023-10-17T18:19:52.875 DEBUG:boto:Token: None 2023-10-17T18:19:52.875 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.876 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ObRJaeRVer4JDN6hoAi7oz/jugQ= 2023-10-17T18:19:52.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ObRJaeRVer4JDN6hoAi7oz/jugQ=', 'Content-Length': '0'} 2023-10-17T18:19:52.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3a3ccff9f43bbfd-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:52.879 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.880 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.880 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.881 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.881 DEBUG:boto:Method: GET 2023-10-17T18:19:52.881 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.882 DEBUG:boto:Data: 2023-10-17T18:19:52.882 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.883 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.883 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.884 DEBUG:boto:Params: {} 2023-10-17T18:19:52.884 DEBUG:boto:Token: None 2023-10-17T18:19:52.885 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.885 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE= 2023-10-17T18:19:52.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE=', 'Content-Length': '0'} 2023-10-17T18:19:52.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004985a53f24325338-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.888 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.888 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.889 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.889 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.890 DEBUG:boto:Method: GET 2023-10-17T18:19:52.890 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.891 DEBUG:boto:Data: 2023-10-17T18:19:52.891 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.892 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.892 DEBUG:boto:Params: {} 2023-10-17T18:19:52.893 DEBUG:boto:Token: None 2023-10-17T18:19:52.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.894 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE= 2023-10-17T18:19:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE=', 'Content-Length': '0'} 2023-10-17T18:19:52.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a997f7ddc9e0a02-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.938 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.939 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.939 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.940 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.940 DEBUG:boto:Method: GET 2023-10-17T18:19:52.941 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.941 DEBUG:boto:Data: 2023-10-17T18:19:52.942 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.942 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.943 DEBUG:boto:Params: {} 2023-10-17T18:19:52.943 DEBUG:boto:Token: None 2023-10-17T18:19:52.944 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc704a612e88d749-00652ed048-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.948 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.949 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:52.949 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:52.950 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.950 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.950 DEBUG:boto:Method: GET 2023-10-17T18:19:52.951 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:52.951 DEBUG:boto:Data: 2023-10-17T18:19:52.952 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.952 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.953 DEBUG:boto:Params: {} 2023-10-17T18:19:52.953 DEBUG:boto:Token: None 2023-10-17T18:19:52.954 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/ 2023-10-17T18:19:52.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0= 2023-10-17T18:19:52.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DYQUs7WwzX3Ja0Dt6SnCeRdZLj0=', 'Content-Length': '0'} 2023-10-17T18:19:52.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a548f0801a4259f-00652ed048-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.958 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:52.958 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.959 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.959 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.960 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:19:52.960 DEBUG:boto:Data: 2023-10-17T18:19:52.961 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.961 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.961 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.962 DEBUG:boto:Params: {} 2023-10-17T18:19:52.962 DEBUG:boto:Token: None 2023-10-17T18:19:52.963 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1 2023-10-17T18:19:52.963 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D/xZf6soj/GvmDrMFj3YpYFK5yo= 2023-10-17T18:19:52.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D/xZf6soj/GvmDrMFj3YpYFK5yo=', 'Content-Length': '0'} 2023-10-17T18:19:52.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c12112b3c0452a5-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.967 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.967 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.968 DEBUG:boto:Method: HEAD 2023-10-17T18:19:52.968 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:19:52.969 DEBUG:boto:Data: 2023-10-17T18:19:52.969 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:52.970 DEBUG:boto:Port: 8000 2023-10-17T18:19:52.970 DEBUG:boto:Params: {} 2023-10-17T18:19:52.971 DEBUG:boto:Token: None 2023-10-17T18:19:52.971 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1 2023-10-17T18:19:52.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D/xZf6soj/GvmDrMFj3YpYFK5yo= 2023-10-17T18:19:52.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D/xZf6soj/GvmDrMFj3YpYFK5yo=', 'Content-Length': '0'} 2023-10-17T18:19:52.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fa2ac71a345fa1e-00652ed048-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:52.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:52.975 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.976 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:52.976 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.977 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.977 DEBUG:boto:Method: GET 2023-10-17T18:19:52.978 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.978 DEBUG:boto:Data: 2023-10-17T18:19:52.978 DEBUG:boto:Headers: {} 2023-10-17T18:19:52.979 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:52.979 DEBUG:boto:Port: 8001 2023-10-17T18:19:52.980 DEBUG:boto:Params: {} 2023-10-17T18:19:52.980 DEBUG:boto:Token: None 2023-10-17T18:19:52.981 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:52 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:52.981 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE= 2023-10-17T18:19:52.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0Psug1mFtaYGh77qQAtLfDWINE=', 'Content-Length': '0'} 2023-10-17T18:19:52.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7205bf11e6a4367-00652ed048-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.030 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:53.031 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:53.031 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:53.032 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:53.032 DEBUG:boto:Method: GET 2023-10-17T18:19:53.033 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:53.033 DEBUG:boto:Data: 2023-10-17T18:19:53.034 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.035 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.035 DEBUG:boto:Params: {} 2023-10-17T18:19:53.036 DEBUG:boto:Token: None 2023-10-17T18:19:53.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:19:53.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m410t5xSrO7mJoWAiTD8Q1268Zk= 2023-10-17T18:19:53.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m410t5xSrO7mJoWAiTD8Q1268Zk=', 'Content-Length': '0'} 2023-10-17T18:19:53.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bcecb4adbcf8972-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.040 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.040 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.041 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.041 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.042 DEBUG:boto:Method: GET 2023-10-17T18:19:53.042 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.042 DEBUG:boto:Data: 2023-10-17T18:19:53.043 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.043 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.044 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.044 DEBUG:boto:Params: {} 2023-10-17T18:19:53.045 DEBUG:boto:Token: None 2023-10-17T18:19:53.045 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.046 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c79468d4228e267c-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.049 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.050 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.050 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.051 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.051 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.052 DEBUG:boto:Method: GET 2023-10-17T18:19:53.052 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:19:53.052 DEBUG:boto:Data: 2023-10-17T18:19:53.053 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.054 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.054 DEBUG:boto:Params: {} 2023-10-17T18:19:53.055 DEBUG:boto:Token: None 2023-10-17T18:19:53.055 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1845b534b01466-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.059 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:53.060 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.061 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.061 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.061 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.062 DEBUG:boto:Method: GET 2023-10-17T18:19:53.062 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.063 DEBUG:boto:Data: 2023-10-17T18:19:53.063 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.064 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.064 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.064 DEBUG:boto:Params: {} 2023-10-17T18:19:53.065 DEBUG:boto:Token: None 2023-10-17T18:19:53.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000130d1499b352d60f-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.069 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.069 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.070 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.070 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.071 DEBUG:boto:Method: GET 2023-10-17T18:19:53.071 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.071 DEBUG:boto:Data: 2023-10-17T18:19:53.072 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.072 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.073 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.073 DEBUG:boto:Params: {} 2023-10-17T18:19:53.074 DEBUG:boto:Token: None 2023-10-17T18:19:53.074 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.075 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3dd5f82a5e0187e-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.118 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.119 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.119 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.119 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.120 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.120 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.121 DEBUG:boto:Data: 2023-10-17T18:19:53.121 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.121 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.122 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.122 DEBUG:boto:Params: {} 2023-10-17T18:19:53.123 DEBUG:boto:Token: None 2023-10-17T18:19:53.123 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.124 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hchUGDwK3wtgEgEE+YaqNyag0wc= 2023-10-17T18:19:53.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hchUGDwK3wtgEgEE+YaqNyag0wc=', 'Content-Length': '0'} 2023-10-17T18:19:53.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077359cac64ad51e2-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.127 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.127 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.128 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.128 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.128 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.129 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.129 DEBUG:boto:Data: 2023-10-17T18:19:53.130 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.130 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.131 DEBUG:boto:Params: {} 2023-10-17T18:19:53.131 DEBUG:boto:Token: None 2023-10-17T18:19:53.132 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.132 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hchUGDwK3wtgEgEE+YaqNyag0wc= 2023-10-17T18:19:53.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hchUGDwK3wtgEgEE+YaqNyag0wc=', 'Content-Length': '0'} 2023-10-17T18:19:53.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c6204c23cc4f46d-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:53.136 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.136 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.137 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.137 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.137 DEBUG:boto:Method: GET 2023-10-17T18:19:53.138 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.138 DEBUG:boto:Data: 2023-10-17T18:19:53.139 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.139 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.140 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.140 DEBUG:boto:Params: {} 2023-10-17T18:19:53.140 DEBUG:boto:Token: None 2023-10-17T18:19:53.141 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.141 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bb188869a2f4c8d-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.144 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.145 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.145 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.146 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.146 DEBUG:boto:Method: GET 2023-10-17T18:19:53.147 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.147 DEBUG:boto:Data: 2023-10-17T18:19:53.147 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.148 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.148 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.149 DEBUG:boto:Params: {} 2023-10-17T18:19:53.149 DEBUG:boto:Token: None 2023-10-17T18:19:53.150 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.150 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1ead93d9b1007f6-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.153 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.153 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.154 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.154 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.155 DEBUG:boto:Method: GET 2023-10-17T18:19:53.155 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.156 DEBUG:boto:Data: 2023-10-17T18:19:53.156 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.157 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.158 DEBUG:boto:Params: {} 2023-10-17T18:19:53.158 DEBUG:boto:Token: None 2023-10-17T18:19:53.159 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.159 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd40f0c590124ed-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.163 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.164 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.164 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.165 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.165 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.165 DEBUG:boto:Method: GET 2023-10-17T18:19:53.166 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.166 DEBUG:boto:Data: 2023-10-17T18:19:53.167 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.167 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.168 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.168 DEBUG:boto:Params: {} 2023-10-17T18:19:53.168 DEBUG:boto:Token: None 2023-10-17T18:19:53.169 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.169 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084cf576cfdae6047-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.173 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.173 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.174 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.174 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.175 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:19:53.175 DEBUG:boto:Data: 2023-10-17T18:19:53.175 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.176 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.176 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.177 DEBUG:boto:Params: {} 2023-10-17T18:19:53.177 DEBUG:boto:Token: None 2023-10-17T18:19:53.178 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2 2023-10-17T18:19:53.178 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6cCDtfJZ+uL4NmXmqzOJB1N3JTw= 2023-10-17T18:19:53.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6cCDtfJZ+uL4NmXmqzOJB1N3JTw=', 'Content-Length': '0'} 2023-10-17T18:19:53.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c3fca092ed34fd0-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.181 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.182 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.182 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.183 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:19:53.183 DEBUG:boto:Data: 2023-10-17T18:19:53.183 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.184 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.184 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.185 DEBUG:boto:Params: {} 2023-10-17T18:19:53.185 DEBUG:boto:Token: None 2023-10-17T18:19:53.186 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2 2023-10-17T18:19:53.186 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6cCDtfJZ+uL4NmXmqzOJB1N3JTw= 2023-10-17T18:19:53.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6cCDtfJZ+uL4NmXmqzOJB1N3JTw=', 'Content-Length': '0'} 2023-10-17T18:19:53.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005341dcf5451cdb76-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:53.190 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.190 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.191 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.191 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.192 DEBUG:boto:Method: GET 2023-10-17T18:19:53.192 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.192 DEBUG:boto:Data: 2023-10-17T18:19:53.193 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.193 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.194 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.194 DEBUG:boto:Params: {} 2023-10-17T18:19:53.195 DEBUG:boto:Token: None 2023-10-17T18:19:53.195 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea798bb279a27bbe-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.238 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.239 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:53.239 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.240 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.240 DEBUG:boto:Method: GET 2023-10-17T18:19:53.240 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.241 DEBUG:boto:Data: 2023-10-17T18:19:53.241 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.242 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.243 DEBUG:boto:Params: {} 2023-10-17T18:19:53.243 DEBUG:boto:Token: None 2023-10-17T18:19:53.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:19:53.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY= 2023-10-17T18:19:53.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HcqGRecSzyzHCH7Sw9n4K2R4EXY=', 'Content-Length': '0'} 2023-10-17T18:19:53.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000838a7989e23ad0b6-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.247 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.247 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.248 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.248 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.249 DEBUG:boto:Method: GET 2023-10-17T18:19:53.249 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.250 DEBUG:boto:Data: 2023-10-17T18:19:53.250 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.250 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.251 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.251 DEBUG:boto:Params: {} 2023-10-17T18:19:53.252 DEBUG:boto:Token: None 2023-10-17T18:19:53.252 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000192cf533487896dc-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.256 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.257 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:53.257 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:53.257 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.258 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.258 DEBUG:boto:Method: GET 2023-10-17T18:19:53.259 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:19:53.259 DEBUG:boto:Data: 2023-10-17T18:19:53.260 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.260 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.261 DEBUG:boto:Params: {} 2023-10-17T18:19:53.261 DEBUG:boto:Token: None 2023-10-17T18:19:53.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-11/ 2023-10-17T18:19:53.262 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0= 2023-10-17T18:19:53.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BCYvCnyIcDSRR2PO3RkjpjuqI0=', 'Content-Length': '0'} 2023-10-17T18:19:53.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bdf4fc6b461fde4-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.310 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-11 zones={a2, a1} 2023-10-17T18:19:53.311 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-12 2023-10-17T18:19:53.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:53.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:53.313 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:19:53.379 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:19:53.379 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:19:53.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:53.416+0000 7f89ed254780 1 bucket sync caught up with source: 2023-10-17T18:19:53.417 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.8.11, , , 00000000002.3.11, 00000000010.28.6, ] 2023-10-17T18:19:53.418 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.8.11, , , 00000000002.3.11, 00000000010.28.6, ] 2023-10-17T18:19:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:53.416+0000 7f89ed254780 0 bucket checkpoint complete 2023-10-17T18:19:53.430 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-12 2023-10-17T18:19:53.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:53.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:53.432 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:19:53.490 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:19:53.491 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:19:53.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:53.523+0000 7f97f7b82780 1 bucket sync caught up with source: 2023-10-17T18:19:53.525 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.42.10, , , 00000000010.31.6, 00000000002.45.11, ] 2023-10-17T18:19:53.525 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.42.10, , , 00000000010.31.6, 00000000002.45.11, ] 2023-10-17T18:19:53.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:53.523+0000 7f97f7b82780 0 bucket checkpoint complete 2023-10-17T18:19:53.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-12 zones={a2, a1} 2023-10-17T18:19:53.537 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.537 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.538 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.538 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:53.538 DEBUG:boto:Data: 2023-10-17T18:19:53.539 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.540 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.540 DEBUG:boto:Params: {} 2023-10-17T18:19:53.541 DEBUG:boto:Token: None 2023-10-17T18:19:53.541 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.542 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PIxGJEmAcj5uDoT6AFESxtiervg= 2023-10-17T18:19:53.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PIxGJEmAcj5uDoT6AFESxtiervg=', 'Content-Length': '0'} 2023-10-17T18:19:53.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd960ab614de5281-00652ed049-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.545 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.545 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.546 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.546 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:53.546 DEBUG:boto:Data: 2023-10-17T18:19:53.547 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.547 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.548 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.548 DEBUG:boto:Params: {} 2023-10-17T18:19:53.548 DEBUG:boto:Token: None 2023-10-17T18:19:53.549 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.549 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PIxGJEmAcj5uDoT6AFESxtiervg= 2023-10-17T18:19:53.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PIxGJEmAcj5uDoT6AFESxtiervg=', 'Content-Length': '0'} 2023-10-17T18:19:53.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d8942216685fac8-00652ed049-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:19:53.553 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.553 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.555 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.555 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.555 DEBUG:boto:Method: GET 2023-10-17T18:19:53.556 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.556 DEBUG:boto:Data: 2023-10-17T18:19:53.557 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.557 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.558 DEBUG:boto:Params: {} 2023-10-17T18:19:53.558 DEBUG:boto:Token: None 2023-10-17T18:19:53.559 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/?versions 2023-10-17T18:19:53.559 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8= 2023-10-17T18:19:53.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8=', 'Content-Length': '0'} 2023-10-17T18:19:53.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000630c602876ca330c-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.563 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:19:53.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:19:53.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:19:53.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:19:53.565 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.565 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.566 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.566 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.567 DEBUG:boto:Method: GET 2023-10-17T18:19:53.567 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.568 DEBUG:boto:Data: 2023-10-17T18:19:53.568 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.569 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.569 DEBUG:boto:Params: {} 2023-10-17T18:19:53.570 DEBUG:boto:Token: None 2023-10-17T18:19:53.570 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/?versions 2023-10-17T18:19:53.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8= 2023-10-17T18:19:53.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8=', 'Content-Length': '0'} 2023-10-17T18:19:53.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006683f4202812a2a5-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.574 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.575 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:19:53.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:19:53.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:19:53.576 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.576 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.577 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.577 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.578 DEBUG:boto:Method: GET 2023-10-17T18:19:53.578 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.578 DEBUG:boto:Data: 2023-10-17T18:19:53.579 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.581 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.581 DEBUG:boto:Params: {} 2023-10-17T18:19:53.582 DEBUG:boto:Token: None 2023-10-17T18:19:53.582 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/?versions 2023-10-17T18:19:53.583 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8= 2023-10-17T18:19:53.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8=', 'Content-Length': '0'} 2023-10-17T18:19:53.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5562bb163801a8-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.586 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.587 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.587 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.588 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.588 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.589 DEBUG:boto:Method: GET 2023-10-17T18:19:53.589 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:19:53.590 DEBUG:boto:Data: 2023-10-17T18:19:53.590 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.591 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.591 DEBUG:boto:Params: {} 2023-10-17T18:19:53.592 DEBUG:boto:Token: None 2023-10-17T18:19:53.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/?versions 2023-10-17T18:19:53.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8= 2023-10-17T18:19:53.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1ixMr/Hoi3jaEaW/0WqoUg1mVi8=', 'Content-Length': '0'} 2023-10-17T18:19:53.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16108fceda9d9dc-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.596 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:53.597 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.598 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.598 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.599 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.599 DEBUG:boto:Method: GET 2023-10-17T18:19:53.599 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.600 DEBUG:boto:Data: 2023-10-17T18:19:53.600 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.601 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.601 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.602 DEBUG:boto:Params: {} 2023-10-17T18:19:53.602 DEBUG:boto:Token: None 2023-10-17T18:19:53.603 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.603 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009504186c2bb0f94-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.606 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.606 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.607 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.607 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.608 DEBUG:boto:Method: GET 2023-10-17T18:19:53.608 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.608 DEBUG:boto:Data: 2023-10-17T18:19:53.609 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.610 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.610 DEBUG:boto:Params: {} 2023-10-17T18:19:53.611 DEBUG:boto:Token: None 2023-10-17T18:19:53.611 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af3a4aa4fb0ec39f-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.614 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.615 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.615 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.616 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.616 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.616 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.617 DEBUG:boto:Data: 2023-10-17T18:19:53.617 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.618 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.618 DEBUG:boto:Params: {} 2023-10-17T18:19:53.619 DEBUG:boto:Token: None 2023-10-17T18:19:53.619 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aYqarF9UJyX780BnwX1CnzSC9WQ= 2023-10-17T18:19:53.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aYqarF9UJyX780BnwX1CnzSC9WQ=', 'Content-Length': '0'} 2023-10-17T18:19:53.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048b33fb0bd690baf-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.622 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.623 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.623 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.623 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.624 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.624 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.625 DEBUG:boto:Data: 2023-10-17T18:19:53.625 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.626 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.626 DEBUG:boto:Params: {} 2023-10-17T18:19:53.627 DEBUG:boto:Token: None 2023-10-17T18:19:53.627 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aYqarF9UJyX780BnwX1CnzSC9WQ= 2023-10-17T18:19:53.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aYqarF9UJyX780BnwX1CnzSC9WQ=', 'Content-Length': '0'} 2023-10-17T18:19:53.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000512bae9fd0fdb41b-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:53.631 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.631 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.632 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.632 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.633 DEBUG:boto:Method: GET 2023-10-17T18:19:53.633 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.633 DEBUG:boto:Data: 2023-10-17T18:19:53.634 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.635 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.635 DEBUG:boto:Params: {} 2023-10-17T18:19:53.636 DEBUG:boto:Token: None 2023-10-17T18:19:53.636 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d56f12758890e1e1-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.639 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.640 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.640 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.640 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.641 DEBUG:boto:Method: GET 2023-10-17T18:19:53.641 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.642 DEBUG:boto:Data: 2023-10-17T18:19:53.642 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.643 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.643 DEBUG:boto:Params: {} 2023-10-17T18:19:53.644 DEBUG:boto:Token: None 2023-10-17T18:19:53.644 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7552aeb9397b3d0-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.648 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.648 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.648 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.649 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.649 DEBUG:boto:Method: GET 2023-10-17T18:19:53.650 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.650 DEBUG:boto:Data: 2023-10-17T18:19:53.651 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.651 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.652 DEBUG:boto:Params: {} 2023-10-17T18:19:53.652 DEBUG:boto:Token: None 2023-10-17T18:19:53.653 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2d8db0cab009e2-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.698 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.699 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.700 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.700 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.700 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.701 DEBUG:boto:Method: GET 2023-10-17T18:19:53.701 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.702 DEBUG:boto:Data: 2023-10-17T18:19:53.702 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.703 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.703 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.703 DEBUG:boto:Params: {} 2023-10-17T18:19:53.704 DEBUG:boto:Token: None 2023-10-17T18:19:53.705 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.705 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0876336cef38ed-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.709 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.709 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.710 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.710 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.711 DEBUG:boto:Path: /hbmuog-12/dummy 2023-10-17T18:19:53.711 DEBUG:boto:Data: 2023-10-17T18:19:53.711 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.712 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.712 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.713 DEBUG:boto:Params: {} 2023-10-17T18:19:53.713 DEBUG:boto:Token: None 2023-10-17T18:19:53.713 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy 2023-10-17T18:19:53.714 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Vw2X4T9AdbYoEjpe2ZnghmJ8RI= 2023-10-17T18:19:53.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Vw2X4T9AdbYoEjpe2ZnghmJ8RI=', 'Content-Length': '0'} 2023-10-17T18:19:53.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1657c40b5edaf4b-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.717 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.717 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.718 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.718 DEBUG:boto:Path: /hbmuog-12/dummy 2023-10-17T18:19:53.718 DEBUG:boto:Data: 2023-10-17T18:19:53.719 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.720 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.720 DEBUG:boto:Params: {} 2023-10-17T18:19:53.720 DEBUG:boto:Token: None 2023-10-17T18:19:53.721 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy 2023-10-17T18:19:53.721 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Vw2X4T9AdbYoEjpe2ZnghmJ8RI= 2023-10-17T18:19:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Vw2X4T9AdbYoEjpe2ZnghmJ8RI=', 'Content-Length': '0'} 2023-10-17T18:19:53.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e842c8c35029021-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:19:53.725 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.726 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.726 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.726 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.727 DEBUG:boto:Method: GET 2023-10-17T18:19:53.727 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.728 DEBUG:boto:Data: 2023-10-17T18:19:53.728 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.729 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.729 DEBUG:boto:Params: {} 2023-10-17T18:19:53.730 DEBUG:boto:Token: None 2023-10-17T18:19:53.730 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000884f1c10e23d2746-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.734 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:19:53.734 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:19:53.734 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.735 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.735 DEBUG:boto:Method: GET 2023-10-17T18:19:53.736 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.736 DEBUG:boto:Data: 2023-10-17T18:19:53.737 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.737 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.738 DEBUG:boto:Params: {} 2023-10-17T18:19:53.738 DEBUG:boto:Token: None 2023-10-17T18:19:53.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:19:53.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4= 2023-10-17T18:19:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9aZ/QjrMp7mEwvJywiMdfaOFMq4=', 'Content-Length': '0'} 2023-10-17T18:19:53.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000988ab0cc1091e6fa-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.742 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.743 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.743 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.744 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.744 DEBUG:boto:Method: GET 2023-10-17T18:19:53.744 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.745 DEBUG:boto:Data: 2023-10-17T18:19:53.745 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.746 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.747 DEBUG:boto:Params: {} 2023-10-17T18:19:53.747 DEBUG:boto:Token: None 2023-10-17T18:19:53.747 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ff872b720b1eff-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.752 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.752 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.753 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.753 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.754 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.754 DEBUG:boto:Method: GET 2023-10-17T18:19:53.755 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:19:53.755 DEBUG:boto:Data: 2023-10-17T18:19:53.756 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.757 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.757 DEBUG:boto:Params: {} 2023-10-17T18:19:53.757 DEBUG:boto:Token: None 2023-10-17T18:19:53.758 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b58719da9b71c8f-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.762 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:53.763 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.764 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.764 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.764 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.765 DEBUG:boto:Method: GET 2023-10-17T18:19:53.765 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.766 DEBUG:boto:Data: 2023-10-17T18:19:53.766 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.767 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.767 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.767 DEBUG:boto:Params: {} 2023-10-17T18:19:53.768 DEBUG:boto:Token: None 2023-10-17T18:19:53.768 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d959c3ccc327efb1-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.772 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.772 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.773 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.773 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.773 DEBUG:boto:Method: GET 2023-10-17T18:19:53.774 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.774 DEBUG:boto:Data: 2023-10-17T18:19:53.775 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.776 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.776 DEBUG:boto:Params: {} 2023-10-17T18:19:53.776 DEBUG:boto:Token: None 2023-10-17T18:19:53.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017d552608cd18b7d-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.780 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.781 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.781 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.781 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.782 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.782 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.783 DEBUG:boto:Data: 2023-10-17T18:19:53.785 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.786 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.786 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.786 DEBUG:boto:Params: {} 2023-10-17T18:19:53.787 DEBUG:boto:Token: None 2023-10-17T18:19:53.787 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3UhI0u+F8+GiTlaZS96iEve18aI= 2023-10-17T18:19:53.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3UhI0u+F8+GiTlaZS96iEve18aI=', 'Content-Length': '0'} 2023-10-17T18:19:53.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007363e9eafb969d88-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.791 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.792 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.792 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.792 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.793 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.793 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.794 DEBUG:boto:Data: 2023-10-17T18:19:53.794 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.795 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.795 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.795 DEBUG:boto:Params: {} 2023-10-17T18:19:53.796 DEBUG:boto:Token: None 2023-10-17T18:19:53.796 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.797 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3UhI0u+F8+GiTlaZS96iEve18aI= 2023-10-17T18:19:53.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3UhI0u+F8+GiTlaZS96iEve18aI=', 'Content-Length': '0'} 2023-10-17T18:19:53.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000249058406ee32cf2-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:53.800 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.800 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.801 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.801 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.801 DEBUG:boto:Method: GET 2023-10-17T18:19:53.802 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.802 DEBUG:boto:Data: 2023-10-17T18:19:53.803 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.803 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.803 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.804 DEBUG:boto:Params: {} 2023-10-17T18:19:53.804 DEBUG:boto:Token: None 2023-10-17T18:19:53.805 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ff85d58159009ff-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.850 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.851 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.851 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.852 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.852 DEBUG:boto:Method: GET 2023-10-17T18:19:53.853 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.853 DEBUG:boto:Data: 2023-10-17T18:19:53.853 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.854 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.854 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.855 DEBUG:boto:Params: {} 2023-10-17T18:19:53.855 DEBUG:boto:Token: None 2023-10-17T18:19:53.856 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bade6057cfbb951-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.859 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.860 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.860 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.861 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.861 DEBUG:boto:Method: GET 2023-10-17T18:19:53.861 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.862 DEBUG:boto:Data: 2023-10-17T18:19:53.862 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.863 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.863 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.864 DEBUG:boto:Params: {} 2023-10-17T18:19:53.864 DEBUG:boto:Token: None 2023-10-17T18:19:53.865 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005788b61c42b75f1-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.868 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.869 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.869 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.870 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.870 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.871 DEBUG:boto:Method: GET 2023-10-17T18:19:53.871 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.871 DEBUG:boto:Data: 2023-10-17T18:19:53.872 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.873 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.873 DEBUG:boto:Params: {} 2023-10-17T18:19:53.874 DEBUG:boto:Token: None 2023-10-17T18:19:53.874 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84408aa541d2f1b-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.878 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.878 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.879 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.879 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.880 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:19:53.880 DEBUG:boto:Data: 2023-10-17T18:19:53.881 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.881 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.881 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.882 DEBUG:boto:Params: {} 2023-10-17T18:19:53.882 DEBUG:boto:Token: None 2023-10-17T18:19:53.883 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1 2023-10-17T18:19:53.883 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JOG4jLIzRCtYkMAy6hkxHPKMp9M= 2023-10-17T18:19:53.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JOG4jLIzRCtYkMAy6hkxHPKMp9M=', 'Content-Length': '0'} 2023-10-17T18:19:53.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000629a1f77dea8ec6c-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.886 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.887 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.887 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.887 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:19:53.888 DEBUG:boto:Data: 2023-10-17T18:19:53.888 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.889 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.889 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.889 DEBUG:boto:Params: {} 2023-10-17T18:19:53.890 DEBUG:boto:Token: None 2023-10-17T18:19:53.890 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1 2023-10-17T18:19:53.891 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JOG4jLIzRCtYkMAy6hkxHPKMp9M= 2023-10-17T18:19:53.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JOG4jLIzRCtYkMAy6hkxHPKMp9M=', 'Content-Length': '0'} 2023-10-17T18:19:53.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034de42b484625f33-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:19:53.894 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.895 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.895 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.896 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.896 DEBUG:boto:Method: GET 2023-10-17T18:19:53.897 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.897 DEBUG:boto:Data: 2023-10-17T18:19:53.897 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.898 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.898 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.899 DEBUG:boto:Params: {} 2023-10-17T18:19:53.899 DEBUG:boto:Token: None 2023-10-17T18:19:53.900 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.900 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a70404d9d5a76bdf-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.903 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.903 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:53.904 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.904 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.905 DEBUG:boto:Method: GET 2023-10-17T18:19:53.905 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.905 DEBUG:boto:Data: 2023-10-17T18:19:53.906 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.907 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.907 DEBUG:boto:Params: {} 2023-10-17T18:19:53.908 DEBUG:boto:Token: None 2023-10-17T18:19:53.908 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:19:53.909 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88= 2023-10-17T18:19:53.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w9qKSYZ43mhNzfpDIwkyQW0PE88=', 'Content-Length': '0'} 2023-10-17T18:19:53.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a39c5265c95c76d9-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.912 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.912 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.913 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.913 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.913 DEBUG:boto:Method: GET 2023-10-17T18:19:53.914 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.914 DEBUG:boto:Data: 2023-10-17T18:19:53.915 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.916 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.916 DEBUG:boto:Params: {} 2023-10-17T18:19:53.916 DEBUG:boto:Token: None 2023-10-17T18:19:53.917 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf4174aa67d3dc0-00652ed049-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.921 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.921 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:53.922 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:53.922 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.923 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.923 DEBUG:boto:Method: GET 2023-10-17T18:19:53.924 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:19:53.924 DEBUG:boto:Data: 2023-10-17T18:19:53.924 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.926 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.926 DEBUG:boto:Params: {} 2023-10-17T18:19:53.926 DEBUG:boto:Token: None 2023-10-17T18:19:53.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/ 2023-10-17T18:19:53.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA= 2023-10-17T18:19:53.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OWtoY0dneWB5LNA+gpKc9zthacA=', 'Content-Length': '0'} 2023-10-17T18:19:53.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06540062d0a4572-00652ed049-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.970 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:53.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:53.972 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.972 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.973 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.973 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.973 DEBUG:boto:Method: GET 2023-10-17T18:19:53.974 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.974 DEBUG:boto:Data: 2023-10-17T18:19:53.975 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.975 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.976 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.976 DEBUG:boto:Params: {} 2023-10-17T18:19:53.977 DEBUG:boto:Token: None 2023-10-17T18:19:53.977 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.978 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+iMCjDhkB39ytE4COftICqmpZmw= 2023-10-17T18:19:53.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+iMCjDhkB39ytE4COftICqmpZmw=', 'Content-Length': '0'} 2023-10-17T18:19:53.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006da720ff3833bf6d-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.981 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.981 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.982 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.982 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.982 DEBUG:boto:Method: GET 2023-10-17T18:19:53.983 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.983 DEBUG:boto:Data: 2023-10-17T18:19:53.984 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.984 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:53.985 DEBUG:boto:Port: 8000 2023-10-17T18:19:53.985 DEBUG:boto:Params: {} 2023-10-17T18:19:53.985 DEBUG:boto:Token: None 2023-10-17T18:19:53.986 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.987 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+iMCjDhkB39ytE4COftICqmpZmw= 2023-10-17T18:19:53.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+iMCjDhkB39ytE4COftICqmpZmw=', 'Content-Length': '0'} 2023-10-17T18:19:53.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a77555c072cf041a-00652ed049-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:53.992 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.992 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:53.993 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.993 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.993 DEBUG:boto:Method: HEAD 2023-10-17T18:19:53.994 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.994 DEBUG:boto:Data: 2023-10-17T18:19:53.995 DEBUG:boto:Headers: {} 2023-10-17T18:19:53.995 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:53.995 DEBUG:boto:Port: 8001 2023-10-17T18:19:53.996 DEBUG:boto:Params: {} 2023-10-17T18:19:53.996 DEBUG:boto:Token: None 2023-10-17T18:19:53.997 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:53 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:53.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+nBDJYROGrcExQ6FWZh4e8LN9Uc= 2023-10-17T18:19:53.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+nBDJYROGrcExQ6FWZh4e8LN9Uc=', 'Content-Length': '0'} 2023-10-17T18:19:53.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004229648ece6d4726-00652ed049-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.000 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.000 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.001 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.001 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.001 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.002 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.002 DEBUG:boto:Data: 2023-10-17T18:19:54.003 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.003 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.004 DEBUG:boto:Params: {} 2023-10-17T18:19:54.004 DEBUG:boto:Token: None 2023-10-17T18:19:54.005 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y8W6XG3k34yOzg8ojmdLyJuKvUY= 2023-10-17T18:19:54.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y8W6XG3k34yOzg8ojmdLyJuKvUY=', 'Content-Length': '0'} 2023-10-17T18:19:54.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060e4cdf017870301-00652ed04a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:54.008 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.009 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.009 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.010 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.010 DEBUG:boto:Method: GET 2023-10-17T18:19:54.011 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.011 DEBUG:boto:Data: 2023-10-17T18:19:54.012 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.012 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.012 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.013 DEBUG:boto:Params: {} 2023-10-17T18:19:54.013 DEBUG:boto:Token: None 2023-10-17T18:19:54.014 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.014 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I= 2023-10-17T18:19:54.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb256bf6bf27a9c-00652ed04a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.017 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.018 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.018 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.019 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.019 DEBUG:boto:Method: GET 2023-10-17T18:19:54.020 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.020 DEBUG:boto:Data: 2023-10-17T18:19:54.020 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.021 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.021 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.022 DEBUG:boto:Params: {} 2023-10-17T18:19:54.022 DEBUG:boto:Token: None 2023-10-17T18:19:54.023 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I= 2023-10-17T18:19:54.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc28520ac5a6ca7e-00652ed04a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.026 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.027 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.027 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.027 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.028 DEBUG:boto:Method: GET 2023-10-17T18:19:54.029 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.029 DEBUG:boto:Data: 2023-10-17T18:19:54.030 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.031 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.031 DEBUG:boto:Params: {} 2023-10-17T18:19:54.031 DEBUG:boto:Token: None 2023-10-17T18:19:54.032 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE= 2023-10-17T18:19:54.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE=', 'Content-Length': '0'} 2023-10-17T18:19:54.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efbf64e7bdf96978-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.035 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.036 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.037 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.037 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.037 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.038 DEBUG:boto:Method: GET 2023-10-17T18:19:54.038 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.039 DEBUG:boto:Data: 2023-10-17T18:19:54.039 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.040 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.040 DEBUG:boto:Params: {} 2023-10-17T18:19:54.041 DEBUG:boto:Token: None 2023-10-17T18:19:54.041 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.042 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE= 2023-10-17T18:19:54.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE=', 'Content-Length': '0'} 2023-10-17T18:19:54.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe176f0d709dd92b-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.045 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.046 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.046 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.046 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.047 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:19:54.047 DEBUG:boto:Data: 2023-10-17T18:19:54.048 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.048 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.048 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.049 DEBUG:boto:Params: {} 2023-10-17T18:19:54.049 DEBUG:boto:Token: None 2023-10-17T18:19:54.050 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2 2023-10-17T18:19:54.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jUn0imdTThMQAODyOi9rE+PRx1I= 2023-10-17T18:19:54.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jUn0imdTThMQAODyOi9rE+PRx1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007991341ef1a6b2e6-00652ed04a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.053 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.054 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.054 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.055 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:19:54.055 DEBUG:boto:Data: 2023-10-17T18:19:54.055 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.056 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.057 DEBUG:boto:Params: {} 2023-10-17T18:19:54.057 DEBUG:boto:Token: None 2023-10-17T18:19:54.057 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2 2023-10-17T18:19:54.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jUn0imdTThMQAODyOi9rE+PRx1I= 2023-10-17T18:19:54.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jUn0imdTThMQAODyOi9rE+PRx1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dc761f802c59010-00652ed04a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:19:54.061 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.062 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.062 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.063 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.063 DEBUG:boto:Method: GET 2023-10-17T18:19:54.064 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.064 DEBUG:boto:Data: 2023-10-17T18:19:54.064 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.065 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.066 DEBUG:boto:Params: {} 2023-10-17T18:19:54.066 DEBUG:boto:Token: None 2023-10-17T18:19:54.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I= 2023-10-17T18:19:54.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a061a766d4e688d3-00652ed04a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.110 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.111 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.111 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.112 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.112 DEBUG:boto:Method: GET 2023-10-17T18:19:54.113 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.113 DEBUG:boto:Data: 2023-10-17T18:19:54.114 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.114 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.114 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.115 DEBUG:boto:Params: {} 2023-10-17T18:19:54.115 DEBUG:boto:Token: None 2023-10-17T18:19:54.116 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:19:54.116 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I= 2023-10-17T18:19:54.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3e7kxxYBOzfepdQB4Z/PwMXO1I=', 'Content-Length': '0'} 2023-10-17T18:19:54.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e51fd4e3854d221-00652ed04a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.120 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.120 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.120 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.121 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.121 DEBUG:boto:Method: GET 2023-10-17T18:19:54.122 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.122 DEBUG:boto:Data: 2023-10-17T18:19:54.123 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.124 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.124 DEBUG:boto:Params: {} 2023-10-17T18:19:54.125 DEBUG:boto:Token: None 2023-10-17T18:19:54.125 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.126 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE= 2023-10-17T18:19:54.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE=', 'Content-Length': '0'} 2023-10-17T18:19:54.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec817d06a07470d-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.129 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.130 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.130 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.131 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.131 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.132 DEBUG:boto:Method: GET 2023-10-17T18:19:54.132 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:19:54.132 DEBUG:boto:Data: 2023-10-17T18:19:54.133 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.133 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.134 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.134 DEBUG:boto:Params: {} 2023-10-17T18:19:54.135 DEBUG:boto:Token: None 2023-10-17T18:19:54.135 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.135 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE= 2023-10-17T18:19:54.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqKZD+D4TN6MYPxt63iI0jwSBpE=', 'Content-Length': '0'} 2023-10-17T18:19:54.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004706da21899b81a2-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.182 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.183 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-12 zones={a2, a1} 2023-10-17T18:19:54.184 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:54.184 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:54.184 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.185 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:54.185 DEBUG:boto:Data: 2023-10-17T18:19:54.186 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.186 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.187 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.187 DEBUG:boto:Params: {} 2023-10-17T18:19:54.187 DEBUG:boto:Token: None 2023-10-17T18:19:54.188 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-11/ 2023-10-17T18:19:54.189 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CK6r7L1aY+mv0No8+O3iovefC14= 2023-10-17T18:19:54.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CK6r7L1aY+mv0No8+O3iovefC14=', 'Content-Length': '0'} 2023-10-17T18:19:54.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000691d237f4b97f3c5-00652ed04a-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.192 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:54.192 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:54.192 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.193 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.193 DEBUG:boto:Method: GET 2023-10-17T18:19:54.194 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.194 DEBUG:boto:Data: 2023-10-17T18:19:54.195 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.198 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.198 DEBUG:boto:Params: {} 2023-10-17T18:19:54.199 DEBUG:boto:Token: None 2023-10-17T18:19:54.199 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-11/?versions 2023-10-17T18:19:54.199 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/4gfCUVXtCXkvAWBBaBrpXmNu0U= 2023-10-17T18:19:54.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/4gfCUVXtCXkvAWBBaBrpXmNu0U=', 'Content-Length': '0'} 2023-10-17T18:19:54.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6becc36d5aaaa12-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.203 DEBUG:boto:b'hbmuog-11obj-a1999falseobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.204 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:19:54.204 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:19:54.204 DEBUG:boto:path=/hbmuog-11/obj-a1?acl 2023-10-17T18:19:54.205 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?acl 2023-10-17T18:19:54.205 DEBUG:boto:Method: PUT 2023-10-17T18:19:54.206 DEBUG:boto:Path: /hbmuog-11/obj-a1?acl 2023-10-17T18:19:54.206 DEBUG:boto:Data: 2023-10-17T18:19:54.206 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:19:54.207 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.207 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.208 DEBUG:boto:Params: {} 2023-10-17T18:19:54.208 DEBUG:boto:Token: None 2023-10-17T18:19:54.209 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:19:54 GMT x-amz-acl:public-read /hbmuog-11/obj-a1?acl 2023-10-17T18:19:54.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nEcVhPbLJ4+hmau4Q3k30i16CWE= 2023-10-17T18:19:54.209 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nEcVhPbLJ4+hmau4Q3k30i16CWE=', 'Content-Length': '0'} 2023-10-17T18:19:54.214 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca4e7de62520767c-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.214 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:54.215 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:54.215 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.215 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:19:54.216 DEBUG:boto:Data: 2023-10-17T18:19:54.216 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.217 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.218 DEBUG:boto:Params: {} 2023-10-17T18:19:54.218 DEBUG:boto:Token: None 2023-10-17T18:19:54.218 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-11/ 2023-10-17T18:19:54.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CK6r7L1aY+mv0No8+O3iovefC14= 2023-10-17T18:19:54.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CK6r7L1aY+mv0No8+O3iovefC14=', 'Content-Length': '0'} 2023-10-17T18:19:54.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be2a8070b818bde-00652ed04a-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.222 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:19:54.223 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:19:54.223 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.223 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.224 DEBUG:boto:Method: GET 2023-10-17T18:19:54.224 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.225 DEBUG:boto:Data: 2023-10-17T18:19:54.225 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.226 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.226 DEBUG:boto:Params: {} 2023-10-17T18:19:54.227 DEBUG:boto:Token: None 2023-10-17T18:19:54.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-11/?versions 2023-10-17T18:19:54.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/4gfCUVXtCXkvAWBBaBrpXmNu0U= 2023-10-17T18:19:54.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/4gfCUVXtCXkvAWBBaBrpXmNu0U=', 'Content-Length': '0'} 2023-10-17T18:19:54.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7df81a0a5f3f766-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.231 DEBUG:boto:b'hbmuog-11obj-a2999falseobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.231 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:19:54.232 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:19:54.232 DEBUG:boto:path=/hbmuog-11/obj-a2?acl 2023-10-17T18:19:54.233 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?acl 2023-10-17T18:19:54.233 DEBUG:boto:Method: PUT 2023-10-17T18:19:54.234 DEBUG:boto:Path: /hbmuog-11/obj-a2?acl 2023-10-17T18:19:54.234 DEBUG:boto:Data: 2023-10-17T18:19:54.234 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:19:54.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.235 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.236 DEBUG:boto:Params: {} 2023-10-17T18:19:54.236 DEBUG:boto:Token: None 2023-10-17T18:19:54.236 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:19:54 GMT x-amz-acl:public-read /hbmuog-11/obj-a2?acl 2023-10-17T18:19:54.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MhWawlDUDrkDv7QOYKx9Qu1vIwE= 2023-10-17T18:19:54.237 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MhWawlDUDrkDv7QOYKx9Qu1vIwE=', 'Content-Length': '0'} 2023-10-17T18:19:54.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a001f10e3eec895c-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.242 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.243 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.243 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.243 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:54.244 DEBUG:boto:Data: 2023-10-17T18:19:54.244 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.245 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.245 DEBUG:boto:Params: {} 2023-10-17T18:19:54.246 DEBUG:boto:Token: None 2023-10-17T18:19:54.246 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sIM5nt0qnqmNeROo/rEUUpW9Xns= 2023-10-17T18:19:54.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sIM5nt0qnqmNeROo/rEUUpW9Xns=', 'Content-Length': '0'} 2023-10-17T18:19:54.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9f376d696338d8d-00652ed04a-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.250 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.250 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.251 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.251 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.251 DEBUG:boto:Method: GET 2023-10-17T18:19:54.252 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999&prefix=obj-a1 2023-10-17T18:19:54.252 DEBUG:boto:Data: 2023-10-17T18:19:54.253 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.253 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.253 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.254 DEBUG:boto:Params: {} 2023-10-17T18:19:54.254 DEBUG:boto:Token: None 2023-10-17T18:19:54.255 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/?versions 2023-10-17T18:19:54.255 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FSUzawC1a8oKLeFUMOM4k+szpfc= 2023-10-17T18:19:54.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FSUzawC1a8oKLeFUMOM4k+szpfc=', 'Content-Length': '0'} 2023-10-17T18:19:54.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b38641f398b7a5d-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.306 DEBUG:boto:b'hbmuog-12obj-a1999falseobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.307 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:19:54.308 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:19:54.308 DEBUG:boto:path=/hbmuog-12/obj-a1?acl 2023-10-17T18:19:54.308 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?acl 2023-10-17T18:19:54.309 DEBUG:boto:Method: PUT 2023-10-17T18:19:54.309 DEBUG:boto:Path: /hbmuog-12/obj-a1?acl 2023-10-17T18:19:54.310 DEBUG:boto:Data: 2023-10-17T18:19:54.310 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:19:54.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:19:54.311 DEBUG:boto:Port: 8000 2023-10-17T18:19:54.311 DEBUG:boto:Params: {} 2023-10-17T18:19:54.312 DEBUG:boto:Token: None 2023-10-17T18:19:54.312 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:19:54 GMT x-amz-acl:public-read /hbmuog-12/obj-a1?acl 2023-10-17T18:19:54.313 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ygy2sWoT+G75QkJ14QG1VPOkVy4= 2023-10-17T18:19:54.313 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ygy2sWoT+G75QkJ14QG1VPOkVy4=', 'Content-Length': '0'} 2023-10-17T18:19:54.318 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000117ad52b689734a7-00652ed04a-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.318 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.319 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.319 DEBUG:boto:Method: HEAD 2023-10-17T18:19:54.319 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:19:54.320 DEBUG:boto:Data: 2023-10-17T18:19:54.320 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.321 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.321 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.321 DEBUG:boto:Params: {} 2023-10-17T18:19:54.322 DEBUG:boto:Token: None 2023-10-17T18:19:54.322 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/ 2023-10-17T18:19:54.323 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sIM5nt0qnqmNeROo/rEUUpW9Xns= 2023-10-17T18:19:54.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sIM5nt0qnqmNeROo/rEUUpW9Xns=', 'Content-Length': '0'} 2023-10-17T18:19:54.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027f7eab66ac5e88c-00652ed04a-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.325 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:19:54.326 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:19:54.326 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.327 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.327 DEBUG:boto:Method: GET 2023-10-17T18:19:54.327 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999&prefix=obj-a2 2023-10-17T18:19:54.328 DEBUG:boto:Data: 2023-10-17T18:19:54.328 DEBUG:boto:Headers: {} 2023-10-17T18:19:54.329 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.329 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.329 DEBUG:boto:Params: {} 2023-10-17T18:19:54.330 DEBUG:boto:Token: None 2023-10-17T18:19:54.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:19:54 GMT /hbmuog-12/?versions 2023-10-17T18:19:54.331 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FSUzawC1a8oKLeFUMOM4k+szpfc= 2023-10-17T18:19:54.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FSUzawC1a8oKLeFUMOM4k+szpfc=', 'Content-Length': '0'} 2023-10-17T18:19:54.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cfc2bb4ab70254-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.334 DEBUG:boto:b'hbmuog-12obj-a2999falseobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:19:54.335 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.335 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:19:54.336 DEBUG:boto:path=/hbmuog-12/obj-a2?acl 2023-10-17T18:19:54.336 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?acl 2023-10-17T18:19:54.336 DEBUG:boto:Method: PUT 2023-10-17T18:19:54.337 DEBUG:boto:Path: /hbmuog-12/obj-a2?acl 2023-10-17T18:19:54.337 DEBUG:boto:Data: 2023-10-17T18:19:54.338 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:19:54.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:19:54.338 DEBUG:boto:Port: 8001 2023-10-17T18:19:54.339 DEBUG:boto:Params: {} 2023-10-17T18:19:54.339 DEBUG:boto:Token: None 2023-10-17T18:19:54.340 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:19:54 GMT x-amz-acl:public-read /hbmuog-12/obj-a2?acl 2023-10-17T18:19:54.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4UVxksWe+CoF3xRL2K3PDL23UUM= 2023-10-17T18:19:54.341 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:19:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4UVxksWe+CoF3xRL2K3PDL23UUM=', 'Content-Length': '0'} 2023-10-17T18:19:54.344 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b81d2de9a6f0cdaf-00652ed04a-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:19:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:19:54.345 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-11 2023-10-17T18:19:54.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:54.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:19:54.346 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:19:54.404 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:19:54.404 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:19:54.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:54.437+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:54.439 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:19:54.439 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:19:59.438+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:04.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:04.441+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:20:04.441 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:20:04.442 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:09.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:09.442+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:20:09.443 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:20:09.443 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:14.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:14.442+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:20:14.444 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:20:14.444 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:19.444+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:20:19.445 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:20:19.445 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:24.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:24.445+0000 7fdcacead780 1 waiting for incremental sync to catch up: 2023-10-17T18:20:24.447 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000010.20.6, ] 2023-10-17T18:20:24.447 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:29.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:29.446+0000 7fdcacead780 1 bucket sync caught up with source: 2023-10-17T18:20:29.448 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:29.448 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.6.10, , , 00000000002.8.11, 00000000011.23.6, ] 2023-10-17T18:20:29.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:29.446+0000 7fdcacead780 0 bucket checkpoint complete 2023-10-17T18:20:29.462 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-11 2023-10-17T18:20:29.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:29.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:29.464 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:20:29.525 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:29.526 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:29.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:29.560+0000 7f0ce2d26780 1 bucket sync caught up with source: 2023-10-17T18:20:29.562 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000011.43.6, 00000000002.16.11, ] 2023-10-17T18:20:29.562 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000011.43.6, 00000000002.16.11, ] 2023-10-17T18:20:29.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:29.560+0000 7f0ce2d26780 0 bucket checkpoint complete 2023-10-17T18:20:29.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-11 zones={a2, a1} 2023-10-17T18:20:29.575 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.575 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.576 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.576 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:20:29.576 DEBUG:boto:Data: 2023-10-17T18:20:29.577 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.577 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.578 DEBUG:boto:Params: {} 2023-10-17T18:20:29.578 DEBUG:boto:Token: None 2023-10-17T18:20:29.579 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5BhsIjwky5+HJuAOOTO3/NoJPf4= 2023-10-17T18:20:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5BhsIjwky5+HJuAOOTO3/NoJPf4=', 'Content-Length': '0'} 2023-10-17T18:20:29.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf6a5896c015b22e-00652ed06d-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.583 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.583 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.584 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.584 DEBUG:boto:Path: /hbmuog-11/ 2023-10-17T18:20:29.585 DEBUG:boto:Data: 2023-10-17T18:20:29.585 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.586 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.586 DEBUG:boto:Params: {} 2023-10-17T18:20:29.587 DEBUG:boto:Token: None 2023-10-17T18:20:29.587 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5BhsIjwky5+HJuAOOTO3/NoJPf4= 2023-10-17T18:20:29.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5BhsIjwky5+HJuAOOTO3/NoJPf4=', 'Content-Length': '0'} 2023-10-17T18:20:29.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031cc6f56821228e1-00652ed06d-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:20:29.592 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.592 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.592 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.593 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.593 DEBUG:boto:Method: GET 2023-10-17T18:20:29.594 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.594 DEBUG:boto:Data: 2023-10-17T18:20:29.594 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.595 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.596 DEBUG:boto:Params: {} 2023-10-17T18:20:29.596 DEBUG:boto:Token: None 2023-10-17T18:20:29.597 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/?versions 2023-10-17T18:20:29.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U= 2023-10-17T18:20:29.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U=', 'Content-Length': '0'} 2023-10-17T18:20:29.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009476c9f333b05eb7-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.601 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.602 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:20:29.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:20:29.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:20:29.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:20:29.603 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.604 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.604 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.605 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.605 DEBUG:boto:Method: GET 2023-10-17T18:20:29.605 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.606 DEBUG:boto:Data: 2023-10-17T18:20:29.606 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.609 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.610 DEBUG:boto:Params: {} 2023-10-17T18:20:29.610 DEBUG:boto:Token: None 2023-10-17T18:20:29.611 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/?versions 2023-10-17T18:20:29.611 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U= 2023-10-17T18:20:29.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U=', 'Content-Length': '0'} 2023-10-17T18:20:29.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fc44d97c28f5a9-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.614 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.615 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:20:29.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:20:29.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:20:29.616 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.617 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.617 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.618 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.618 DEBUG:boto:Method: GET 2023-10-17T18:20:29.618 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.619 DEBUG:boto:Data: 2023-10-17T18:20:29.619 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.620 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.621 DEBUG:boto:Params: {} 2023-10-17T18:20:29.621 DEBUG:boto:Token: None 2023-10-17T18:20:29.622 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/?versions 2023-10-17T18:20:29.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U= 2023-10-17T18:20:29.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U=', 'Content-Length': '0'} 2023-10-17T18:20:29.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e1d5da37b96a77-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.625 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.626 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.627 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.627 DEBUG:boto:path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.627 DEBUG:boto:auth_path=/hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.628 DEBUG:boto:Method: GET 2023-10-17T18:20:29.628 DEBUG:boto:Path: /hbmuog-11/?versions&max-keys=999 2023-10-17T18:20:29.628 DEBUG:boto:Data: 2023-10-17T18:20:29.629 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.629 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.630 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.630 DEBUG:boto:Params: {} 2023-10-17T18:20:29.631 DEBUG:boto:Token: None 2023-10-17T18:20:29.631 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/?versions 2023-10-17T18:20:29.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U= 2023-10-17T18:20:29.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mu1yMF7ElKzAlppkLMF77/5+C3U=', 'Content-Length': '0'} 2023-10-17T18:20:29.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b12fc8912b74efd9-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.635 DEBUG:boto:b'hbmuog-11999falsedummy-BdtRI40ti1P7aPyM6f4O1xGPX5wgpRtrue2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1XRS2SbgiLlJsIE-oecwxGN8WPUR0SIbtrue2023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a230-eD5wAiVUgjCn1taTqrI524fxVgHbtrue2023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:29.636 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.636 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.637 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.637 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.638 DEBUG:boto:Method: GET 2023-10-17T18:20:29.638 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.639 DEBUG:boto:Data: 2023-10-17T18:20:29.639 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.640 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.641 DEBUG:boto:Params: {} 2023-10-17T18:20:29.641 DEBUG:boto:Token: None 2023-10-17T18:20:29.642 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000beba236b61f11066-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.645 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.646 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.646 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.647 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.647 DEBUG:boto:Method: GET 2023-10-17T18:20:29.648 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.648 DEBUG:boto:Data: 2023-10-17T18:20:29.649 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.649 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.650 DEBUG:boto:Params: {} 2023-10-17T18:20:29.650 DEBUG:boto:Token: None 2023-10-17T18:20:29.651 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081336ce5a9993ae8-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.654 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.654 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.655 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.655 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.656 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.656 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.656 DEBUG:boto:Data: 2023-10-17T18:20:29.657 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.657 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.658 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.658 DEBUG:boto:Params: {} 2023-10-17T18:20:29.659 DEBUG:boto:Token: None 2023-10-17T18:20:29.659 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.660 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:a68Fz9QFbg3h2s6zlDiZDP9O6s8= 2023-10-17T18:20:29.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a68Fz9QFbg3h2s6zlDiZDP9O6s8=', 'Content-Length': '0'} 2023-10-17T18:20:29.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000521ffd869bfc7736-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.662 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.663 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.663 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.664 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.664 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.664 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.665 DEBUG:boto:Data: 2023-10-17T18:20:29.665 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.666 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.667 DEBUG:boto:Params: {} 2023-10-17T18:20:29.667 DEBUG:boto:Token: None 2023-10-17T18:20:29.668 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.668 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:a68Fz9QFbg3h2s6zlDiZDP9O6s8= 2023-10-17T18:20:29.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a68Fz9QFbg3h2s6zlDiZDP9O6s8=', 'Content-Length': '0'} 2023-10-17T18:20:29.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017cb026d42ed1635-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:29.672 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.672 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.673 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.673 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.673 DEBUG:boto:Method: GET 2023-10-17T18:20:29.674 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.674 DEBUG:boto:Data: 2023-10-17T18:20:29.675 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.675 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.676 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.676 DEBUG:boto:Params: {} 2023-10-17T18:20:29.676 DEBUG:boto:Token: None 2023-10-17T18:20:29.677 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.677 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f252293a16e2cca-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.680 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.681 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.681 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.682 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.682 DEBUG:boto:Method: GET 2023-10-17T18:20:29.683 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.683 DEBUG:boto:Data: 2023-10-17T18:20:29.684 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.684 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.684 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.685 DEBUG:boto:Params: {} 2023-10-17T18:20:29.685 DEBUG:boto:Token: None 2023-10-17T18:20:29.686 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000775cb82f730f4ee8-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.689 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.690 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.690 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.690 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.691 DEBUG:boto:Method: GET 2023-10-17T18:20:29.691 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.692 DEBUG:boto:Data: 2023-10-17T18:20:29.692 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.693 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.693 DEBUG:boto:Params: {} 2023-10-17T18:20:29.694 DEBUG:boto:Token: None 2023-10-17T18:20:29.694 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.695 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd64adf36421460e-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.738 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.739 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.740 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.740 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.740 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.741 DEBUG:boto:Method: GET 2023-10-17T18:20:29.741 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.742 DEBUG:boto:Data: 2023-10-17T18:20:29.742 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.743 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.744 DEBUG:boto:Params: {} 2023-10-17T18:20:29.744 DEBUG:boto:Token: None 2023-10-17T18:20:29.745 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c55490fa50a53129-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.749 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.749 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.750 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.750 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.751 DEBUG:boto:Path: /hbmuog-11/dummy 2023-10-17T18:20:29.751 DEBUG:boto:Data: 2023-10-17T18:20:29.751 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.752 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.752 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.753 DEBUG:boto:Params: {} 2023-10-17T18:20:29.753 DEBUG:boto:Token: None 2023-10-17T18:20:29.754 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy 2023-10-17T18:20:29.754 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzaLzHtAi+HnKCy0hgLKFq+SxU0= 2023-10-17T18:20:29.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzaLzHtAi+HnKCy0hgLKFq+SxU0=', 'Content-Length': '0'} 2023-10-17T18:20:29.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000472831df2324d70b-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.757 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.758 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.758 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.759 DEBUG:boto:Path: /hbmuog-11/dummy 2023-10-17T18:20:29.759 DEBUG:boto:Data: 2023-10-17T18:20:29.760 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.761 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.761 DEBUG:boto:Params: {} 2023-10-17T18:20:29.761 DEBUG:boto:Token: None 2023-10-17T18:20:29.762 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy 2023-10-17T18:20:29.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzaLzHtAi+HnKCy0hgLKFq+SxU0= 2023-10-17T18:20:29.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzaLzHtAi+HnKCy0hgLKFq+SxU0=', 'Content-Length': '0'} 2023-10-17T18:20:29.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020da6eadc5c76ba5-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:29.766 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.767 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.767 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.767 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.768 DEBUG:boto:Method: GET 2023-10-17T18:20:29.768 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.769 DEBUG:boto:Data: 2023-10-17T18:20:29.769 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.770 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.771 DEBUG:boto:Params: {} 2023-10-17T18:20:29.771 DEBUG:boto:Token: None 2023-10-17T18:20:29.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052680ab48f8ed0cc-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.775 DEBUG:boto:path=/hbmuog-11/dummy 2023-10-17T18:20:29.776 DEBUG:boto:auth_path=/hbmuog-11/dummy 2023-10-17T18:20:29.776 DEBUG:boto:path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.776 DEBUG:boto:auth_path=/hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.777 DEBUG:boto:Method: GET 2023-10-17T18:20:29.777 DEBUG:boto:Path: /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.778 DEBUG:boto:Data: 2023-10-17T18:20:29.778 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.779 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.780 DEBUG:boto:Params: {} 2023-10-17T18:20:29.780 DEBUG:boto:Token: None 2023-10-17T18:20:29.781 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/dummy?versionId=-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR 2023-10-17T18:20:29.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g= 2023-10-17T18:20:29.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0+Ibu+Ov98U9hGtsISBogw+tU0g=', 'Content-Length': '0'} 2023-10-17T18:20:29.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:36 GMT'), ('x-amz-version-id', '-BdtRI40ti1P7aPyM6f4O1xGPX5wgpR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007845abc96b0d885b-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.784 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.785 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.785 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.785 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.786 DEBUG:boto:Method: GET 2023-10-17T18:20:29.786 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.787 DEBUG:boto:Data: 2023-10-17T18:20:29.787 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.788 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.788 DEBUG:boto:Params: {} 2023-10-17T18:20:29.789 DEBUG:boto:Token: None 2023-10-17T18:20:29.789 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ddd0a933e61a1c-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.793 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.794 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.794 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.795 DEBUG:boto:path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.795 DEBUG:boto:auth_path=/hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.796 DEBUG:boto:Method: GET 2023-10-17T18:20:29.796 DEBUG:boto:Path: /hbmuog-11/?prefix=dummy 2023-10-17T18:20:29.797 DEBUG:boto:Data: 2023-10-17T18:20:29.797 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.798 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.798 DEBUG:boto:Params: {} 2023-10-17T18:20:29.799 DEBUG:boto:Token: None 2023-10-17T18:20:29.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf5f1dda3e008bf-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.803 DEBUG:boto:b'hbmuog-11dummy1000falsedummy2023-10-17T18:18:36.601Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:29.804 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.804 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.805 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.805 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.806 DEBUG:boto:Method: GET 2023-10-17T18:20:29.806 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.807 DEBUG:boto:Data: 2023-10-17T18:20:29.807 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.808 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.808 DEBUG:boto:Params: {} 2023-10-17T18:20:29.809 DEBUG:boto:Token: None 2023-10-17T18:20:29.809 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.810 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9e33fa4e5449025-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.813 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.814 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.814 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.815 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.815 DEBUG:boto:Method: GET 2023-10-17T18:20:29.816 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.816 DEBUG:boto:Data: 2023-10-17T18:20:29.816 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.817 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.818 DEBUG:boto:Params: {} 2023-10-17T18:20:29.818 DEBUG:boto:Token: None 2023-10-17T18:20:29.819 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb04c3b55fc5314-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.822 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.823 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.823 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.823 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.824 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.824 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.825 DEBUG:boto:Data: 2023-10-17T18:20:29.825 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.825 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.826 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.826 DEBUG:boto:Params: {} 2023-10-17T18:20:29.827 DEBUG:boto:Token: None 2023-10-17T18:20:29.827 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3xqmTPuCOvdUD3hOwqM4YxHQyNk= 2023-10-17T18:20:29.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3xqmTPuCOvdUD3hOwqM4YxHQyNk=', 'Content-Length': '0'} 2023-10-17T18:20:29.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9377ced8efb0409-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.831 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.831 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.832 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.832 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.832 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.833 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.833 DEBUG:boto:Data: 2023-10-17T18:20:29.834 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.834 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.835 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.835 DEBUG:boto:Params: {} 2023-10-17T18:20:29.835 DEBUG:boto:Token: None 2023-10-17T18:20:29.836 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.836 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3xqmTPuCOvdUD3hOwqM4YxHQyNk= 2023-10-17T18:20:29.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3xqmTPuCOvdUD3hOwqM4YxHQyNk=', 'Content-Length': '0'} 2023-10-17T18:20:29.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079b4f3c0470bbc3a-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:29.839 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.840 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.840 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.841 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.841 DEBUG:boto:Method: GET 2023-10-17T18:20:29.842 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.842 DEBUG:boto:Data: 2023-10-17T18:20:29.842 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.843 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.843 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.844 DEBUG:boto:Params: {} 2023-10-17T18:20:29.844 DEBUG:boto:Token: None 2023-10-17T18:20:29.845 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.845 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e469ae6b9ad12a00-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.890 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.891 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.891 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.892 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.892 DEBUG:boto:Method: GET 2023-10-17T18:20:29.893 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.893 DEBUG:boto:Data: 2023-10-17T18:20:29.893 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.894 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.895 DEBUG:boto:Params: {} 2023-10-17T18:20:29.895 DEBUG:boto:Token: None 2023-10-17T18:20:29.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000626ef5d0bbde6cfa-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.899 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.900 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.900 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.901 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.901 DEBUG:boto:Method: GET 2023-10-17T18:20:29.901 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.902 DEBUG:boto:Data: 2023-10-17T18:20:29.902 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.903 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.903 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.904 DEBUG:boto:Params: {} 2023-10-17T18:20:29.904 DEBUG:boto:Token: None 2023-10-17T18:20:29.905 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.905 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000329cea2f6d32b771-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.909 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.909 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.910 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.910 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.911 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.911 DEBUG:boto:Method: GET 2023-10-17T18:20:29.911 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.912 DEBUG:boto:Data: 2023-10-17T18:20:29.912 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.913 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.913 DEBUG:boto:Params: {} 2023-10-17T18:20:29.914 DEBUG:boto:Token: None 2023-10-17T18:20:29.914 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.915 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000867b74c00c72a3d-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.918 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.919 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.919 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.920 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.920 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:20:29.920 DEBUG:boto:Data: 2023-10-17T18:20:29.921 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.922 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.922 DEBUG:boto:Params: {} 2023-10-17T18:20:29.923 DEBUG:boto:Token: None 2023-10-17T18:20:29.923 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1 2023-10-17T18:20:29.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u73rzg7JWLOJKYqdC+vZYjViEQA= 2023-10-17T18:20:29.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u73rzg7JWLOJKYqdC+vZYjViEQA=', 'Content-Length': '0'} 2023-10-17T18:20:29.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015a752b6e1b1e5c2-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.926 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.927 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.927 DEBUG:boto:Method: HEAD 2023-10-17T18:20:29.928 DEBUG:boto:Path: /hbmuog-11/obj-a1 2023-10-17T18:20:29.928 DEBUG:boto:Data: 2023-10-17T18:20:29.929 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.930 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.930 DEBUG:boto:Params: {} 2023-10-17T18:20:29.930 DEBUG:boto:Token: None 2023-10-17T18:20:29.931 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1 2023-10-17T18:20:29.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u73rzg7JWLOJKYqdC+vZYjViEQA= 2023-10-17T18:20:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u73rzg7JWLOJKYqdC+vZYjViEQA=', 'Content-Length': '0'} 2023-10-17T18:20:29.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a94baf84aab444a4-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:29.935 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.935 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.936 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.936 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.937 DEBUG:boto:Method: GET 2023-10-17T18:20:29.937 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.937 DEBUG:boto:Data: 2023-10-17T18:20:29.938 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.939 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.939 DEBUG:boto:Params: {} 2023-10-17T18:20:29.940 DEBUG:boto:Token: None 2023-10-17T18:20:29.940 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d38fd451c13cbf33-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.944 DEBUG:boto:path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.944 DEBUG:boto:auth_path=/hbmuog-11/obj-a1 2023-10-17T18:20:29.945 DEBUG:boto:path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.945 DEBUG:boto:auth_path=/hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.945 DEBUG:boto:Method: GET 2023-10-17T18:20:29.946 DEBUG:boto:Path: /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.946 DEBUG:boto:Data: 2023-10-17T18:20:29.947 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.947 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.948 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.948 DEBUG:boto:Params: {} 2023-10-17T18:20:29.949 DEBUG:boto:Token: None 2023-10-17T18:20:29.949 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a1?versionId=XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb 2023-10-17T18:20:29.950 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo= 2023-10-17T18:20:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aoElhlxE7tPOeS2I3jh9UIdKBo=', 'Content-Length': '0'} 2023-10-17T18:20:29.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:11 GMT'), ('x-amz-version-id', 'XRS2SbgiLlJsIE-oecwxGN8WPUR0SIb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007562c100f7510df0-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.953 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.953 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.954 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.954 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.954 DEBUG:boto:Method: GET 2023-10-17T18:20:29.955 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.955 DEBUG:boto:Data: 2023-10-17T18:20:29.956 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.956 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.957 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.957 DEBUG:boto:Params: {} 2023-10-17T18:20:29.958 DEBUG:boto:Token: None 2023-10-17T18:20:29.958 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000404c20718a9f0cb-00652ed06d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.962 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.963 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:29.963 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:29.964 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.964 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.965 DEBUG:boto:Method: GET 2023-10-17T18:20:29.965 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a1 2023-10-17T18:20:29.965 DEBUG:boto:Data: 2023-10-17T18:20:29.966 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.967 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.967 DEBUG:boto:Params: {} 2023-10-17T18:20:29.968 DEBUG:boto:Token: None 2023-10-17T18:20:29.968 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/ 2023-10-17T18:20:29.969 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk= 2023-10-17T18:20:29.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vf78C86uxM06vxRNh4C8lt10MVk=', 'Content-Length': '0'} 2023-10-17T18:20:29.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda614ad0e9a367f-00652ed06d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.972 DEBUG:boto:b'hbmuog-11obj-a11000falseobj-a12023-10-17T18:19:11.889Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:29.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:29.973 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:29.974 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:29.974 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.975 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.975 DEBUG:boto:Method: GET 2023-10-17T18:20:29.976 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.976 DEBUG:boto:Data: 2023-10-17T18:20:29.977 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:29.978 DEBUG:boto:Port: 8001 2023-10-17T18:20:29.978 DEBUG:boto:Params: {} 2023-10-17T18:20:29.979 DEBUG:boto:Token: None 2023-10-17T18:20:29.979 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvlyxxYPykvOUtECcYLTBG2z038= 2023-10-17T18:20:29.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvlyxxYPykvOUtECcYLTBG2z038=', 'Content-Length': '0'} 2023-10-17T18:20:29.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d25e1f48f93ace5f-00652ed06d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:29.983 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:29.983 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:29.984 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.984 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.985 DEBUG:boto:Method: GET 2023-10-17T18:20:29.985 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.986 DEBUG:boto:Data: 2023-10-17T18:20:29.986 DEBUG:boto:Headers: {} 2023-10-17T18:20:29.987 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:29.987 DEBUG:boto:Port: 8000 2023-10-17T18:20:29.988 DEBUG:boto:Params: {} 2023-10-17T18:20:29.988 DEBUG:boto:Token: None 2023-10-17T18:20:29.989 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:29 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:29.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvlyxxYPykvOUtECcYLTBG2z038= 2023-10-17T18:20:29.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvlyxxYPykvOUtECcYLTBG2z038=', 'Content-Length': '0'} 2023-10-17T18:20:29.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077743be782428a30-00652ed06d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.038 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.039 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.039 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.040 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.040 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.041 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.041 DEBUG:boto:Data: 2023-10-17T18:20:30.041 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.042 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.042 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.043 DEBUG:boto:Params: {} 2023-10-17T18:20:30.043 DEBUG:boto:Token: None 2023-10-17T18:20:30.044 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.044 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A996Aa3PfX5bL3jlK9IK/oABGkM= 2023-10-17T18:20:30.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A996Aa3PfX5bL3jlK9IK/oABGkM=', 'Content-Length': '0'} 2023-10-17T18:20:30.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de97d5608a17c5da-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.047 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.047 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.048 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.048 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.049 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.049 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.049 DEBUG:boto:Data: 2023-10-17T18:20:30.050 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.050 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.051 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.051 DEBUG:boto:Params: {} 2023-10-17T18:20:30.052 DEBUG:boto:Token: None 2023-10-17T18:20:30.052 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.053 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A996Aa3PfX5bL3jlK9IK/oABGkM= 2023-10-17T18:20:30.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A996Aa3PfX5bL3jlK9IK/oABGkM=', 'Content-Length': '0'} 2023-10-17T18:20:30.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040a10a8516f06255-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:30.056 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.057 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.057 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.057 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.058 DEBUG:boto:Method: GET 2023-10-17T18:20:30.058 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.059 DEBUG:boto:Data: 2023-10-17T18:20:30.059 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.060 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.060 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.060 DEBUG:boto:Params: {} 2023-10-17T18:20:30.061 DEBUG:boto:Token: None 2023-10-17T18:20:30.061 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.062 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s= 2023-10-17T18:20:30.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s=', 'Content-Length': '0'} 2023-10-17T18:20:30.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fdd07b010d44f53-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.106 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.107 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.107 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.108 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.108 DEBUG:boto:Method: GET 2023-10-17T18:20:30.108 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.109 DEBUG:boto:Data: 2023-10-17T18:20:30.109 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.110 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.111 DEBUG:boto:Params: {} 2023-10-17T18:20:30.111 DEBUG:boto:Token: None 2023-10-17T18:20:30.112 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s= 2023-10-17T18:20:30.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s=', 'Content-Length': '0'} 2023-10-17T18:20:30.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000503f25af828e619-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.115 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:30.116 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:30.116 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.117 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.117 DEBUG:boto:Method: GET 2023-10-17T18:20:30.118 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.118 DEBUG:boto:Data: 2023-10-17T18:20:30.119 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.120 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.120 DEBUG:boto:Params: {} 2023-10-17T18:20:30.121 DEBUG:boto:Token: None 2023-10-17T18:20:30.121 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/ 2023-10-17T18:20:30.122 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac= 2023-10-17T18:20:30.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac=', 'Content-Length': '0'} 2023-10-17T18:20:30.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e56f5683cb7809fc-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.125 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.126 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:30.126 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:30.127 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.127 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.128 DEBUG:boto:Method: GET 2023-10-17T18:20:30.128 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.128 DEBUG:boto:Data: 2023-10-17T18:20:30.129 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.129 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.130 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.130 DEBUG:boto:Params: {} 2023-10-17T18:20:30.130 DEBUG:boto:Token: None 2023-10-17T18:20:30.131 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/ 2023-10-17T18:20:30.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac= 2023-10-17T18:20:30.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac=', 'Content-Length': '0'} 2023-10-17T18:20:30.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008efdeb1cc4b82b99-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.135 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.135 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.136 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.136 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.137 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:20:30.137 DEBUG:boto:Data: 2023-10-17T18:20:30.138 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.138 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.139 DEBUG:boto:Params: {} 2023-10-17T18:20:30.139 DEBUG:boto:Token: None 2023-10-17T18:20:30.140 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2 2023-10-17T18:20:30.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YVeKMXO9VrVvwpvEetggKm5I4yU= 2023-10-17T18:20:30.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YVeKMXO9VrVvwpvEetggKm5I4yU=', 'Content-Length': '0'} 2023-10-17T18:20:30.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003261b2928e9f4cd2-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.144 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.144 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.145 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.145 DEBUG:boto:Path: /hbmuog-11/obj-a2 2023-10-17T18:20:30.145 DEBUG:boto:Data: 2023-10-17T18:20:30.146 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.147 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.147 DEBUG:boto:Params: {} 2023-10-17T18:20:30.148 DEBUG:boto:Token: None 2023-10-17T18:20:30.148 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2 2023-10-17T18:20:30.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YVeKMXO9VrVvwpvEetggKm5I4yU= 2023-10-17T18:20:30.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YVeKMXO9VrVvwpvEetggKm5I4yU=', 'Content-Length': '0'} 2023-10-17T18:20:30.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000831d6613f1bdce2a-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:30.152 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.153 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.153 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.154 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.154 DEBUG:boto:Method: GET 2023-10-17T18:20:30.155 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.155 DEBUG:boto:Data: 2023-10-17T18:20:30.155 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.156 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.157 DEBUG:boto:Params: {} 2023-10-17T18:20:30.157 DEBUG:boto:Token: None 2023-10-17T18:20:30.158 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.158 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s= 2023-10-17T18:20:30.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s=', 'Content-Length': '0'} 2023-10-17T18:20:30.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008358c468642c5054-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.161 DEBUG:boto:path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.162 DEBUG:boto:auth_path=/hbmuog-11/obj-a2 2023-10-17T18:20:30.162 DEBUG:boto:path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.163 DEBUG:boto:auth_path=/hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.163 DEBUG:boto:Method: GET 2023-10-17T18:20:30.164 DEBUG:boto:Path: /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.164 DEBUG:boto:Data: 2023-10-17T18:20:30.164 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.165 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.165 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.166 DEBUG:boto:Params: {} 2023-10-17T18:20:30.166 DEBUG:boto:Token: None 2023-10-17T18:20:30.167 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/obj-a2?versionId=30-eD5wAiVUgjCn1taTqrI524fxVgHb 2023-10-17T18:20:30.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s= 2023-10-17T18:20:30.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W0WydsWm+KbEmO7OaCeVAJNrv/s=', 'Content-Length': '0'} 2023-10-17T18:20:30.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', '30-eD5wAiVUgjCn1taTqrI524fxVgHb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007b48d999ea6c46c-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.210 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:30.211 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:30.212 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.212 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.212 DEBUG:boto:Method: GET 2023-10-17T18:20:30.213 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.213 DEBUG:boto:Data: 2023-10-17T18:20:30.214 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.214 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.215 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.215 DEBUG:boto:Params: {} 2023-10-17T18:20:30.215 DEBUG:boto:Token: None 2023-10-17T18:20:30.216 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/ 2023-10-17T18:20:30.216 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac= 2023-10-17T18:20:30.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac=', 'Content-Length': '0'} 2023-10-17T18:20:30.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f60458ad56a9d03-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.220 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.220 DEBUG:boto:path=/hbmuog-11/ 2023-10-17T18:20:30.221 DEBUG:boto:auth_path=/hbmuog-11/ 2023-10-17T18:20:30.221 DEBUG:boto:path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.222 DEBUG:boto:auth_path=/hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.222 DEBUG:boto:Method: GET 2023-10-17T18:20:30.223 DEBUG:boto:Path: /hbmuog-11/?prefix=obj-a2 2023-10-17T18:20:30.223 DEBUG:boto:Data: 2023-10-17T18:20:30.224 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.224 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.225 DEBUG:boto:Params: {} 2023-10-17T18:20:30.225 DEBUG:boto:Token: None 2023-10-17T18:20:30.226 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-11/ 2023-10-17T18:20:30.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac= 2023-10-17T18:20:30.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kDsYxSndjtyIdPmHK4p3MX0R3ac=', 'Content-Length': '0'} 2023-10-17T18:20:30.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72925e2204af61a-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.229 DEBUG:boto:b'hbmuog-11obj-a21000falseobj-a22023-10-17T18:19:12.016Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-11 zones={a2, a1} 2023-10-17T18:20:30.230 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-12 2023-10-17T18:20:30.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:30.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:30.232 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:20:30.291 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:30.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:30.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:30.324+0000 7f604ddf5780 1 bucket sync caught up with source: 2023-10-17T18:20:30.326 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.8.11, , , 00000000002.3.11, 00000000011.31.6, ] 2023-10-17T18:20:30.326 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.8.11, , , 00000000002.3.11, 00000000011.31.6, ] 2023-10-17T18:20:30.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:30.324+0000 7f604ddf5780 0 bucket checkpoint complete 2023-10-17T18:20:30.338 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-12 2023-10-17T18:20:30.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:30.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:30.339 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:20:30.396 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:30.397 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:30.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:30.430+0000 7fc583dd9780 1 bucket sync caught up with source: 2023-10-17T18:20:30.431 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.42.10, , , 00000000011.34.6, 00000000002.45.11, ] 2023-10-17T18:20:30.431 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.42.10, , , 00000000011.34.6, 00000000002.45.11, ] 2023-10-17T18:20:30.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:30.430+0000 7fc583dd9780 0 bucket checkpoint complete 2023-10-17T18:20:30.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-12 zones={a2, a1} 2023-10-17T18:20:30.443 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.444 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.444 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.445 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:20:30.445 DEBUG:boto:Data: 2023-10-17T18:20:30.445 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.446 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.447 DEBUG:boto:Params: {} 2023-10-17T18:20:30.447 DEBUG:boto:Token: None 2023-10-17T18:20:30.447 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8A7ttw070I9DJrgvxC1Pz9n/Rlg= 2023-10-17T18:20:30.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8A7ttw070I9DJrgvxC1Pz9n/Rlg=', 'Content-Length': '0'} 2023-10-17T18:20:30.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a34f6936d7ce01f4-00652ed06e-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.451 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.452 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.452 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.452 DEBUG:boto:Path: /hbmuog-12/ 2023-10-17T18:20:30.453 DEBUG:boto:Data: 2023-10-17T18:20:30.453 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.454 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.454 DEBUG:boto:Params: {} 2023-10-17T18:20:30.455 DEBUG:boto:Token: None 2023-10-17T18:20:30.455 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8A7ttw070I9DJrgvxC1Pz9n/Rlg= 2023-10-17T18:20:30.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8A7ttw070I9DJrgvxC1Pz9n/Rlg=', 'Content-Length': '0'} 2023-10-17T18:20:30.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edc1169688635103-00652ed06e-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.459 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:20:30.459 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.460 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.460 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.461 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.461 DEBUG:boto:Method: GET 2023-10-17T18:20:30.462 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.462 DEBUG:boto:Data: 2023-10-17T18:20:30.462 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.463 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.464 DEBUG:boto:Params: {} 2023-10-17T18:20:30.464 DEBUG:boto:Token: None 2023-10-17T18:20:30.465 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/?versions 2023-10-17T18:20:30.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8= 2023-10-17T18:20:30.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8=', 'Content-Length': '0'} 2023-10-17T18:20:30.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c706dd140b45f11-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.468 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.469 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:20:30.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:20:30.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:20:30.470 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:20:30.471 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.471 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.472 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.472 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.473 DEBUG:boto:Method: GET 2023-10-17T18:20:30.473 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.473 DEBUG:boto:Data: 2023-10-17T18:20:30.474 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.474 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.475 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.475 DEBUG:boto:Params: {} 2023-10-17T18:20:30.475 DEBUG:boto:Token: None 2023-10-17T18:20:30.476 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/?versions 2023-10-17T18:20:30.476 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8= 2023-10-17T18:20:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8=', 'Content-Length': '0'} 2023-10-17T18:20:30.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe4d47350dce640-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.480 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.480 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:20:30.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-17T18:20:30.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-17T18:20:30.481 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.482 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.482 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.483 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.483 DEBUG:boto:Method: GET 2023-10-17T18:20:30.483 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.484 DEBUG:boto:Data: 2023-10-17T18:20:30.484 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.485 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.485 DEBUG:boto:Params: {} 2023-10-17T18:20:30.486 DEBUG:boto:Token: None 2023-10-17T18:20:30.486 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/?versions 2023-10-17T18:20:30.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8= 2023-10-17T18:20:30.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8=', 'Content-Length': '0'} 2023-10-17T18:20:30.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b8a5aa9c298a64-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.490 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.491 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.491 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.491 DEBUG:boto:path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.492 DEBUG:boto:auth_path=/hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.492 DEBUG:boto:Method: GET 2023-10-17T18:20:30.493 DEBUG:boto:Path: /hbmuog-12/?versions&max-keys=999 2023-10-17T18:20:30.493 DEBUG:boto:Data: 2023-10-17T18:20:30.493 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.494 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.494 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.495 DEBUG:boto:Params: {} 2023-10-17T18:20:30.495 DEBUG:boto:Token: None 2023-10-17T18:20:30.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/?versions 2023-10-17T18:20:30.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8= 2023-10-17T18:20:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WziCOEvyf4v73mztcYkwbgaVQq8=', 'Content-Length': '0'} 2023-10-17T18:20:30.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e67aa147632df1-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.499 DEBUG:boto:b'hbmuog-12999falsedummy8gRsS0ltVjkXZWoOxLiDKUaAwXmibNDtrue2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1clr6stDijy1J.3Xn8QwlLlUbest3f3Itrue2023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2esajST8EdHFfu02nUN6XL5w.kLwvifwtrue2023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:30.500 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.501 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.501 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.502 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.502 DEBUG:boto:Method: GET 2023-10-17T18:20:30.502 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.503 DEBUG:boto:Data: 2023-10-17T18:20:30.503 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.504 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.505 DEBUG:boto:Params: {} 2023-10-17T18:20:30.505 DEBUG:boto:Token: None 2023-10-17T18:20:30.506 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.506 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036db09d9f15c2294-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.509 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.509 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.510 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.510 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.511 DEBUG:boto:Method: GET 2023-10-17T18:20:30.511 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.512 DEBUG:boto:Data: 2023-10-17T18:20:30.512 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.513 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.513 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.513 DEBUG:boto:Params: {} 2023-10-17T18:20:30.514 DEBUG:boto:Token: None 2023-10-17T18:20:30.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d7fefaa7ea1f3e1-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.518 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.518 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.519 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.519 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.519 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.520 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.520 DEBUG:boto:Data: 2023-10-17T18:20:30.521 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.522 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.522 DEBUG:boto:Params: {} 2023-10-17T18:20:30.523 DEBUG:boto:Token: None 2023-10-17T18:20:30.523 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BhYjsBYKshJjCmIyK/bQ4G5CpqQ= 2023-10-17T18:20:30.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BhYjsBYKshJjCmIyK/bQ4G5CpqQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000875160dbc218a84a-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.526 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.527 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.527 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.527 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.528 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.528 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.528 DEBUG:boto:Data: 2023-10-17T18:20:30.529 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.529 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.530 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.530 DEBUG:boto:Params: {} 2023-10-17T18:20:30.531 DEBUG:boto:Token: None 2023-10-17T18:20:30.531 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BhYjsBYKshJjCmIyK/bQ4G5CpqQ= 2023-10-17T18:20:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BhYjsBYKshJjCmIyK/bQ4G5CpqQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069f5e8f1f74ae0e2-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:30.535 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.535 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.535 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.536 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.536 DEBUG:boto:Method: GET 2023-10-17T18:20:30.537 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.537 DEBUG:boto:Data: 2023-10-17T18:20:30.538 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.538 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.539 DEBUG:boto:Params: {} 2023-10-17T18:20:30.539 DEBUG:boto:Token: None 2023-10-17T18:20:30.540 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e640c4bfb38dffa-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.543 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.543 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.544 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.544 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.545 DEBUG:boto:Method: GET 2023-10-17T18:20:30.545 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.545 DEBUG:boto:Data: 2023-10-17T18:20:30.546 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.547 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.547 DEBUG:boto:Params: {} 2023-10-17T18:20:30.548 DEBUG:boto:Token: None 2023-10-17T18:20:30.548 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003f98b34d85e6066-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.551 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.552 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.552 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.553 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.553 DEBUG:boto:Method: GET 2023-10-17T18:20:30.553 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.554 DEBUG:boto:Data: 2023-10-17T18:20:30.554 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.555 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.555 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.556 DEBUG:boto:Params: {} 2023-10-17T18:20:30.556 DEBUG:boto:Token: None 2023-10-17T18:20:30.557 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.557 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa136fb41bca21ac-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.602 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.603 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.603 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.604 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.604 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.605 DEBUG:boto:Method: GET 2023-10-17T18:20:30.605 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.606 DEBUG:boto:Data: 2023-10-17T18:20:30.606 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.607 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.607 DEBUG:boto:Params: {} 2023-10-17T18:20:30.608 DEBUG:boto:Token: None 2023-10-17T18:20:30.608 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc385098b988657b-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.612 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.613 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.613 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.613 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.614 DEBUG:boto:Path: /hbmuog-12/dummy 2023-10-17T18:20:30.614 DEBUG:boto:Data: 2023-10-17T18:20:30.615 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.615 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.616 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.616 DEBUG:boto:Params: {} 2023-10-17T18:20:30.616 DEBUG:boto:Token: None 2023-10-17T18:20:30.617 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy 2023-10-17T18:20:30.617 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4UHpIm8j4tN+w0N21t6kBKiRtxA= 2023-10-17T18:20:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4UHpIm8j4tN+w0N21t6kBKiRtxA=', 'Content-Length': '0'} 2023-10-17T18:20:30.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aefa08ad86ba0b52-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.620 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.621 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.621 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.621 DEBUG:boto:Path: /hbmuog-12/dummy 2023-10-17T18:20:30.622 DEBUG:boto:Data: 2023-10-17T18:20:30.622 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.623 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.624 DEBUG:boto:Params: {} 2023-10-17T18:20:30.624 DEBUG:boto:Token: None 2023-10-17T18:20:30.625 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy 2023-10-17T18:20:30.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4UHpIm8j4tN+w0N21t6kBKiRtxA= 2023-10-17T18:20:30.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4UHpIm8j4tN+w0N21t6kBKiRtxA=', 'Content-Length': '0'} 2023-10-17T18:20:30.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000795c55855475cdd1-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:20:30.630 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.631 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.631 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.632 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.632 DEBUG:boto:Method: GET 2023-10-17T18:20:30.633 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.633 DEBUG:boto:Data: 2023-10-17T18:20:30.633 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.634 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.635 DEBUG:boto:Params: {} 2023-10-17T18:20:30.635 DEBUG:boto:Token: None 2023-10-17T18:20:30.636 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.636 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fa8bd1f3f8f94cb-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.639 DEBUG:boto:path=/hbmuog-12/dummy 2023-10-17T18:20:30.640 DEBUG:boto:auth_path=/hbmuog-12/dummy 2023-10-17T18:20:30.640 DEBUG:boto:path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.641 DEBUG:boto:auth_path=/hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.641 DEBUG:boto:Method: GET 2023-10-17T18:20:30.642 DEBUG:boto:Path: /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.642 DEBUG:boto:Data: 2023-10-17T18:20:30.643 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.643 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.644 DEBUG:boto:Params: {} 2023-10-17T18:20:30.644 DEBUG:boto:Token: None 2023-10-17T18:20:30.645 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/dummy?versionId=8gRsS0ltVjkXZWoOxLiDKUaAwXmibND 2023-10-17T18:20:30.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4= 2023-10-17T18:20:30.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7n6yfgzGPX6QkKKACAwgUU4RcA4=', 'Content-Length': '0'} 2023-10-17T18:20:30.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:18:51 GMT'), ('x-amz-version-id', '8gRsS0ltVjkXZWoOxLiDKUaAwXmibND'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083fed9f59135f466-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.648 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.649 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.649 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.650 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.650 DEBUG:boto:Method: GET 2023-10-17T18:20:30.650 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.651 DEBUG:boto:Data: 2023-10-17T18:20:30.651 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.652 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.652 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.653 DEBUG:boto:Params: {} 2023-10-17T18:20:30.653 DEBUG:boto:Token: None 2023-10-17T18:20:30.654 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.654 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000d8da2cb65418e1-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.658 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.658 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.659 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.659 DEBUG:boto:path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.660 DEBUG:boto:auth_path=/hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.660 DEBUG:boto:Method: GET 2023-10-17T18:20:30.661 DEBUG:boto:Path: /hbmuog-12/?prefix=dummy 2023-10-17T18:20:30.661 DEBUG:boto:Data: 2023-10-17T18:20:30.661 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.662 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.663 DEBUG:boto:Params: {} 2023-10-17T18:20:30.663 DEBUG:boto:Token: None 2023-10-17T18:20:30.664 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000134d1e4d0acc7eb4-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.667 DEBUG:boto:b'hbmuog-12dummy1000falsedummy2023-10-17T18:18:51.741Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:30.668 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.669 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.669 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.670 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.670 DEBUG:boto:Method: GET 2023-10-17T18:20:30.671 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.671 DEBUG:boto:Data: 2023-10-17T18:20:30.671 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.672 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.672 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.673 DEBUG:boto:Params: {} 2023-10-17T18:20:30.673 DEBUG:boto:Token: None 2023-10-17T18:20:30.674 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.674 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004270447ba78048f1-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.677 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.677 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.678 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.678 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.679 DEBUG:boto:Method: GET 2023-10-17T18:20:30.679 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.680 DEBUG:boto:Data: 2023-10-17T18:20:30.680 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.681 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.681 DEBUG:boto:Params: {} 2023-10-17T18:20:30.682 DEBUG:boto:Token: None 2023-10-17T18:20:30.682 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000145145c476320eca-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.686 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.686 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.687 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.687 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.687 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.688 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.688 DEBUG:boto:Data: 2023-10-17T18:20:30.689 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.689 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.690 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.690 DEBUG:boto:Params: {} 2023-10-17T18:20:30.690 DEBUG:boto:Token: None 2023-10-17T18:20:30.691 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.691 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HYubGMTt6zwpVOpkeDsaIHBxjyY= 2023-10-17T18:20:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HYubGMTt6zwpVOpkeDsaIHBxjyY=', 'Content-Length': '0'} 2023-10-17T18:20:30.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfd84af4a2e3dbcb-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.694 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.695 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.695 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.695 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.696 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.696 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.697 DEBUG:boto:Data: 2023-10-17T18:20:30.697 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.698 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.698 DEBUG:boto:Params: {} 2023-10-17T18:20:30.699 DEBUG:boto:Token: None 2023-10-17T18:20:30.699 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HYubGMTt6zwpVOpkeDsaIHBxjyY= 2023-10-17T18:20:30.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HYubGMTt6zwpVOpkeDsaIHBxjyY=', 'Content-Length': '0'} 2023-10-17T18:20:30.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d897af9bbab905c0-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:30.703 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.704 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.704 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.705 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.705 DEBUG:boto:Method: GET 2023-10-17T18:20:30.705 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.706 DEBUG:boto:Data: 2023-10-17T18:20:30.706 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.707 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.708 DEBUG:boto:Params: {} 2023-10-17T18:20:30.708 DEBUG:boto:Token: None 2023-10-17T18:20:30.709 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010d31d6a65145331-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.712 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.713 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.713 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.713 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.714 DEBUG:boto:Method: GET 2023-10-17T18:20:30.714 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.715 DEBUG:boto:Data: 2023-10-17T18:20:30.715 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.716 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.717 DEBUG:boto:Params: {} 2023-10-17T18:20:30.717 DEBUG:boto:Token: None 2023-10-17T18:20:30.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.718 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001016c973cface9af-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.721 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.721 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.722 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.722 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.722 DEBUG:boto:Method: GET 2023-10-17T18:20:30.723 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.723 DEBUG:boto:Data: 2023-10-17T18:20:30.724 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.725 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.725 DEBUG:boto:Params: {} 2023-10-17T18:20:30.725 DEBUG:boto:Token: None 2023-10-17T18:20:30.726 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.726 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7bb20771dd0536-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.729 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.730 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.731 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.731 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.731 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.732 DEBUG:boto:Method: GET 2023-10-17T18:20:30.732 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.733 DEBUG:boto:Data: 2023-10-17T18:20:30.733 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.734 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.734 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.735 DEBUG:boto:Params: {} 2023-10-17T18:20:30.735 DEBUG:boto:Token: None 2023-10-17T18:20:30.736 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003364fee709a29bc9-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.739 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.740 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.740 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.741 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.741 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:20:30.741 DEBUG:boto:Data: 2023-10-17T18:20:30.742 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.743 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.743 DEBUG:boto:Params: {} 2023-10-17T18:20:30.744 DEBUG:boto:Token: None 2023-10-17T18:20:30.744 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1 2023-10-17T18:20:30.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pRJBYAkixCE+IjK0LkWYwCB/D78= 2023-10-17T18:20:30.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pRJBYAkixCE+IjK0LkWYwCB/D78=', 'Content-Length': '0'} 2023-10-17T18:20:30.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8179a182aaf8c8b-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.748 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.748 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.749 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.749 DEBUG:boto:Path: /hbmuog-12/obj-a1 2023-10-17T18:20:30.749 DEBUG:boto:Data: 2023-10-17T18:20:30.750 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.751 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.751 DEBUG:boto:Params: {} 2023-10-17T18:20:30.752 DEBUG:boto:Token: None 2023-10-17T18:20:30.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1 2023-10-17T18:20:30.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pRJBYAkixCE+IjK0LkWYwCB/D78= 2023-10-17T18:20:30.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pRJBYAkixCE+IjK0LkWYwCB/D78=', 'Content-Length': '0'} 2023-10-17T18:20:30.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000430139f17f029fa2-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-17T18:20:30.756 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.756 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.757 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.757 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.758 DEBUG:boto:Method: GET 2023-10-17T18:20:30.758 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.758 DEBUG:boto:Data: 2023-10-17T18:20:30.759 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.760 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.760 DEBUG:boto:Params: {} 2023-10-17T18:20:30.761 DEBUG:boto:Token: None 2023-10-17T18:20:30.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000797ce7ba08b0ee9f-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.806 DEBUG:boto:path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.807 DEBUG:boto:auth_path=/hbmuog-12/obj-a1 2023-10-17T18:20:30.807 DEBUG:boto:path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.808 DEBUG:boto:auth_path=/hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.808 DEBUG:boto:Method: GET 2023-10-17T18:20:30.808 DEBUG:boto:Path: /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.809 DEBUG:boto:Data: 2023-10-17T18:20:30.809 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.810 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.811 DEBUG:boto:Params: {} 2023-10-17T18:20:30.811 DEBUG:boto:Token: None 2023-10-17T18:20:30.812 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a1?versionId=clr6stDijy1J.3Xn8QwlLlUbest3f3I 2023-10-17T18:20:30.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww= 2023-10-17T18:20:30.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w+jNHpTZcLB7dl9R/QqXcSlQCww=', 'Content-Length': '0'} 2023-10-17T18:20:30.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'clr6stDijy1J.3Xn8QwlLlUbest3f3I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071a5b61d26193832-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.815 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.816 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.816 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.816 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.817 DEBUG:boto:Method: GET 2023-10-17T18:20:30.817 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.818 DEBUG:boto:Data: 2023-10-17T18:20:30.818 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.819 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.819 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.819 DEBUG:boto:Params: {} 2023-10-17T18:20:30.820 DEBUG:boto:Token: None 2023-10-17T18:20:30.820 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f2ff0d64c35545-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.824 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.825 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.825 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.826 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.826 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.826 DEBUG:boto:Method: GET 2023-10-17T18:20:30.827 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a1 2023-10-17T18:20:30.827 DEBUG:boto:Data: 2023-10-17T18:20:30.828 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.828 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.828 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.829 DEBUG:boto:Params: {} 2023-10-17T18:20:30.829 DEBUG:boto:Token: None 2023-10-17T18:20:30.830 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049276b8e517012ad-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.836 DEBUG:boto:b'hbmuog-12obj-a11000falseobj-a12023-10-17T18:19:12.139Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:30.838 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.838 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.839 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.839 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.839 DEBUG:boto:Method: GET 2023-10-17T18:20:30.840 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.840 DEBUG:boto:Data: 2023-10-17T18:20:30.841 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.842 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.842 DEBUG:boto:Params: {} 2023-10-17T18:20:30.843 DEBUG:boto:Token: None 2023-10-17T18:20:30.843 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f6f21655438f682-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.847 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.847 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.848 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.848 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.849 DEBUG:boto:Method: GET 2023-10-17T18:20:30.849 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.850 DEBUG:boto:Data: 2023-10-17T18:20:30.850 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.851 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.851 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.851 DEBUG:boto:Params: {} 2023-10-17T18:20:30.852 DEBUG:boto:Token: None 2023-10-17T18:20:30.852 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.853 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003954df7a80e50703-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.856 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.856 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.856 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.857 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.857 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.858 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.858 DEBUG:boto:Data: 2023-10-17T18:20:30.858 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.859 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.859 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.860 DEBUG:boto:Params: {} 2023-10-17T18:20:30.860 DEBUG:boto:Token: None 2023-10-17T18:20:30.861 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A9d5sxm8NrMSEN96rzVdwDlA/KE= 2023-10-17T18:20:30.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A9d5sxm8NrMSEN96rzVdwDlA/KE=', 'Content-Length': '0'} 2023-10-17T18:20:30.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000469551e25011a052-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.864 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.864 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.865 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.865 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.865 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.866 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.866 DEBUG:boto:Data: 2023-10-17T18:20:30.867 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.867 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.868 DEBUG:boto:Params: {} 2023-10-17T18:20:30.868 DEBUG:boto:Token: None 2023-10-17T18:20:30.869 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A9d5sxm8NrMSEN96rzVdwDlA/KE= 2023-10-17T18:20:30.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A9d5sxm8NrMSEN96rzVdwDlA/KE=', 'Content-Length': '0'} 2023-10-17T18:20:30.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc287610d820463a-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:30.872 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.873 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.873 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.874 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.874 DEBUG:boto:Method: GET 2023-10-17T18:20:30.874 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.875 DEBUG:boto:Data: 2023-10-17T18:20:30.875 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.876 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.876 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.876 DEBUG:boto:Params: {} 2023-10-17T18:20:30.877 DEBUG:boto:Token: None 2023-10-17T18:20:30.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc1cae590d84bede-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.926 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.927 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.927 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.928 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.928 DEBUG:boto:Method: GET 2023-10-17T18:20:30.929 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.929 DEBUG:boto:Data: 2023-10-17T18:20:30.929 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.930 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.931 DEBUG:boto:Params: {} 2023-10-17T18:20:30.931 DEBUG:boto:Token: None 2023-10-17T18:20:30.932 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c4be65bc4d37255-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.935 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.936 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.936 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.936 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.937 DEBUG:boto:Method: GET 2023-10-17T18:20:30.937 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.938 DEBUG:boto:Data: 2023-10-17T18:20:30.938 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.939 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.940 DEBUG:boto:Params: {} 2023-10-17T18:20:30.940 DEBUG:boto:Token: None 2023-10-17T18:20:30.940 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0cb3900f8ed745b-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.944 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.945 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.946 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.946 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.946 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.947 DEBUG:boto:Method: GET 2023-10-17T18:20:30.947 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.948 DEBUG:boto:Data: 2023-10-17T18:20:30.948 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.949 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.949 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.950 DEBUG:boto:Params: {} 2023-10-17T18:20:30.950 DEBUG:boto:Token: None 2023-10-17T18:20:30.951 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.951 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd89e406998ef2c1-00652ed06e-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.954 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:30.955 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.955 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.956 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.956 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:20:30.957 DEBUG:boto:Data: 2023-10-17T18:20:30.957 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.958 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.958 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.958 DEBUG:boto:Params: {} 2023-10-17T18:20:30.959 DEBUG:boto:Token: None 2023-10-17T18:20:30.959 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2 2023-10-17T18:20:30.960 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4vnYqFB2J3eLmLQdjAsaO9rrSKc= 2023-10-17T18:20:30.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4vnYqFB2J3eLmLQdjAsaO9rrSKc=', 'Content-Length': '0'} 2023-10-17T18:20:30.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a02782868352a30-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.963 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.963 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.964 DEBUG:boto:Method: HEAD 2023-10-17T18:20:30.964 DEBUG:boto:Path: /hbmuog-12/obj-a2 2023-10-17T18:20:30.965 DEBUG:boto:Data: 2023-10-17T18:20:30.965 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.966 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.966 DEBUG:boto:Params: {} 2023-10-17T18:20:30.967 DEBUG:boto:Token: None 2023-10-17T18:20:30.967 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2 2023-10-17T18:20:30.968 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4vnYqFB2J3eLmLQdjAsaO9rrSKc= 2023-10-17T18:20:30.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4vnYqFB2J3eLmLQdjAsaO9rrSKc=', 'Content-Length': '0'} 2023-10-17T18:20:30.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43734d2380076ed-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-17T18:20:30.971 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.971 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.972 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.972 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.973 DEBUG:boto:Method: GET 2023-10-17T18:20:30.973 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.974 DEBUG:boto:Data: 2023-10-17T18:20:30.974 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.974 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.975 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.975 DEBUG:boto:Params: {} 2023-10-17T18:20:30.976 DEBUG:boto:Token: None 2023-10-17T18:20:30.976 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.977 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d718e17fe8bb790-00652ed06e-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.980 DEBUG:boto:path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.980 DEBUG:boto:auth_path=/hbmuog-12/obj-a2 2023-10-17T18:20:30.980 DEBUG:boto:path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.981 DEBUG:boto:auth_path=/hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.981 DEBUG:boto:Method: GET 2023-10-17T18:20:30.982 DEBUG:boto:Path: /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.982 DEBUG:boto:Data: 2023-10-17T18:20:30.983 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.983 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:30.983 DEBUG:boto:Port: 8000 2023-10-17T18:20:30.984 DEBUG:boto:Params: {} 2023-10-17T18:20:30.984 DEBUG:boto:Token: None 2023-10-17T18:20:30.985 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/obj-a2?versionId=esajST8EdHFfu02nUN6XL5w.kLwvifw 2023-10-17T18:20:30.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ= 2023-10-17T18:20:30.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oAe211Kh6HRyDsmHdyKg8f2xeoQ=', 'Content-Length': '0'} 2023-10-17T18:20:30.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:19:12 GMT'), ('x-amz-version-id', 'esajST8EdHFfu02nUN6XL5w.kLwvifw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d2d2b792c586f8-00652ed06e-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:30.988 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:30.989 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:30.989 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.989 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.990 DEBUG:boto:Method: GET 2023-10-17T18:20:30.990 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:30.991 DEBUG:boto:Data: 2023-10-17T18:20:30.991 DEBUG:boto:Headers: {} 2023-10-17T18:20:30.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:20:30.992 DEBUG:boto:Port: 8001 2023-10-17T18:20:30.993 DEBUG:boto:Params: {} 2023-10-17T18:20:30.993 DEBUG:boto:Token: None 2023-10-17T18:20:30.994 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:30 GMT /hbmuog-12/ 2023-10-17T18:20:30.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0= 2023-10-17T18:20:30.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2fDNm5tAspFF20dDPg0TfNsqce0=', 'Content-Length': '0'} 2023-10-17T18:20:30.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004616f57b824276b0-00652ed06e-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.038 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:31.039 DEBUG:boto:path=/hbmuog-12/ 2023-10-17T18:20:31.039 DEBUG:boto:auth_path=/hbmuog-12/ 2023-10-17T18:20:31.040 DEBUG:boto:path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:31.040 DEBUG:boto:auth_path=/hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:31.041 DEBUG:boto:Method: GET 2023-10-17T18:20:31.041 DEBUG:boto:Path: /hbmuog-12/?prefix=obj-a2 2023-10-17T18:20:31.042 DEBUG:boto:Data: 2023-10-17T18:20:31.042 DEBUG:boto:Headers: {} 2023-10-17T18:20:31.042 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:31.043 DEBUG:boto:Port: 8000 2023-10-17T18:20:31.043 DEBUG:boto:Params: {} 2023-10-17T18:20:31.044 DEBUG:boto:Token: None 2023-10-17T18:20:31.044 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:20:31 GMT /hbmuog-12/ 2023-10-17T18:20:31.045 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YrNmrODpWTse/pYWoQrbA+JifIU= 2023-10-17T18:20:31.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YrNmrODpWTse/pYWoQrbA+JifIU=', 'Content-Length': '0'} 2023-10-17T18:20:31.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b3bdc020106e2f-00652ed06f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.048 DEBUG:boto:b'hbmuog-12obj-a21000falseobj-a22023-10-17T18:19:12.259Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:20:31.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-12 zones={a2, a1} 2023-10-17T18:20:31.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-10-17T18:20:31.051 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:20:31.051 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:20:31.052 DEBUG:boto:Method: PUT 2023-10-17T18:20:31.052 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:20:31.052 DEBUG:boto:Data: 2023-10-17T18:20:31.053 DEBUG:boto:Headers: {} 2023-10-17T18:20:31.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:31.054 DEBUG:boto:Port: 8000 2023-10-17T18:20:31.054 DEBUG:boto:Params: {} 2023-10-17T18:20:31.054 DEBUG:boto:Token: None 2023-10-17T18:20:31.055 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:20:31 GMT /hbmuog-13/ 2023-10-17T18:20:31.055 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eOsFhDOB2MJiwWpy3NnzkLgFUog= 2023-10-17T18:20:31.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eOsFhDOB2MJiwWpy3NnzkLgFUog=', 'Content-Length': '0'} 2023-10-17T18:20:31.068 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027db6c14cdbc679a-00652ed06f-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.068 DEBUG:rgw_multi.tests:created bucket=hbmuog-13 2023-10-17T18:20:31.069 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:20:31.069 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:20:31.070 DEBUG:boto:path=/hbmuog-13/?versioning 2023-10-17T18:20:31.070 DEBUG:boto:auth_path=/hbmuog-13/?versioning 2023-10-17T18:20:31.071 DEBUG:boto:Method: PUT 2023-10-17T18:20:31.071 DEBUG:boto:Path: /hbmuog-13/?versioning 2023-10-17T18:20:31.071 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:20:31.072 DEBUG:boto:Headers: {} 2023-10-17T18:20:31.072 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:31.073 DEBUG:boto:Port: 8000 2023-10-17T18:20:31.073 DEBUG:boto:Params: {} 2023-10-17T18:20:31.073 DEBUG:boto:Token: None 2023-10-17T18:20:31.074 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:20:31 GMT /hbmuog-13/?versioning 2023-10-17T18:20:31.074 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nsnLHKMh13PIYapWGN3Xe0OcGtY= 2023-10-17T18:20:31.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nsnLHKMh13PIYapWGN3Xe0OcGtY=', 'Content-Length': '228'} 2023-10-17T18:20:31.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009a8c896596f1538e-00652ed06f-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:20:31.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:20:31.081 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:20:31.134 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:31.134 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:31.168 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:20:31.179 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:20:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566831.079522_166.1", 2023-10-17T18:20:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:20:31.079522Z" 2023-10-17T18:20:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.181 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:20:31.181 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:20:31.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.181 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:20:31.182 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:20:31.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.182 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:20:31.182 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:20:31.183 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.183 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:20:31.184 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566831.079522_166.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:20:31.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:20:31.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:20:31.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:20:31.186 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:20:31.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:31.237 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:31.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:31.264+0000 7f2ad0200780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:20:31.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:31.265+0000 7f2ad0200780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:20:31.277 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:20:31.278 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:20:31.278 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:20:31.278 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:20:31.278 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:20:31.278 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:20:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:20:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:20:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:20:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:20:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:20:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566831.079522_166.1", 2023-10-17T18:20:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:20:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:20:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:20:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:31.079522Z", 2023-10-17T18:20:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:20:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:20:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:20:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:20:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:20:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:20:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:20:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:20:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:20:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:20:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:20:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:20:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:20:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:20:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:20:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:20:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:20:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:20:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:20:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:20:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:20:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:20:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:20:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:20:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:20:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:20:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:20:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:20:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:20:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:20:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:20:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:20:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:20:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:20:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:20:31.292 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:20:31Z" 2023-10-17T18:20:31.292 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:20:31.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566831.079522_166.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:20:31.079522Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:20:31Z" } 2023-10-17T18:20:31.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566831.079522_166.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:20:31.079522Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:20:31.294 DEBUG:rgw_multi.tests:log_status={0: '1_1697566831.079522_166.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:20:31.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566831.079522_166.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:20:31.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:20:31.295 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:20:31.295 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:20:31.296 DEBUG:boto:Method: HEAD 2023-10-17T18:20:31.296 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:20:31.297 DEBUG:boto:Data: 2023-10-17T18:20:31.297 DEBUG:boto:Headers: {} 2023-10-17T18:20:31.297 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:31.298 DEBUG:boto:Port: 8000 2023-10-17T18:20:31.298 DEBUG:boto:Params: {} 2023-10-17T18:20:31.299 DEBUG:boto:Token: None 2023-10-17T18:20:31.299 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:20:31 GMT /hbmuog-13/ 2023-10-17T18:20:31.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vF2kVNV0fs/YEGZqtE608FE1ru0= 2023-10-17T18:20:31.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:20:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vF2kVNV0fs/YEGZqtE608FE1ru0=', 'Content-Length': '0'} 2023-10-17T18:20:31.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020aea153cf090174-00652ed06f-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.303 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:20:31.304 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:20:31.304 DEBUG:boto:Method: PUT 2023-10-17T18:20:31.305 DEBUG:boto:Path: /hbmuog-13/dummy 2023-10-17T18:20:31.305 DEBUG:boto:Data: 2023-10-17T18:20:31.305 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:20:31.306 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:20:31.306 DEBUG:boto:Port: 8000 2023-10-17T18:20:31.307 DEBUG:boto:Params: {} 2023-10-17T18:20:31.307 DEBUG:boto:Token: None 2023-10-17T18:20:31.308 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:20:31 GMT /hbmuog-13/dummy 2023-10-17T18:20:31.308 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JavbLQuOmbHKGNxCWQFgTrx0wpM= 2023-10-17T18:20:31.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:20:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JavbLQuOmbHKGNxCWQFgTrx0wpM='} 2023-10-17T18:20:31.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-amz-request-id', 'tx000008da80b28704b500d-00652ed06f-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:20:31.319 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-13 2023-10-17T18:20:31.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:31.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:31.320 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:20:31.377 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:31.378 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:31.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:31.411+0000 7fb2a2401780 1 waiting to reach incremental sync.. 2023-10-17T18:20:36.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:36.412+0000 7fb2a2401780 1 waiting to reach incremental sync.. 2023-10-17T18:20:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:41.413+0000 7fb2a2401780 1 waiting to reach incremental sync.. 2023-10-17T18:20:46.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:46.413+0000 7fb2a2401780 1 waiting to reach incremental sync.. 2023-10-17T18:20:51.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:51.414+0000 7fb2a2401780 1 bucket sync caught up with source: 2023-10-17T18:20:51.416 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:20:51.416 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:20:51.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:51.414+0000 7fb2a2401780 0 bucket checkpoint complete 2023-10-17T18:20:51.432 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-13 2023-10-17T18:20:51.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:51.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:20:51.434 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:20:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:20:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:20:51.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:51.534+0000 7f20d6c59780 1 waiting to reach incremental sync.. 2023-10-17T18:20:56.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:20:56.535+0000 7f20d6c59780 1 waiting to reach incremental sync.. 2023-10-17T18:21:01.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:01.535+0000 7f20d6c59780 1 waiting to reach incremental sync.. 2023-10-17T18:21:06.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:06.536+0000 7f20d6c59780 1 waiting to reach incremental sync.. 2023-10-17T18:21:11.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:11.538+0000 7f20d6c59780 1 bucket sync caught up with source: 2023-10-17T18:21:11.539 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , , , ] 2023-10-17T18:21:11.540 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , , , ] 2023-10-17T18:21:11.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:11.538+0000 7f20d6c59780 0 bucket checkpoint complete 2023-10-17T18:21:11.556 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-13 zones={a2, a1} 2023-10-17T18:21:11.557 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.557 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.558 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.558 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.558 DEBUG:boto:Data: 2023-10-17T18:21:11.559 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.560 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.560 DEBUG:boto:Params: {} 2023-10-17T18:21:11.561 DEBUG:boto:Token: None 2023-10-17T18:21:11.561 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c20687476d1fc707-00652ed097-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.566 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.566 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.567 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.567 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.568 DEBUG:boto:Data: 2023-10-17T18:21:11.568 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.569 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.569 DEBUG:boto:Params: {} 2023-10-17T18:21:11.570 DEBUG:boto:Token: None 2023-10-17T18:21:11.570 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb40760a9d3e8ee-00652ed097-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.575 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:21:11.576 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.576 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.577 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.577 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.578 DEBUG:boto:Method: GET 2023-10-17T18:21:11.578 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.579 DEBUG:boto:Data: 2023-10-17T18:21:11.579 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.580 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.580 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.580 DEBUG:boto:Params: {} 2023-10-17T18:21:11.581 DEBUG:boto:Token: None 2023-10-17T18:21:11.581 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/?versions 2023-10-17T18:21:11.582 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ= 2023-10-17T18:21:11.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ=', 'Content-Length': '0'} 2023-10-17T18:21:11.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023aab17828285c91-00652ed097-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.586 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.586 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:21:11.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:21:11.587 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.588 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.588 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.589 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.589 DEBUG:boto:Method: GET 2023-10-17T18:21:11.589 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.590 DEBUG:boto:Data: 2023-10-17T18:21:11.590 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.591 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.593 DEBUG:boto:Params: {} 2023-10-17T18:21:11.593 DEBUG:boto:Token: None 2023-10-17T18:21:11.594 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/?versions 2023-10-17T18:21:11.594 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ= 2023-10-17T18:21:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ=', 'Content-Length': '0'} 2023-10-17T18:21:11.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001be8b728a6ff4b21-00652ed097-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.599 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.600 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:21:11.600 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.601 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.601 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.602 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.602 DEBUG:boto:Method: GET 2023-10-17T18:21:11.603 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.603 DEBUG:boto:Data: 2023-10-17T18:21:11.604 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.604 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.604 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.605 DEBUG:boto:Params: {} 2023-10-17T18:21:11.605 DEBUG:boto:Token: None 2023-10-17T18:21:11.606 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/?versions 2023-10-17T18:21:11.606 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ= 2023-10-17T18:21:11.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ=', 'Content-Length': '0'} 2023-10-17T18:21:11.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7b18c5f743d35b-00652ed097-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.610 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.610 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.611 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.611 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.611 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.612 DEBUG:boto:Method: GET 2023-10-17T18:21:11.612 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:21:11.613 DEBUG:boto:Data: 2023-10-17T18:21:11.613 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.614 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.614 DEBUG:boto:Params: {} 2023-10-17T18:21:11.615 DEBUG:boto:Token: None 2023-10-17T18:21:11.615 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/?versions 2023-10-17T18:21:11.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ= 2023-10-17T18:21:11.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3i6iuhi6BzqZ6J71t4OjryIV7lQ=', 'Content-Length': '0'} 2023-10-17T18:21:11.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076af35e0f7397805-00652ed097-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.662 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:21:11.664 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.664 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.665 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.665 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.665 DEBUG:boto:Method: GET 2023-10-17T18:21:11.666 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.666 DEBUG:boto:Data: 2023-10-17T18:21:11.667 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.667 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.668 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.668 DEBUG:boto:Params: {} 2023-10-17T18:21:11.669 DEBUG:boto:Token: None 2023-10-17T18:21:11.669 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.670 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000969300a0bfd4c31-00652ed097-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.673 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.674 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.674 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.675 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.675 DEBUG:boto:Method: GET 2023-10-17T18:21:11.676 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.676 DEBUG:boto:Data: 2023-10-17T18:21:11.676 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.677 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.678 DEBUG:boto:Params: {} 2023-10-17T18:21:11.678 DEBUG:boto:Token: None 2023-10-17T18:21:11.679 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.679 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006267bdef082fe144-00652ed097-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.682 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.683 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.683 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.684 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.684 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.684 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.685 DEBUG:boto:Data: 2023-10-17T18:21:11.685 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.686 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.687 DEBUG:boto:Params: {} 2023-10-17T18:21:11.687 DEBUG:boto:Token: None 2023-10-17T18:21:11.688 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.688 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:11/OemPpHtLG3PJHeDETM+wprrE= 2023-10-17T18:21:11.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:11/OemPpHtLG3PJHeDETM+wprrE=', 'Content-Length': '0'} 2023-10-17T18:21:11.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f020eaa841c2d13-00652ed097-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.691 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.692 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.692 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.693 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.693 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.693 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.694 DEBUG:boto:Data: 2023-10-17T18:21:11.694 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.695 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.695 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.696 DEBUG:boto:Params: {} 2023-10-17T18:21:11.696 DEBUG:boto:Token: None 2023-10-17T18:21:11.697 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:11/OemPpHtLG3PJHeDETM+wprrE= 2023-10-17T18:21:11.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:11/OemPpHtLG3PJHeDETM+wprrE=', 'Content-Length': '0'} 2023-10-17T18:21:11.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d06ede02fbc6ce2-00652ed097-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:21:11.701 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.701 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.702 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.702 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.703 DEBUG:boto:Method: GET 2023-10-17T18:21:11.703 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.703 DEBUG:boto:Data: 2023-10-17T18:21:11.704 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.704 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.705 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.705 DEBUG:boto:Params: {} 2023-10-17T18:21:11.706 DEBUG:boto:Token: None 2023-10-17T18:21:11.706 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.706 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001872a2bb3a69576d-00652ed097-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.711 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.712 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.712 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.712 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.713 DEBUG:boto:Method: GET 2023-10-17T18:21:11.713 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.714 DEBUG:boto:Data: 2023-10-17T18:21:11.714 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.715 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.715 DEBUG:boto:Params: {} 2023-10-17T18:21:11.716 DEBUG:boto:Token: None 2023-10-17T18:21:11.716 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8a13c04e055b99c-00652ed097-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.720 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.720 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.721 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.721 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.722 DEBUG:boto:Method: GET 2023-10-17T18:21:11.722 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.723 DEBUG:boto:Data: 2023-10-17T18:21:11.723 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.724 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.725 DEBUG:boto:Params: {} 2023-10-17T18:21:11.725 DEBUG:boto:Token: None 2023-10-17T18:21:11.725 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.726 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw= 2023-10-17T18:21:11.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw=', 'Content-Length': '0'} 2023-10-17T18:21:11.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1787a8173867129-00652ed097-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.729 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.730 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.731 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.731 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.731 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.732 DEBUG:boto:Method: GET 2023-10-17T18:21:11.732 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.733 DEBUG:boto:Data: 2023-10-17T18:21:11.733 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.734 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.734 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.734 DEBUG:boto:Params: {} 2023-10-17T18:21:11.735 DEBUG:boto:Token: None 2023-10-17T18:21:11.735 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw= 2023-10-17T18:21:11.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw=', 'Content-Length': '0'} 2023-10-17T18:21:11.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7111b028bc7d55f-00652ed097-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.739 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.740 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.740 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.741 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.741 DEBUG:boto:Path: /hbmuog-13/dummy 2023-10-17T18:21:11.741 DEBUG:boto:Data: 2023-10-17T18:21:11.742 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.743 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.743 DEBUG:boto:Params: {} 2023-10-17T18:21:11.743 DEBUG:boto:Token: None 2023-10-17T18:21:11.744 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy 2023-10-17T18:21:11.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wssh7TFnqNOc55PDHFXV8A1vYNo= 2023-10-17T18:21:11.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wssh7TFnqNOc55PDHFXV8A1vYNo=', 'Content-Length': '0'} 2023-10-17T18:21:11.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3d9775024ee28f9-00652ed097-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.748 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.748 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.749 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.749 DEBUG:boto:Path: /hbmuog-13/dummy 2023-10-17T18:21:11.750 DEBUG:boto:Data: 2023-10-17T18:21:11.750 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.751 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.751 DEBUG:boto:Params: {} 2023-10-17T18:21:11.752 DEBUG:boto:Token: None 2023-10-17T18:21:11.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy 2023-10-17T18:21:11.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wssh7TFnqNOc55PDHFXV8A1vYNo= 2023-10-17T18:21:11.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wssh7TFnqNOc55PDHFXV8A1vYNo=', 'Content-Length': '0'} 2023-10-17T18:21:11.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009103cbc9dc63a6ae-00652ed097-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:21:11.757 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.757 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.758 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.758 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.759 DEBUG:boto:Method: GET 2023-10-17T18:21:11.759 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.760 DEBUG:boto:Data: 2023-10-17T18:21:11.760 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.761 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.761 DEBUG:boto:Params: {} 2023-10-17T18:21:11.762 DEBUG:boto:Token: None 2023-10-17T18:21:11.762 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.763 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0a5bcfcbfa08875-00652ed097-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.766 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:21:11.767 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:21:11.767 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.768 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.768 DEBUG:boto:Method: GET 2023-10-17T18:21:11.768 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.769 DEBUG:boto:Data: 2023-10-17T18:21:11.769 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.770 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.771 DEBUG:boto:Params: {} 2023-10-17T18:21:11.771 DEBUG:boto:Token: None 2023-10-17T18:21:11.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:21:11.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80= 2023-10-17T18:21:11.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jgk9VLx7+xJRJ8/IMYOKwnWLS80=', 'Content-Length': '0'} 2023-10-17T18:21:11.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c7bf595aa8224d2-00652ed097-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.775 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.776 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.776 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.777 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.777 DEBUG:boto:Method: GET 2023-10-17T18:21:11.777 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.778 DEBUG:boto:Data: 2023-10-17T18:21:11.778 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.779 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.780 DEBUG:boto:Params: {} 2023-10-17T18:21:11.780 DEBUG:boto:Token: None 2023-10-17T18:21:11.781 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw= 2023-10-17T18:21:11.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw=', 'Content-Length': '0'} 2023-10-17T18:21:11.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b2b3dbe5b2fb60-00652ed097-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.826 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.827 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.828 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.828 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.829 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.829 DEBUG:boto:Method: GET 2023-10-17T18:21:11.830 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:21:11.830 DEBUG:boto:Data: 2023-10-17T18:21:11.831 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.831 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.832 DEBUG:boto:Params: {} 2023-10-17T18:21:11.832 DEBUG:boto:Token: None 2023-10-17T18:21:11.833 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw= 2023-10-17T18:21:11.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hJi/+o+oICQRnnr1fVze3Z28Dxw=', 'Content-Length': '0'} 2023-10-17T18:21:11.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60abe5a9e861d93-00652ed097-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.837 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:21:11.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-13 zones={a2, a1} 2023-10-17T18:21:11.838 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.838 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.839 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.839 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.839 DEBUG:boto:Data: 2023-10-17T18:21:11.840 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.840 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.841 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.841 DEBUG:boto:Params: {} 2023-10-17T18:21:11.841 DEBUG:boto:Token: None 2023-10-17T18:21:11.842 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.842 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd3fdcd08277e8ca-00652ed097-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.846 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:11.846 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:11.847 DEBUG:boto:Method: PUT 2023-10-17T18:21:11.847 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:11.848 DEBUG:boto:Data: 2023-10-17T18:21:11.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:11.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.849 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.849 DEBUG:boto:Params: {} 2023-10-17T18:21:11.850 DEBUG:boto:Token: None 2023-10-17T18:21:11.850 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/obj 2023-10-17T18:21:11.851 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98= 2023-10-17T18:21:11.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98='} 2023-10-17T18:21:11.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-amz-request-id', 'tx000000be4da508bde9a9e-00652ed097-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.864 DEBUG:rgw_multi.tests:zone=a1 version=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:21:11.865 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.865 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.866 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.866 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.866 DEBUG:boto:Data: 2023-10-17T18:21:11.867 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.868 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.868 DEBUG:boto:Params: {} 2023-10-17T18:21:11.868 DEBUG:boto:Token: None 2023-10-17T18:21:11.869 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb9483fe587cc460-00652ed097-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.873 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:11.874 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:11.874 DEBUG:boto:Method: PUT 2023-10-17T18:21:11.875 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:11.875 DEBUG:boto:Data: 2023-10-17T18:21:11.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:11.876 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.876 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.877 DEBUG:boto:Params: {} 2023-10-17T18:21:11.877 DEBUG:boto:Token: None 2023-10-17T18:21:11.877 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/obj 2023-10-17T18:21:11.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98= 2023-10-17T18:21:11.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98='} 2023-10-17T18:21:11.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-amz-request-id', 'tx00000947be3b1d1564b4d-00652ed097-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.891 DEBUG:rgw_multi.tests:zone=a2 version=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:21:11.892 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.892 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.893 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.893 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.893 DEBUG:boto:Data: 2023-10-17T18:21:11.894 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.895 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.895 DEBUG:boto:Params: {} 2023-10-17T18:21:11.895 DEBUG:boto:Token: None 2023-10-17T18:21:11.896 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009edfa3215d21bab0-00652ed097-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.900 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:11.900 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:11.900 DEBUG:boto:Method: PUT 2023-10-17T18:21:11.901 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:11.901 DEBUG:boto:Data: 2023-10-17T18:21:11.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:11.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.902 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.903 DEBUG:boto:Params: {} 2023-10-17T18:21:11.903 DEBUG:boto:Token: None 2023-10-17T18:21:11.904 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/obj 2023-10-17T18:21:11.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98= 2023-10-17T18:21:11.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98='} 2023-10-17T18:21:11.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-amz-request-id', 'tx00000423867ab015bc2f3-00652ed097-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.916 DEBUG:rgw_multi.tests:zone=a1 version=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:21:11.916 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.916 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.917 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.917 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.918 DEBUG:boto:Data: 2023-10-17T18:21:11.918 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.919 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.921 DEBUG:boto:Params: {} 2023-10-17T18:21:11.921 DEBUG:boto:Token: None 2023-10-17T18:21:11.922 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.922 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dcae571500fb71b-00652ed097-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.926 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:11.927 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:11.927 DEBUG:boto:Method: PUT 2023-10-17T18:21:11.927 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:11.928 DEBUG:boto:Data: 2023-10-17T18:21:11.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:11.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:11.929 DEBUG:boto:Port: 8001 2023-10-17T18:21:11.929 DEBUG:boto:Params: {} 2023-10-17T18:21:11.930 DEBUG:boto:Token: None 2023-10-17T18:21:11.930 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/obj 2023-10-17T18:21:11.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98= 2023-10-17T18:21:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98='} 2023-10-17T18:21:11.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-amz-request-id', 'tx000008fa8dfe7f166cff6-00652ed097-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.941 DEBUG:rgw_multi.tests:zone=a2 version=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:21:11.942 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:11.942 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:11.943 DEBUG:boto:Method: HEAD 2023-10-17T18:21:11.943 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:11.943 DEBUG:boto:Data: 2023-10-17T18:21:11.944 DEBUG:boto:Headers: {} 2023-10-17T18:21:11.944 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.945 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.945 DEBUG:boto:Params: {} 2023-10-17T18:21:11.946 DEBUG:boto:Token: None 2023-10-17T18:21:11.946 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/ 2023-10-17T18:21:11.947 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw= 2023-10-17T18:21:11.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Am3LYj+h1AmXAasa9EHiowSQufw=', 'Content-Length': '0'} 2023-10-17T18:21:11.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af6ef49cfd41cd85-00652ed097-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.950 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:11.951 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:11.951 DEBUG:boto:Method: PUT 2023-10-17T18:21:11.952 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:11.952 DEBUG:boto:Data: 2023-10-17T18:21:11.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:11.953 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:11.953 DEBUG:boto:Port: 8000 2023-10-17T18:21:11.954 DEBUG:boto:Params: {} 2023-10-17T18:21:11.954 DEBUG:boto:Token: None 2023-10-17T18:21:11.955 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:11 GMT /hbmuog-13/obj 2023-10-17T18:21:11.955 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98= 2023-10-17T18:21:11.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9gmp0ycHE9yh7/rFbJJH9SRt98='} 2023-10-17T18:21:11.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-amz-request-id', 'tx0000019e576e068299a9b-00652ed097-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:11.999 DEBUG:rgw_multi.tests:zone=a1 version=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:21:11.999 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.000 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.000 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.001 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.001 DEBUG:boto:Data: 2023-10-17T18:21:12.002 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.002 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.003 DEBUG:boto:Params: {} 2023-10-17T18:21:12.003 DEBUG:boto:Token: None 2023-10-17T18:21:12.004 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c9585d486b52e5d-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.008 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.008 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.008 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.009 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.009 DEBUG:boto:Data: 2023-10-17T18:21:12.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.010 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.011 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.011 DEBUG:boto:Params: {} 2023-10-17T18:21:12.012 DEBUG:boto:Token: None 2023-10-17T18:21:12.012 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-amz-request-id', 'tx000007a1da727791d4d6b-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.024 DEBUG:rgw_multi.tests:zone=a2 version=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:21:12.024 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.024 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.025 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.025 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.026 DEBUG:boto:Data: 2023-10-17T18:21:12.026 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.027 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.027 DEBUG:boto:Params: {} 2023-10-17T18:21:12.028 DEBUG:boto:Token: None 2023-10-17T18:21:12.028 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a32a18bc34d60ee5-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.032 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.033 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.033 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.034 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.034 DEBUG:boto:Data: 2023-10-17T18:21:12.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.036 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.036 DEBUG:boto:Params: {} 2023-10-17T18:21:12.037 DEBUG:boto:Token: None 2023-10-17T18:21:12.037 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.038 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-amz-request-id', 'tx00000e4fcc23a08778fc8-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.049 DEBUG:rgw_multi.tests:zone=a1 version=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:21:12.049 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.049 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.050 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.050 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.051 DEBUG:boto:Data: 2023-10-17T18:21:12.051 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.052 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.052 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.053 DEBUG:boto:Params: {} 2023-10-17T18:21:12.053 DEBUG:boto:Token: None 2023-10-17T18:21:12.054 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.054 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ca07d7e469e1c35-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.057 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.058 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.058 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.059 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.059 DEBUG:boto:Data: 2023-10-17T18:21:12.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.060 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.060 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.061 DEBUG:boto:Params: {} 2023-10-17T18:21:12.061 DEBUG:boto:Token: None 2023-10-17T18:21:12.062 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.062 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-amz-request-id', 'tx00000ce5a2bbf4a770802-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.107 DEBUG:rgw_multi.tests:zone=a2 version=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:21:12.107 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.108 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.108 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.109 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.109 DEBUG:boto:Data: 2023-10-17T18:21:12.109 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.110 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.111 DEBUG:boto:Params: {} 2023-10-17T18:21:12.111 DEBUG:boto:Token: None 2023-10-17T18:21:12.112 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065c6aa0d14737f65-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.116 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.116 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.116 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.117 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.117 DEBUG:boto:Data: 2023-10-17T18:21:12.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.118 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.118 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.119 DEBUG:boto:Params: {} 2023-10-17T18:21:12.119 DEBUG:boto:Token: None 2023-10-17T18:21:12.120 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-amz-request-id', 'tx000006b7013e3e895225a-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.167 DEBUG:rgw_multi.tests:zone=a1 version=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:21:12.168 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.168 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.169 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.169 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.169 DEBUG:boto:Data: 2023-10-17T18:21:12.170 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.171 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.171 DEBUG:boto:Params: {} 2023-10-17T18:21:12.172 DEBUG:boto:Token: None 2023-10-17T18:21:12.172 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000efda480208336c-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.176 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.177 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.177 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.177 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.178 DEBUG:boto:Data: 2023-10-17T18:21:12.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.179 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.179 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.180 DEBUG:boto:Params: {} 2023-10-17T18:21:12.180 DEBUG:boto:Token: None 2023-10-17T18:21:12.180 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.181 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-amz-request-id', 'tx0000095a56278e334c704-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.195 DEBUG:rgw_multi.tests:zone=a2 version=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:21:12.195 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.196 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.197 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.198 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.198 DEBUG:boto:Data: 2023-10-17T18:21:12.198 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.199 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.200 DEBUG:boto:Params: {} 2023-10-17T18:21:12.200 DEBUG:boto:Token: None 2023-10-17T18:21:12.201 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f59733515d178d6-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.205 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.206 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.206 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.207 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.207 DEBUG:boto:Data: 2023-10-17T18:21:12.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.208 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.209 DEBUG:boto:Params: {} 2023-10-17T18:21:12.209 DEBUG:boto:Token: None 2023-10-17T18:21:12.210 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-amz-request-id', 'tx00000470dfb82b1bfcab3-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.221 DEBUG:rgw_multi.tests:zone=a1 version=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:21:12.222 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.222 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.223 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.223 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.224 DEBUG:boto:Data: 2023-10-17T18:21:12.224 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.225 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.225 DEBUG:boto:Params: {} 2023-10-17T18:21:12.226 DEBUG:boto:Token: None 2023-10-17T18:21:12.226 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.227 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f8fec312578b15e-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.230 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.231 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.231 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.232 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.232 DEBUG:boto:Data: 2023-10-17T18:21:12.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.234 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.234 DEBUG:boto:Params: {} 2023-10-17T18:21:12.234 DEBUG:boto:Token: None 2023-10-17T18:21:12.235 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-amz-request-id', 'tx0000056f97ff6509b8090-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.247 DEBUG:rgw_multi.tests:zone=a2 version=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:21:12.248 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.248 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.249 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.249 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.249 DEBUG:boto:Data: 2023-10-17T18:21:12.250 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.250 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.251 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.251 DEBUG:boto:Params: {} 2023-10-17T18:21:12.251 DEBUG:boto:Token: None 2023-10-17T18:21:12.252 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f488c44eac09f27-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.255 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.256 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.256 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.257 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.257 DEBUG:boto:Data: 2023-10-17T18:21:12.257 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.258 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.259 DEBUG:boto:Params: {} 2023-10-17T18:21:12.259 DEBUG:boto:Token: None 2023-10-17T18:21:12.259 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.260 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-amz-request-id', 'tx000001bbd293850c631ef-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.303 DEBUG:rgw_multi.tests:zone=a1 version=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:21:12.304 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.304 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.304 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.305 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.305 DEBUG:boto:Data: 2023-10-17T18:21:12.306 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.306 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.306 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.307 DEBUG:boto:Params: {} 2023-10-17T18:21:12.307 DEBUG:boto:Token: None 2023-10-17T18:21:12.308 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.308 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6bdab0c615ea657-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.312 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.312 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.312 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.313 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.313 DEBUG:boto:Data: 2023-10-17T18:21:12.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.314 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.315 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.315 DEBUG:boto:Params: {} 2023-10-17T18:21:12.315 DEBUG:boto:Token: None 2023-10-17T18:21:12.316 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-amz-request-id', 'tx000009c1f50e7a77686bf-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.363 DEBUG:rgw_multi.tests:zone=a2 version=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:21:12.363 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.364 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.364 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.365 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.365 DEBUG:boto:Data: 2023-10-17T18:21:12.365 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.366 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.366 DEBUG:boto:Params: {} 2023-10-17T18:21:12.367 DEBUG:boto:Token: None 2023-10-17T18:21:12.367 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b9a5c1577c009dd-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.371 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.372 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.372 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.373 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.373 DEBUG:boto:Data: 2023-10-17T18:21:12.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.374 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.375 DEBUG:boto:Params: {} 2023-10-17T18:21:12.375 DEBUG:boto:Token: None 2023-10-17T18:21:12.376 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-amz-request-id', 'tx00000f17dd23fb0f90eb2-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.386 DEBUG:rgw_multi.tests:zone=a1 version=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:21:12.386 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.387 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.387 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.388 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.388 DEBUG:boto:Data: 2023-10-17T18:21:12.388 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.389 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.389 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.390 DEBUG:boto:Params: {} 2023-10-17T18:21:12.390 DEBUG:boto:Token: None 2023-10-17T18:21:12.391 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.391 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003869146993d2a027-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.394 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.395 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.395 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.396 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.396 DEBUG:boto:Data: 2023-10-17T18:21:12.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.398 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.398 DEBUG:boto:Params: {} 2023-10-17T18:21:12.399 DEBUG:boto:Token: None 2023-10-17T18:21:12.399 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.400 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-amz-request-id', 'tx0000000935a0c91acaf5a-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.410 DEBUG:rgw_multi.tests:zone=a2 version=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:21:12.410 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.411 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.411 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.411 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.412 DEBUG:boto:Data: 2023-10-17T18:21:12.412 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.413 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.413 DEBUG:boto:Params: {} 2023-10-17T18:21:12.414 DEBUG:boto:Token: None 2023-10-17T18:21:12.414 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ab40b6159de8bb9-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.418 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.419 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.419 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.420 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.420 DEBUG:boto:Data: 2023-10-17T18:21:12.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.421 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.421 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.422 DEBUG:boto:Params: {} 2023-10-17T18:21:12.422 DEBUG:boto:Token: None 2023-10-17T18:21:12.422 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-amz-request-id', 'tx00000236a38e097cffa04-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.433 DEBUG:rgw_multi.tests:zone=a1 version=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:21:12.433 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.434 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.434 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.435 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.435 DEBUG:boto:Data: 2023-10-17T18:21:12.436 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.436 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.437 DEBUG:boto:Params: {} 2023-10-17T18:21:12.437 DEBUG:boto:Token: None 2023-10-17T18:21:12.438 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.438 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b452eed490583f6-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.442 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.442 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.442 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.443 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.443 DEBUG:boto:Data: 2023-10-17T18:21:12.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.445 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.445 DEBUG:boto:Params: {} 2023-10-17T18:21:12.445 DEBUG:boto:Token: None 2023-10-17T18:21:12.446 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-amz-request-id', 'tx00000e76ab62780129d6d-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.456 DEBUG:rgw_multi.tests:zone=a2 version=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:21:12.457 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.457 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.458 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.458 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.459 DEBUG:boto:Data: 2023-10-17T18:21:12.459 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.459 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.460 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.460 DEBUG:boto:Params: {} 2023-10-17T18:21:12.461 DEBUG:boto:Token: None 2023-10-17T18:21:12.461 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.462 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f13b61431df32ff1-00652ed098-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.465 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.465 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.466 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.466 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.467 DEBUG:boto:Data: 2023-10-17T18:21:12.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:21:12.468 DEBUG:boto:Port: 8000 2023-10-17T18:21:12.468 DEBUG:boto:Params: {} 2023-10-17T18:21:12.469 DEBUG:boto:Token: None 2023-10-17T18:21:12.469 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.470 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-amz-request-id', 'tx0000085c4b0cad1550ec9-00652ed098-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.515 DEBUG:rgw_multi.tests:zone=a1 version=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:21:12.516 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:21:12.516 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:21:12.516 DEBUG:boto:Method: HEAD 2023-10-17T18:21:12.517 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:21:12.517 DEBUG:boto:Data: 2023-10-17T18:21:12.518 DEBUG:boto:Headers: {} 2023-10-17T18:21:12.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.519 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.519 DEBUG:boto:Params: {} 2023-10-17T18:21:12.520 DEBUG:boto:Token: None 2023-10-17T18:21:12.520 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/ 2023-10-17T18:21:12.521 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w= 2023-10-17T18:21:12.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiG+MNNeezWxCH+mxZnLZcPXi7w=', 'Content-Length': '0'} 2023-10-17T18:21:12.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059631ecd8f6b722f-00652ed098-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.524 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:21:12.524 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:21:12.525 DEBUG:boto:Method: PUT 2023-10-17T18:21:12.525 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:21:12.526 DEBUG:boto:Data: 2023-10-17T18:21:12.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-17T18:21:12.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:21:12.527 DEBUG:boto:Port: 8001 2023-10-17T18:21:12.527 DEBUG:boto:Params: {} 2023-10-17T18:21:12.528 DEBUG:boto:Token: None 2023-10-17T18:21:12.528 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 17 Oct 2023 18:21:12 GMT /hbmuog-13/obj 2023-10-17T18:21:12.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s= 2023-10-17T18:21:12.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:21:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GO/I8By6l0YKJHbDpO21Dz/hN3s='} 2023-10-17T18:21:12.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-amz-request-id', 'tx00000b949e4d1df7665f3-00652ed098-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:21:12.579 DEBUG:rgw_multi.tests:zone=a2 version=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:21:12.579 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-13 2023-10-17T18:21:12.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:21:12.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:21:12.581 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:21:12.642 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:21:12.643 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:21:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:12.677+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:12.678 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:21:12.679 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:17.679+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:17.680 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:21:17.680 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:22.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:22.681+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:22.681 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:21:22.682 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:27.682 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:27.681+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:27.683 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-10-17T18:21:27.683 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:32.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:32.683+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:32.684 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000017.41.6, , ] 2023-10-17T18:21:32.684 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:37.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:37.684+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:37.686 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000017.41.6, , ] 2023-10-17T18:21:37.686 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:42.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:42.685+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:42.687 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000017.41.6, , ] 2023-10-17T18:21:42.687 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:47.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:47.686+0000 7faa71993780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:47.688 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000017.41.6, , ] 2023-10-17T18:21:47.688 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:52.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:52.689+0000 7faa71993780 1 bucket sync caught up with source: 2023-10-17T18:21:52.690 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000031.77.9, , ] 2023-10-17T18:21:52.690 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000020.46.12, , ] 2023-10-17T18:21:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:52.689+0000 7faa71993780 0 bucket checkpoint complete 2023-10-17T18:21:52.705 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-13 2023-10-17T18:21:52.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:21:52.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:21:52.707 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:21:52.771 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:21:52.771 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:21:52.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:52.808+0000 7f376604d780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:52.809 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , 00000000028.111.6, , ] 2023-10-17T18:21:52.809 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:21:57.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:21:57.810+0000 7f376604d780 1 waiting for incremental sync to catch up: 2023-10-17T18:21:57.810 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , 00000000028.111.6, , ] 2023-10-17T18:21:57.810 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:22:02.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:02.810+0000 7f376604d780 1 waiting for incremental sync to catch up: 2023-10-17T18:22:02.812 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , 00000000028.111.6, , ] 2023-10-17T18:22:02.812 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:22:07.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:07.811+0000 7f376604d780 1 waiting for incremental sync to catch up: 2023-10-17T18:22:07.813 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , 00000000028.111.6, , ] 2023-10-17T18:22:07.813 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:22:12.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:12.813+0000 7f376604d780 1 bucket sync caught up with source: 2023-10-17T18:22:12.814 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:22:12.815 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000002.51.11, , , 00000000031.118.12, , ] 2023-10-17T18:22:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:12.813+0000 7f376604d780 0 bucket checkpoint complete 2023-10-17T18:22:12.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-13 zones={a2, a1} 2023-10-17T18:22:12.831 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.831 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.832 DEBUG:boto:Method: HEAD 2023-10-17T18:22:12.832 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:22:12.833 DEBUG:boto:Data: 2023-10-17T18:22:12.833 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.834 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.834 DEBUG:boto:Params: {} 2023-10-17T18:22:12.835 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:22:12.836 DEBUG:boto:Token: None 2023-10-17T18:22:12.836 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/ 2023-10-17T18:22:12.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MKGoqrUaijAKJDGX7wx9+xpC+y0= 2023-10-17T18:22:12.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MKGoqrUaijAKJDGX7wx9+xpC+y0=', 'Content-Length': '0'} 2023-10-17T18:22:12.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b5e8f580cb13e39-00652ed0d4-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.842 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.842 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.843 DEBUG:boto:Method: HEAD 2023-10-17T18:22:12.843 DEBUG:boto:Path: /hbmuog-13/ 2023-10-17T18:22:12.844 DEBUG:boto:Data: 2023-10-17T18:22:12.844 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.844 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.845 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.845 DEBUG:boto:Params: {} 2023-10-17T18:22:12.846 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:22:12.846 DEBUG:boto:Token: None 2023-10-17T18:22:12.847 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/ 2023-10-17T18:22:12.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MKGoqrUaijAKJDGX7wx9+xpC+y0= 2023-10-17T18:22:12.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MKGoqrUaijAKJDGX7wx9+xpC+y0=', 'Content-Length': '0'} 2023-10-17T18:22:12.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c26c6c50aec7d8af-00652ed0d4-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.852 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:22:12.852 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.852 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.853 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.853 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.854 DEBUG:boto:Method: GET 2023-10-17T18:22:12.854 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.855 DEBUG:boto:Data: 2023-10-17T18:22:12.855 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.855 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.856 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.856 DEBUG:boto:Params: {} 2023-10-17T18:22:12.857 DEBUG:boto:Token: None 2023-10-17T18:22:12.857 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/?versions 2023-10-17T18:22:12.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM= 2023-10-17T18:22:12.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM=', 'Content-Length': '0'} 2023-10-17T18:22:12.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000519189550d5a0f2-00652ed0d4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.861 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMhtrue2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtxzYLw8omWVuA3.LdXpKY6DjMSuzNNtfalse2023-10-17T18:21:12.531Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOdHYC8pdXrOuCg4RYy64JQ1o2QDyPqnfalse2023-10-17T18:21:12.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrG80Z6RP5ssXNPx6txr-pL1IEj-Ibozfalse2023-10-17T18:21:12.449Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.pPFaokTIv6RMPRkCpuToO8yY29matefalse2023-10-17T18:21:12.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEpUZ.-OLSUZcByZDwOabzUO9OaVD.KGfalse2023-10-17T18:21:12.402Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3false2023-10-17T18:21:12.319Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjw3lLb1lmq4GvkGFK0XRgyJqunVLVxB9false2023-10-17T18:21:12.262Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs3dLv7gwdXVgtTA6evp.aG64TtsPNjEfalse2023-10-17T18:21:12.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjwfalse2023-10-17T18:21:12.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vEfalse2023-10-17T18:21:12.183Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbNYB8oGeDeoT3eNcfxSR3SPKav3E0PIfalse2023-10-17T18:21:12.123Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTsUld4HsC0SqKDxbDEGOA493mZHoSCzfalse2023-10-17T18:21:12.040Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdXQC0aagQwy9HHQaDPPZUv4y6-4ewTLfalse2023-10-17T18:21:12.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCi1VxMdlhJnaYDdbpQem.RUfluTWjXdfalse2023-10-17T18:21:12.015Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cbfalse2023-10-17T18:21:11.958Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9false2023-10-17T18:21:11.907Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOKEhtnlf6XQGZFxQtWGUXOIJqaWWH-ufalse2023-10-17T18:21:11.933Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30false2023-10-17T18:21:11.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHuNXlITqWHRXWomQb1bjamN0YoudhQfalse2023-10-17T18:21:11.853Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:12.863 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:22:12.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.871 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:22:12.871 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.872 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.872 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.873 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.873 DEBUG:boto:Method: GET 2023-10-17T18:22:12.874 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.874 DEBUG:boto:Data: 2023-10-17T18:22:12.874 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.875 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.876 DEBUG:boto:Params: {} 2023-10-17T18:22:12.876 DEBUG:boto:Token: None 2023-10-17T18:22:12.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/?versions 2023-10-17T18:22:12.877 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM= 2023-10-17T18:22:12.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM=', 'Content-Length': '0'} 2023-10-17T18:22:12.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d259dce8cdde24-00652ed0d4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.881 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMhtrue2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtxzYLw8omWVuA3.LdXpKY6DjMSuzNNtfalse2023-10-17T18:21:12.531Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOdHYC8pdXrOuCg4RYy64JQ1o2QDyPqnfalse2023-10-17T18:21:12.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrG80Z6RP5ssXNPx6txr-pL1IEj-Ibozfalse2023-10-17T18:21:12.449Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.pPFaokTIv6RMPRkCpuToO8yY29matefalse2023-10-17T18:21:12.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEpUZ.-OLSUZcByZDwOabzUO9OaVD.KGfalse2023-10-17T18:21:12.402Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3false2023-10-17T18:21:12.319Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjw3lLb1lmq4GvkGFK0XRgyJqunVLVxB9false2023-10-17T18:21:12.262Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs3dLv7gwdXVgtTA6evp.aG64TtsPNjEfalse2023-10-17T18:21:12.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjwfalse2023-10-17T18:21:12.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vEfalse2023-10-17T18:21:12.183Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbNYB8oGeDeoT3eNcfxSR3SPKav3E0PIfalse2023-10-17T18:21:12.123Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTsUld4HsC0SqKDxbDEGOA493mZHoSCzfalse2023-10-17T18:21:12.040Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdXQC0aagQwy9HHQaDPPZUv4y6-4ewTLfalse2023-10-17T18:21:12.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCi1VxMdlhJnaYDdbpQem.RUfluTWjXdfalse2023-10-17T18:21:12.015Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cbfalse2023-10-17T18:21:11.958Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9false2023-10-17T18:21:11.907Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOKEhtnlf6XQGZFxQtWGUXOIJqaWWH-ufalse2023-10-17T18:21:11.933Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30false2023-10-17T18:21:11.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHuNXlITqWHRXWomQb1bjamN0YoudhQfalse2023-10-17T18:21:11.853Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:12.883 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T18:22:12.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:12.891 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.891 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.891 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.892 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.892 DEBUG:boto:Method: GET 2023-10-17T18:22:12.893 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.893 DEBUG:boto:Data: 2023-10-17T18:22:12.893 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.894 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.895 DEBUG:boto:Params: {} 2023-10-17T18:22:12.895 DEBUG:boto:Token: None 2023-10-17T18:22:12.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/?versions 2023-10-17T18:22:12.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM= 2023-10-17T18:22:12.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM=', 'Content-Length': '0'} 2023-10-17T18:22:12.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c6558cf31e378b-00652ed0d4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.900 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMhtrue2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtxzYLw8omWVuA3.LdXpKY6DjMSuzNNtfalse2023-10-17T18:21:12.531Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOdHYC8pdXrOuCg4RYy64JQ1o2QDyPqnfalse2023-10-17T18:21:12.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrG80Z6RP5ssXNPx6txr-pL1IEj-Ibozfalse2023-10-17T18:21:12.449Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.pPFaokTIv6RMPRkCpuToO8yY29matefalse2023-10-17T18:21:12.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEpUZ.-OLSUZcByZDwOabzUO9OaVD.KGfalse2023-10-17T18:21:12.402Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3false2023-10-17T18:21:12.319Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjw3lLb1lmq4GvkGFK0XRgyJqunVLVxB9false2023-10-17T18:21:12.262Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs3dLv7gwdXVgtTA6evp.aG64TtsPNjEfalse2023-10-17T18:21:12.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjwfalse2023-10-17T18:21:12.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vEfalse2023-10-17T18:21:12.183Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbNYB8oGeDeoT3eNcfxSR3SPKav3E0PIfalse2023-10-17T18:21:12.123Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTsUld4HsC0SqKDxbDEGOA493mZHoSCzfalse2023-10-17T18:21:12.040Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdXQC0aagQwy9HHQaDPPZUv4y6-4ewTLfalse2023-10-17T18:21:12.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCi1VxMdlhJnaYDdbpQem.RUfluTWjXdfalse2023-10-17T18:21:12.015Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cbfalse2023-10-17T18:21:11.958Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9false2023-10-17T18:21:11.907Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOKEhtnlf6XQGZFxQtWGUXOIJqaWWH-ufalse2023-10-17T18:21:11.933Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30false2023-10-17T18:21:11.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHuNXlITqWHRXWomQb1bjamN0YoudhQfalse2023-10-17T18:21:11.853Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:12.901 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.902 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.902 DEBUG:boto:path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.903 DEBUG:boto:auth_path=/hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.903 DEBUG:boto:Method: GET 2023-10-17T18:22:12.903 DEBUG:boto:Path: /hbmuog-13/?versions&max-keys=999 2023-10-17T18:22:12.904 DEBUG:boto:Data: 2023-10-17T18:22:12.904 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.905 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.905 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.905 DEBUG:boto:Params: {} 2023-10-17T18:22:12.906 DEBUG:boto:Token: None 2023-10-17T18:22:12.906 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/?versions 2023-10-17T18:22:12.907 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM= 2023-10-17T18:22:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9vpYMreIn1H5BKUhSWg5qnATCqM=', 'Content-Length': '0'} 2023-10-17T18:22:12.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f845277727335fd7-00652ed0d4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.911 DEBUG:boto:b'hbmuog-13999falsedummyE7EpANTvY6oF6FBYmGoU34dtXFRGlhptrue2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMhtrue2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtxzYLw8omWVuA3.LdXpKY6DjMSuzNNtfalse2023-10-17T18:21:12.531Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOdHYC8pdXrOuCg4RYy64JQ1o2QDyPqnfalse2023-10-17T18:21:12.425Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrG80Z6RP5ssXNPx6txr-pL1IEj-Ibozfalse2023-10-17T18:21:12.449Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.pPFaokTIv6RMPRkCpuToO8yY29matefalse2023-10-17T18:21:12.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjEpUZ.-OLSUZcByZDwOabzUO9OaVD.KGfalse2023-10-17T18:21:12.402Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3false2023-10-17T18:21:12.319Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjw3lLb1lmq4GvkGFK0XRgyJqunVLVxB9false2023-10-17T18:21:12.262Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjs3dLv7gwdXVgtTA6evp.aG64TtsPNjEfalse2023-10-17T18:21:12.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjyPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjwfalse2023-10-17T18:21:12.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vEfalse2023-10-17T18:21:12.183Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbNYB8oGeDeoT3eNcfxSR3SPKav3E0PIfalse2023-10-17T18:21:12.123Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTsUld4HsC0SqKDxbDEGOA493mZHoSCzfalse2023-10-17T18:21:12.040Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdXQC0aagQwy9HHQaDPPZUv4y6-4ewTLfalse2023-10-17T18:21:12.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCi1VxMdlhJnaYDdbpQem.RUfluTWjXdfalse2023-10-17T18:21:12.015Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cbfalse2023-10-17T18:21:11.958Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9false2023-10-17T18:21:11.907Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOKEhtnlf6XQGZFxQtWGUXOIJqaWWH-ufalse2023-10-17T18:21:11.933Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30false2023-10-17T18:21:11.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHuNXlITqWHRXWomQb1bjamN0YoudhQfalse2023-10-17T18:21:11.853Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:12.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:22:12.913 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.913 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.914 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.914 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.914 DEBUG:boto:Method: GET 2023-10-17T18:22:12.915 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.915 DEBUG:boto:Data: 2023-10-17T18:22:12.916 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.917 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.917 DEBUG:boto:Params: {} 2023-10-17T18:22:12.918 DEBUG:boto:Token: None 2023-10-17T18:22:12.918 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A= 2023-10-17T18:22:12.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A=', 'Content-Length': '0'} 2023-10-17T18:22:12.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f17c23ac5441040-00652ed0d4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.922 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.922 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.923 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.923 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.924 DEBUG:boto:Method: GET 2023-10-17T18:22:12.924 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.924 DEBUG:boto:Data: 2023-10-17T18:22:12.925 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.926 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.926 DEBUG:boto:Params: {} 2023-10-17T18:22:12.926 DEBUG:boto:Token: None 2023-10-17T18:22:12.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A= 2023-10-17T18:22:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A=', 'Content-Length': '0'} 2023-10-17T18:22:12.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5789d384d54753a-00652ed0d4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.930 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.931 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.931 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.932 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.932 DEBUG:boto:Method: HEAD 2023-10-17T18:22:12.933 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.933 DEBUG:boto:Data: 2023-10-17T18:22:12.933 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.934 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.935 DEBUG:boto:Params: {} 2023-10-17T18:22:12.935 DEBUG:boto:Token: None 2023-10-17T18:22:12.936 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BAwEFveTMfEcl/MvbECnbT/4A8w= 2023-10-17T18:22:12.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BAwEFveTMfEcl/MvbECnbT/4A8w=', 'Content-Length': '0'} 2023-10-17T18:22:12.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004482c89e3a78248a-00652ed0d4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.939 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.939 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.940 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.940 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.941 DEBUG:boto:Method: HEAD 2023-10-17T18:22:12.941 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.941 DEBUG:boto:Data: 2023-10-17T18:22:12.942 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.943 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.943 DEBUG:boto:Params: {} 2023-10-17T18:22:12.944 DEBUG:boto:Token: None 2023-10-17T18:22:12.944 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BAwEFveTMfEcl/MvbECnbT/4A8w= 2023-10-17T18:22:12.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BAwEFveTMfEcl/MvbECnbT/4A8w=', 'Content-Length': '0'} 2023-10-17T18:22:12.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007508d227d818f7da-00652ed0d4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:22:12.948 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.948 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.949 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.949 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.950 DEBUG:boto:Method: GET 2023-10-17T18:22:12.950 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.951 DEBUG:boto:Data: 2023-10-17T18:22:12.951 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.952 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.952 DEBUG:boto:Params: {} 2023-10-17T18:22:12.953 DEBUG:boto:Token: None 2023-10-17T18:22:12.953 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A= 2023-10-17T18:22:12.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A=', 'Content-Length': '0'} 2023-10-17T18:22:12.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073e56ebb67fdb3d2-00652ed0d4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.957 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:12.957 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:12.958 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.958 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.959 DEBUG:boto:Method: GET 2023-10-17T18:22:12.959 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.959 DEBUG:boto:Data: 2023-10-17T18:22:12.960 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.961 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.961 DEBUG:boto:Params: {} 2023-10-17T18:22:12.962 DEBUG:boto:Token: None 2023-10-17T18:22:12.962 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:12.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A= 2023-10-17T18:22:12.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AttR4SDTFwJ1Tuz7ReANxszNf/A=', 'Content-Length': '0'} 2023-10-17T18:22:12.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b90778364eb346f-00652ed0d4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.965 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.966 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.966 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.967 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.967 DEBUG:boto:Method: GET 2023-10-17T18:22:12.967 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.968 DEBUG:boto:Data: 2023-10-17T18:22:12.968 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:12.969 DEBUG:boto:Port: 8001 2023-10-17T18:22:12.970 DEBUG:boto:Params: {} 2023-10-17T18:22:12.970 DEBUG:boto:Token: None 2023-10-17T18:22:12.970 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/ 2023-10-17T18:22:12.971 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6wPa5vyWLjMuyhLp9YdVfio2QPw= 2023-10-17T18:22:12.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6wPa5vyWLjMuyhLp9YdVfio2QPw=', 'Content-Length': '0'} 2023-10-17T18:22:12.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842aac628a8e8f15-00652ed0d4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:12.974 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:12.975 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:12.976 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:12.976 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.976 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.977 DEBUG:boto:Method: GET 2023-10-17T18:22:12.977 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:22:12.978 DEBUG:boto:Data: 2023-10-17T18:22:12.978 DEBUG:boto:Headers: {} 2023-10-17T18:22:12.979 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:12.979 DEBUG:boto:Port: 8000 2023-10-17T18:22:12.979 DEBUG:boto:Params: {} 2023-10-17T18:22:12.980 DEBUG:boto:Token: None 2023-10-17T18:22:12.980 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:12 GMT /hbmuog-13/ 2023-10-17T18:22:12.981 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6wPa5vyWLjMuyhLp9YdVfio2QPw= 2023-10-17T18:22:12.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6wPa5vyWLjMuyhLp9YdVfio2QPw=', 'Content-Length': '0'} 2023-10-17T18:22:12.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546478374e9083b4-00652ed0d4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.026 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.027 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:13.028 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:13.028 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.029 DEBUG:boto:Path: /hbmuog-13/dummy 2023-10-17T18:22:13.029 DEBUG:boto:Data: 2023-10-17T18:22:13.029 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.030 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.031 DEBUG:boto:Params: {} 2023-10-17T18:22:13.031 DEBUG:boto:Token: None 2023-10-17T18:22:13.031 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/dummy 2023-10-17T18:22:13.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gcHF9uoQckYqvpKrRdMZQ0RQr5A= 2023-10-17T18:22:13.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gcHF9uoQckYqvpKrRdMZQ0RQr5A=', 'Content-Length': '0'} 2023-10-17T18:22:13.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f154e6d6b4ae472e-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.035 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:13.035 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:13.036 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.036 DEBUG:boto:Path: /hbmuog-13/dummy 2023-10-17T18:22:13.037 DEBUG:boto:Data: 2023-10-17T18:22:13.037 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.038 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.038 DEBUG:boto:Params: {} 2023-10-17T18:22:13.039 DEBUG:boto:Token: None 2023-10-17T18:22:13.039 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/dummy 2023-10-17T18:22:13.040 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gcHF9uoQckYqvpKrRdMZQ0RQr5A= 2023-10-17T18:22:13.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gcHF9uoQckYqvpKrRdMZQ0RQr5A=', 'Content-Length': '0'} 2023-10-17T18:22:13.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cd0fa0ca45f0918-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T18:22:13.044 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:13.044 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:13.044 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.045 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.045 DEBUG:boto:Method: GET 2023-10-17T18:22:13.046 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.046 DEBUG:boto:Data: 2023-10-17T18:22:13.047 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.047 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.048 DEBUG:boto:Params: {} 2023-10-17T18:22:13.048 DEBUG:boto:Token: None 2023-10-17T18:22:13.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eAcaeHGbuPYWIpx+4VnTYSO9ZNg= 2023-10-17T18:22:13.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eAcaeHGbuPYWIpx+4VnTYSO9ZNg=', 'Content-Length': '0'} 2023-10-17T18:22:13.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072ec3874992dad4c-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.052 DEBUG:boto:path=/hbmuog-13/dummy 2023-10-17T18:22:13.053 DEBUG:boto:auth_path=/hbmuog-13/dummy 2023-10-17T18:22:13.053 DEBUG:boto:path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.054 DEBUG:boto:auth_path=/hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.054 DEBUG:boto:Method: GET 2023-10-17T18:22:13.055 DEBUG:boto:Path: /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.055 DEBUG:boto:Data: 2023-10-17T18:22:13.056 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.057 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.057 DEBUG:boto:Params: {} 2023-10-17T18:22:13.058 DEBUG:boto:Token: None 2023-10-17T18:22:13.058 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/dummy?versionId=E7EpANTvY6oF6FBYmGoU34dtXFRGlhp 2023-10-17T18:22:13.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eAcaeHGbuPYWIpx+4VnTYSO9ZNg= 2023-10-17T18:22:13.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eAcaeHGbuPYWIpx+4VnTYSO9ZNg=', 'Content-Length': '0'} 2023-10-17T18:22:13.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:20:31 GMT'), ('x-amz-version-id', 'E7EpANTvY6oF6FBYmGoU34dtXFRGlhp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000419c1b10d8fa8e29-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.062 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.062 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.063 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.063 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.064 DEBUG:boto:Method: GET 2023-10-17T18:22:13.064 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.064 DEBUG:boto:Data: 2023-10-17T18:22:13.065 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.066 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.066 DEBUG:boto:Params: {} 2023-10-17T18:22:13.067 DEBUG:boto:Token: None 2023-10-17T18:22:13.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005900927a852068cc-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.114 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.115 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.116 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.116 DEBUG:boto:path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.116 DEBUG:boto:auth_path=/hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.117 DEBUG:boto:Method: GET 2023-10-17T18:22:13.117 DEBUG:boto:Path: /hbmuog-13/?prefix=dummy 2023-10-17T18:22:13.118 DEBUG:boto:Data: 2023-10-17T18:22:13.118 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.119 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.119 DEBUG:boto:Params: {} 2023-10-17T18:22:13.120 DEBUG:boto:Token: None 2023-10-17T18:22:13.120 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4dd2f1cb744371-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.124 DEBUG:boto:b'hbmuog-13dummy1000falsedummy2023-10-17T18:20:31.311Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.126 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.126 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.126 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.127 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.127 DEBUG:boto:Method: GET 2023-10-17T18:22:13.128 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.128 DEBUG:boto:Data: 2023-10-17T18:22:13.128 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.129 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.129 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.130 DEBUG:boto:Params: {} 2023-10-17T18:22:13.130 DEBUG:boto:Token: None 2023-10-17T18:22:13.131 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092793976df74e32f-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.134 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.135 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.135 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.136 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.136 DEBUG:boto:Method: GET 2023-10-17T18:22:13.136 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.137 DEBUG:boto:Data: 2023-10-17T18:22:13.137 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.138 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.138 DEBUG:boto:Params: {} 2023-10-17T18:22:13.139 DEBUG:boto:Token: None 2023-10-17T18:22:13.139 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba107024b22625bd-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.186 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.187 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.187 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.188 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.188 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.189 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.189 DEBUG:boto:Data: 2023-10-17T18:22:13.190 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.190 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.191 DEBUG:boto:Params: {} 2023-10-17T18:22:13.191 DEBUG:boto:Token: None 2023-10-17T18:22:13.192 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.192 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sVnjaBTeGyyvnlSmD58/lKH/8rA= 2023-10-17T18:22:13.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVnjaBTeGyyvnlSmD58/lKH/8rA=', 'Content-Length': '0'} 2023-10-17T18:22:13.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085bd2227f128ac49-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.196 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.196 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.196 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.197 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.197 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.198 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.198 DEBUG:boto:Data: 2023-10-17T18:22:13.198 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.199 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.200 DEBUG:boto:Params: {} 2023-10-17T18:22:13.200 DEBUG:boto:Token: None 2023-10-17T18:22:13.201 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sVnjaBTeGyyvnlSmD58/lKH/8rA= 2023-10-17T18:22:13.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVnjaBTeGyyvnlSmD58/lKH/8rA=', 'Content-Length': '0'} 2023-10-17T18:22:13.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5650fb406436377-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.205 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.205 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.205 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.206 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.206 DEBUG:boto:Method: GET 2023-10-17T18:22:13.206 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.207 DEBUG:boto:Data: 2023-10-17T18:22:13.207 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.208 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.209 DEBUG:boto:Params: {} 2023-10-17T18:22:13.209 DEBUG:boto:Token: None 2023-10-17T18:22:13.210 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3c65043b268e61-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.213 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.214 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.214 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.215 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.215 DEBUG:boto:Method: GET 2023-10-17T18:22:13.216 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.216 DEBUG:boto:Data: 2023-10-17T18:22:13.216 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.217 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.218 DEBUG:boto:Params: {} 2023-10-17T18:22:13.218 DEBUG:boto:Token: None 2023-10-17T18:22:13.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061104b3a65361891-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.222 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.223 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.223 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.224 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.224 DEBUG:boto:Method: GET 2023-10-17T18:22:13.225 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.225 DEBUG:boto:Data: 2023-10-17T18:22:13.225 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.227 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.227 DEBUG:boto:Params: {} 2023-10-17T18:22:13.228 DEBUG:boto:Token: None 2023-10-17T18:22:13.228 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac7643b45edc8d1-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.232 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.233 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.233 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.234 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.234 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.235 DEBUG:boto:Method: GET 2023-10-17T18:22:13.235 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.236 DEBUG:boto:Data: 2023-10-17T18:22:13.236 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.237 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.237 DEBUG:boto:Params: {} 2023-10-17T18:22:13.238 DEBUG:boto:Token: None 2023-10-17T18:22:13.238 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000853c756a7391c47e-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.242 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.243 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.244 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.245 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.245 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.246 DEBUG:boto:Data: 2023-10-17T18:22:13.246 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.247 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.247 DEBUG:boto:Params: {} 2023-10-17T18:22:13.248 DEBUG:boto:Token: None 2023-10-17T18:22:13.248 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.249 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3dfc668013a5e59-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.253 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.253 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.254 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.254 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.255 DEBUG:boto:Data: 2023-10-17T18:22:13.255 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.256 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.256 DEBUG:boto:Params: {} 2023-10-17T18:22:13.257 DEBUG:boto:Token: None 2023-10-17T18:22:13.257 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dec6a18a0a1d692-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.262 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.263 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.263 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.263 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.264 DEBUG:boto:Method: GET 2023-10-17T18:22:13.264 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.265 DEBUG:boto:Data: 2023-10-17T18:22:13.265 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.266 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.266 DEBUG:boto:Params: {} 2023-10-17T18:22:13.267 DEBUG:boto:Token: None 2023-10-17T18:22:13.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.268 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd3e10ce368dfa9e-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.314 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.315 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.315 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.316 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.316 DEBUG:boto:Method: GET 2023-10-17T18:22:13.316 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.317 DEBUG:boto:Data: 2023-10-17T18:22:13.317 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.318 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.318 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.319 DEBUG:boto:Params: {} 2023-10-17T18:22:13.319 DEBUG:boto:Token: None 2023-10-17T18:22:13.320 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025cb867192a19b5f-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.324 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.324 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.325 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.325 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.325 DEBUG:boto:Method: GET 2023-10-17T18:22:13.326 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.326 DEBUG:boto:Data: 2023-10-17T18:22:13.327 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.327 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.328 DEBUG:boto:Params: {} 2023-10-17T18:22:13.328 DEBUG:boto:Token: None 2023-10-17T18:22:13.329 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec8c1d3793779d50-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.333 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.334 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.335 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.335 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.335 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.336 DEBUG:boto:Method: GET 2023-10-17T18:22:13.336 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.337 DEBUG:boto:Data: 2023-10-17T18:22:13.337 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.338 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.338 DEBUG:boto:Params: {} 2023-10-17T18:22:13.339 DEBUG:boto:Token: None 2023-10-17T18:22:13.339 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c916c7ce48d78a4c-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.344 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.345 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.345 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.346 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.346 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.346 DEBUG:boto:Method: GET 2023-10-17T18:22:13.347 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.347 DEBUG:boto:Data: 2023-10-17T18:22:13.348 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.348 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.349 DEBUG:boto:Params: {} 2023-10-17T18:22:13.349 DEBUG:boto:Token: None 2023-10-17T18:22:13.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig= 2023-10-17T18:22:13.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig=', 'Content-Length': '0'} 2023-10-17T18:22:13.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001822482884d5ef9c-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.354 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.354 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.354 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.355 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.355 DEBUG:boto:Method: GET 2023-10-17T18:22:13.356 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.356 DEBUG:boto:Data: 2023-10-17T18:22:13.356 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.357 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.358 DEBUG:boto:Params: {} 2023-10-17T18:22:13.358 DEBUG:boto:Token: None 2023-10-17T18:22:13.359 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig= 2023-10-17T18:22:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig=', 'Content-Length': '0'} 2023-10-17T18:22:13.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc1f0a9b8369d41-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.402 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.403 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.403 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.404 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.404 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.404 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.405 DEBUG:boto:Data: 2023-10-17T18:22:13.405 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.406 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.406 DEBUG:boto:Params: {} 2023-10-17T18:22:13.407 DEBUG:boto:Token: None 2023-10-17T18:22:13.407 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+yzwlHymgq/cq5pfCYEq+mZ9Cg= 2023-10-17T18:22:13.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+yzwlHymgq/cq5pfCYEq+mZ9Cg=', 'Content-Length': '0'} 2023-10-17T18:22:13.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000287e2a1c4406b6d-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.411 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.411 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.412 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.412 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.412 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.413 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.413 DEBUG:boto:Data: 2023-10-17T18:22:13.413 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.414 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.415 DEBUG:boto:Params: {} 2023-10-17T18:22:13.415 DEBUG:boto:Token: None 2023-10-17T18:22:13.416 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+yzwlHymgq/cq5pfCYEq+mZ9Cg= 2023-10-17T18:22:13.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+yzwlHymgq/cq5pfCYEq+mZ9Cg=', 'Content-Length': '0'} 2023-10-17T18:22:13.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e27fe2e76a17863c-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.419 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.420 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.420 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.421 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.421 DEBUG:boto:Method: GET 2023-10-17T18:22:13.421 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.422 DEBUG:boto:Data: 2023-10-17T18:22:13.422 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.423 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.424 DEBUG:boto:Params: {} 2023-10-17T18:22:13.424 DEBUG:boto:Token: None 2023-10-17T18:22:13.424 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig= 2023-10-17T18:22:13.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig=', 'Content-Length': '0'} 2023-10-17T18:22:13.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000652459cb567685aa-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.428 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.428 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.429 DEBUG:boto:path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.429 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.430 DEBUG:boto:Method: GET 2023-10-17T18:22:13.430 DEBUG:boto:Path: /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.431 DEBUG:boto:Data: 2023-10-17T18:22:13.431 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.431 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.432 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.432 DEBUG:boto:Params: {} 2023-10-17T18:22:13.433 DEBUG:boto:Token: None 2023-10-17T18:22:13.433 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=txzYLw8omWVuA3.LdXpKY6DjMSuzNNt 2023-10-17T18:22:13.434 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig= 2023-10-17T18:22:13.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y1rHtXu0VFKZ2EYKURx2CjjrQig=', 'Content-Length': '0'} 2023-10-17T18:22:13.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'txzYLw8omWVuA3.LdXpKY6DjMSuzNNt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000930425001e241457-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.478 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.479 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.479 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.480 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.480 DEBUG:boto:Method: GET 2023-10-17T18:22:13.480 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.481 DEBUG:boto:Data: 2023-10-17T18:22:13.481 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.482 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.482 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.482 DEBUG:boto:Params: {} 2023-10-17T18:22:13.483 DEBUG:boto:Token: None 2023-10-17T18:22:13.483 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.484 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a4468fdd9a8905-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.487 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.488 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.489 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.489 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.489 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.490 DEBUG:boto:Method: GET 2023-10-17T18:22:13.490 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.491 DEBUG:boto:Data: 2023-10-17T18:22:13.491 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.492 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.492 DEBUG:boto:Params: {} 2023-10-17T18:22:13.493 DEBUG:boto:Token: None 2023-10-17T18:22:13.493 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70fca73df2c586f-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.538 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.539 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.539 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.540 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.540 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.540 DEBUG:boto:Data: 2023-10-17T18:22:13.541 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.542 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.542 DEBUG:boto:Params: {} 2023-10-17T18:22:13.542 DEBUG:boto:Token: None 2023-10-17T18:22:13.543 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.543 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0a09af53dbb2dd4-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.548 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.548 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.549 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.549 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.549 DEBUG:boto:Data: 2023-10-17T18:22:13.550 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.550 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.551 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.551 DEBUG:boto:Params: {} 2023-10-17T18:22:13.551 DEBUG:boto:Token: None 2023-10-17T18:22:13.552 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3630c5229f2de9b-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.557 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.557 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.558 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.558 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.558 DEBUG:boto:Method: GET 2023-10-17T18:22:13.559 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.559 DEBUG:boto:Data: 2023-10-17T18:22:13.560 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.561 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.561 DEBUG:boto:Params: {} 2023-10-17T18:22:13.561 DEBUG:boto:Token: None 2023-10-17T18:22:13.562 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16c79f74bc8b1d1-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.606 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.607 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.607 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.607 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.608 DEBUG:boto:Method: GET 2023-10-17T18:22:13.608 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.609 DEBUG:boto:Data: 2023-10-17T18:22:13.609 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.610 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.610 DEBUG:boto:Params: {} 2023-10-17T18:22:13.611 DEBUG:boto:Token: None 2023-10-17T18:22:13.611 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048472029ca05d93a-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.615 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.615 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.616 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.616 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.617 DEBUG:boto:Method: GET 2023-10-17T18:22:13.617 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.617 DEBUG:boto:Data: 2023-10-17T18:22:13.618 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.619 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.619 DEBUG:boto:Params: {} 2023-10-17T18:22:13.620 DEBUG:boto:Token: None 2023-10-17T18:22:13.620 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090c442afc6ddb6e9-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.624 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.625 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.625 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.626 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.626 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.627 DEBUG:boto:Method: GET 2023-10-17T18:22:13.627 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.627 DEBUG:boto:Data: 2023-10-17T18:22:13.628 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.628 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.629 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.629 DEBUG:boto:Params: {} 2023-10-17T18:22:13.629 DEBUG:boto:Token: None 2023-10-17T18:22:13.630 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.630 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c85c0058feeadc5-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.634 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.635 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.635 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.636 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.636 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.636 DEBUG:boto:Method: GET 2023-10-17T18:22:13.637 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.637 DEBUG:boto:Data: 2023-10-17T18:22:13.638 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.638 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.639 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.639 DEBUG:boto:Params: {} 2023-10-17T18:22:13.640 DEBUG:boto:Token: None 2023-10-17T18:22:13.640 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.640 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc= 2023-10-17T18:22:13.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc=', 'Content-Length': '0'} 2023-10-17T18:22:13.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077b9acc61f170648-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.644 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.644 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.645 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.645 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.645 DEBUG:boto:Method: GET 2023-10-17T18:22:13.646 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.646 DEBUG:boto:Data: 2023-10-17T18:22:13.647 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.647 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.647 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.648 DEBUG:boto:Params: {} 2023-10-17T18:22:13.648 DEBUG:boto:Token: None 2023-10-17T18:22:13.649 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.649 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc= 2023-10-17T18:22:13.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc=', 'Content-Length': '0'} 2023-10-17T18:22:13.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000061585c6c5ad86c4-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.652 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.653 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.653 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.654 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.654 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.654 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.655 DEBUG:boto:Data: 2023-10-17T18:22:13.655 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.655 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.656 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.656 DEBUG:boto:Params: {} 2023-10-17T18:22:13.657 DEBUG:boto:Token: None 2023-10-17T18:22:13.657 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HRS+OfKbepS3+ltBWSOjf7h4B3M= 2023-10-17T18:22:13.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HRS+OfKbepS3+ltBWSOjf7h4B3M=', 'Content-Length': '0'} 2023-10-17T18:22:13.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa15db9159afa836-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.661 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.661 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.662 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.662 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.662 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.663 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.663 DEBUG:boto:Data: 2023-10-17T18:22:13.664 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.664 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.664 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.665 DEBUG:boto:Params: {} 2023-10-17T18:22:13.665 DEBUG:boto:Token: None 2023-10-17T18:22:13.666 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HRS+OfKbepS3+ltBWSOjf7h4B3M= 2023-10-17T18:22:13.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HRS+OfKbepS3+ltBWSOjf7h4B3M=', 'Content-Length': '0'} 2023-10-17T18:22:13.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e323d2277ce8b10a-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.670 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.670 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.670 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.671 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.671 DEBUG:boto:Method: GET 2023-10-17T18:22:13.672 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.672 DEBUG:boto:Data: 2023-10-17T18:22:13.673 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.673 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.674 DEBUG:boto:Params: {} 2023-10-17T18:22:13.674 DEBUG:boto:Token: None 2023-10-17T18:22:13.675 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc= 2023-10-17T18:22:13.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc=', 'Content-Length': '0'} 2023-10-17T18:22:13.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc00a9d02f1a9c65-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.722 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.723 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.723 DEBUG:boto:path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.724 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.724 DEBUG:boto:Method: GET 2023-10-17T18:22:13.725 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.725 DEBUG:boto:Data: 2023-10-17T18:22:13.726 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.727 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.727 DEBUG:boto:Params: {} 2023-10-17T18:22:13.727 DEBUG:boto:Token: None 2023-10-17T18:22:13.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn 2023-10-17T18:22:13.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc= 2023-10-17T18:22:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CP2lKAGCEnREocPAjyS+6f95Hpc=', 'Content-Length': '0'} 2023-10-17T18:22:13.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'OdHYC8pdXrOuCg4RYy64JQ1o2QDyPqn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f0f35eca2fdecf5-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.732 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.732 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.733 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.733 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.734 DEBUG:boto:Method: GET 2023-10-17T18:22:13.734 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.735 DEBUG:boto:Data: 2023-10-17T18:22:13.735 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.735 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.736 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.736 DEBUG:boto:Params: {} 2023-10-17T18:22:13.737 DEBUG:boto:Token: None 2023-10-17T18:22:13.737 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.738 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e3d4d611c66aa8-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.741 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.742 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.743 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.743 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.743 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.744 DEBUG:boto:Method: GET 2023-10-17T18:22:13.744 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.745 DEBUG:boto:Data: 2023-10-17T18:22:13.745 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.746 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.747 DEBUG:boto:Params: {} 2023-10-17T18:22:13.747 DEBUG:boto:Token: None 2023-10-17T18:22:13.748 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ee46b939137372-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.794 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.795 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.795 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.796 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.796 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.797 DEBUG:boto:Data: 2023-10-17T18:22:13.797 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.798 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.798 DEBUG:boto:Params: {} 2023-10-17T18:22:13.799 DEBUG:boto:Token: None 2023-10-17T18:22:13.799 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004818151729c22e31-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.804 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.805 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.805 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.806 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:13.806 DEBUG:boto:Data: 2023-10-17T18:22:13.806 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.807 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.808 DEBUG:boto:Params: {} 2023-10-17T18:22:13.808 DEBUG:boto:Token: None 2023-10-17T18:22:13.808 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj 2023-10-17T18:22:13.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8= 2023-10-17T18:22:13.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wkBsITQMUYPn9Eys4uQ/WNmltr8=', 'Content-Length': '0'} 2023-10-17T18:22:13.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fee9a92cc120c59e-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.814 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.814 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.815 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.815 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.816 DEBUG:boto:Method: GET 2023-10-17T18:22:13.816 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.816 DEBUG:boto:Data: 2023-10-17T18:22:13.817 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.818 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.818 DEBUG:boto:Params: {} 2023-10-17T18:22:13.819 DEBUG:boto:Token: None 2023-10-17T18:22:13.819 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a78f51884628e25f-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.823 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.823 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.824 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.824 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.825 DEBUG:boto:Method: GET 2023-10-17T18:22:13.825 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.826 DEBUG:boto:Data: 2023-10-17T18:22:13.826 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.827 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.827 DEBUG:boto:Params: {} 2023-10-17T18:22:13.828 DEBUG:boto:Token: None 2023-10-17T18:22:13.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:13.829 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E= 2023-10-17T18:22:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fuwAk0qVRhsWo9F0ujsXj686e0E=', 'Content-Length': '0'} 2023-10-17T18:22:13.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0ca10c400ffe72f-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.832 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.832 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.833 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.833 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.834 DEBUG:boto:Method: GET 2023-10-17T18:22:13.834 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.835 DEBUG:boto:Data: 2023-10-17T18:22:13.835 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.836 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.836 DEBUG:boto:Params: {} 2023-10-17T18:22:13.837 DEBUG:boto:Token: None 2023-10-17T18:22:13.837 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000510683764128fd76-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.882 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.883 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.883 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.884 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.884 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.885 DEBUG:boto:Method: GET 2023-10-17T18:22:13.885 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.886 DEBUG:boto:Data: 2023-10-17T18:22:13.886 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.887 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.887 DEBUG:boto:Params: {} 2023-10-17T18:22:13.888 DEBUG:boto:Token: None 2023-10-17T18:22:13.888 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.889 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7ac9ce06c37fda-00652ed0d5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.892 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:13.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.894 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.894 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.895 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.895 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.895 DEBUG:boto:Method: GET 2023-10-17T18:22:13.896 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.896 DEBUG:boto:Data: 2023-10-17T18:22:13.897 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.897 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.898 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.898 DEBUG:boto:Params: {} 2023-10-17T18:22:13.899 DEBUG:boto:Token: None 2023-10-17T18:22:13.899 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.900 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ= 2023-10-17T18:22:13.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ=', 'Content-Length': '0'} 2023-10-17T18:22:13.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089eb7c74dec431d5-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.903 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.903 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.903 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.904 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.904 DEBUG:boto:Method: GET 2023-10-17T18:22:13.905 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.905 DEBUG:boto:Data: 2023-10-17T18:22:13.906 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.906 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.907 DEBUG:boto:Params: {} 2023-10-17T18:22:13.907 DEBUG:boto:Token: None 2023-10-17T18:22:13.908 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ= 2023-10-17T18:22:13.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ=', 'Content-Length': '0'} 2023-10-17T18:22:13.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049d7b234388b715e-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.911 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.912 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.912 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.913 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.913 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.914 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.914 DEBUG:boto:Data: 2023-10-17T18:22:13.914 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.915 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.915 DEBUG:boto:Params: {} 2023-10-17T18:22:13.916 DEBUG:boto:Token: None 2023-10-17T18:22:13.916 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nmJKdg/qg7jx76OGKpv/cTAY9pM= 2023-10-17T18:22:13.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nmJKdg/qg7jx76OGKpv/cTAY9pM=', 'Content-Length': '0'} 2023-10-17T18:22:13.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e87801ea9c60c2-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.920 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.920 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.920 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.921 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.921 DEBUG:boto:Method: HEAD 2023-10-17T18:22:13.922 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.922 DEBUG:boto:Data: 2023-10-17T18:22:13.923 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.923 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.924 DEBUG:boto:Params: {} 2023-10-17T18:22:13.924 DEBUG:boto:Token: None 2023-10-17T18:22:13.925 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.925 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nmJKdg/qg7jx76OGKpv/cTAY9pM= 2023-10-17T18:22:13.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nmJKdg/qg7jx76OGKpv/cTAY9pM=', 'Content-Length': '0'} 2023-10-17T18:22:13.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a6c8b872ec9159c-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:13.929 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.929 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.930 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.930 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.930 DEBUG:boto:Method: GET 2023-10-17T18:22:13.931 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.931 DEBUG:boto:Data: 2023-10-17T18:22:13.932 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.932 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.933 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.933 DEBUG:boto:Params: {} 2023-10-17T18:22:13.934 DEBUG:boto:Token: None 2023-10-17T18:22:13.934 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.935 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ= 2023-10-17T18:22:13.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ=', 'Content-Length': '0'} 2023-10-17T18:22:13.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027e4dc1c24caa04f-00652ed0d5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.938 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:13.939 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:13.939 DEBUG:boto:path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.940 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.940 DEBUG:boto:Method: GET 2023-10-17T18:22:13.940 DEBUG:boto:Path: /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.941 DEBUG:boto:Data: 2023-10-17T18:22:13.941 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:13.942 DEBUG:boto:Port: 8000 2023-10-17T18:22:13.943 DEBUG:boto:Params: {} 2023-10-17T18:22:13.943 DEBUG:boto:Token: None 2023-10-17T18:22:13.944 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/obj?versionId=rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz 2023-10-17T18:22:13.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ= 2023-10-17T18:22:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LVxlFI5VLDar2P0ztMZOLLfMIwQ=', 'Content-Length': '0'} 2023-10-17T18:22:13.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'rG80Z6RP5ssXNPx6txr-pL1IEj-Iboz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000514c6586180a44b8-00652ed0d5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:13.990 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:13.991 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:13.991 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.992 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:13.992 DEBUG:boto:Method: GET 2023-10-17T18:22:13.992 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:13.993 DEBUG:boto:Data: 2023-10-17T18:22:13.993 DEBUG:boto:Headers: {} 2023-10-17T18:22:13.994 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:13.994 DEBUG:boto:Port: 8001 2023-10-17T18:22:13.994 DEBUG:boto:Params: {} 2023-10-17T18:22:13.995 DEBUG:boto:Token: None 2023-10-17T18:22:13.995 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:13 GMT /hbmuog-13/ 2023-10-17T18:22:13.996 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY= 2023-10-17T18:22:13.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jtwOBGdy3k0h5otmTo8FMjyB9IY=', 'Content-Length': '0'} 2023-10-17T18:22:13.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641789c27a4e08a2-00652ed0d5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.000 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.001 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.002 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.002 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.003 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.003 DEBUG:boto:Method: GET 2023-10-17T18:22:14.004 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.004 DEBUG:boto:Data: 2023-10-17T18:22:14.005 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.005 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.006 DEBUG:boto:Params: {} 2023-10-17T18:22:14.006 DEBUG:boto:Token: None 2023-10-17T18:22:14.007 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.008 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585722549fc2ccc8-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.011 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.012 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.012 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.013 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.013 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.013 DEBUG:boto:Data: 2023-10-17T18:22:14.014 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.014 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.015 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.015 DEBUG:boto:Params: {} 2023-10-17T18:22:14.016 DEBUG:boto:Token: None 2023-10-17T18:22:14.016 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5d7e6104b265e1b-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.022 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.022 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.023 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.023 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.024 DEBUG:boto:Data: 2023-10-17T18:22:14.024 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.024 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.025 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.025 DEBUG:boto:Params: {} 2023-10-17T18:22:14.026 DEBUG:boto:Token: None 2023-10-17T18:22:14.026 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.027 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074f7e50d29df9cfa-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.031 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.032 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.032 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.033 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.033 DEBUG:boto:Method: GET 2023-10-17T18:22:14.033 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.034 DEBUG:boto:Data: 2023-10-17T18:22:14.035 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.036 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.037 DEBUG:boto:Params: {} 2023-10-17T18:22:14.037 DEBUG:boto:Token: None 2023-10-17T18:22:14.038 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.039 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1c982e6c23f7373-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.082 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.083 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.083 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.084 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.084 DEBUG:boto:Method: GET 2023-10-17T18:22:14.084 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.085 DEBUG:boto:Data: 2023-10-17T18:22:14.085 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.085 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.086 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.086 DEBUG:boto:Params: {} 2023-10-17T18:22:14.087 DEBUG:boto:Token: None 2023-10-17T18:22:14.087 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.088 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000372cd2dd62dd086b-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.091 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.092 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.092 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.092 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.093 DEBUG:boto:Method: GET 2023-10-17T18:22:14.093 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.094 DEBUG:boto:Data: 2023-10-17T18:22:14.094 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.094 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.095 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.095 DEBUG:boto:Params: {} 2023-10-17T18:22:14.096 DEBUG:boto:Token: None 2023-10-17T18:22:14.096 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.097 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d8ed92e446e0352-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.100 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.101 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.102 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.102 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.102 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.103 DEBUG:boto:Method: GET 2023-10-17T18:22:14.103 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.104 DEBUG:boto:Data: 2023-10-17T18:22:14.104 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.104 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.105 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.105 DEBUG:boto:Params: {} 2023-10-17T18:22:14.106 DEBUG:boto:Token: None 2023-10-17T18:22:14.106 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.107 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005666927e769e7349-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.150 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.151 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.152 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.152 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.153 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.153 DEBUG:boto:Method: GET 2023-10-17T18:22:14.154 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.154 DEBUG:boto:Data: 2023-10-17T18:22:14.155 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.155 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.156 DEBUG:boto:Params: {} 2023-10-17T18:22:14.156 DEBUG:boto:Token: None 2023-10-17T18:22:14.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY= 2023-10-17T18:22:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY=', 'Content-Length': '0'} 2023-10-17T18:22:14.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004447dbc4823d25e-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.160 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.161 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.161 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.162 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.162 DEBUG:boto:Method: GET 2023-10-17T18:22:14.163 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.163 DEBUG:boto:Data: 2023-10-17T18:22:14.163 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.164 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.165 DEBUG:boto:Params: {} 2023-10-17T18:22:14.165 DEBUG:boto:Token: None 2023-10-17T18:22:14.166 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY= 2023-10-17T18:22:14.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY=', 'Content-Length': '0'} 2023-10-17T18:22:14.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69a1a6c2413afb5-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.169 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.170 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.170 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.171 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.171 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.171 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.172 DEBUG:boto:Data: 2023-10-17T18:22:14.172 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.173 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.173 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.173 DEBUG:boto:Params: {} 2023-10-17T18:22:14.174 DEBUG:boto:Token: None 2023-10-17T18:22:14.174 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.175 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cvVGF7/wMxhBfTZ/V38CJ+XSfwQ= 2023-10-17T18:22:14.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cvVGF7/wMxhBfTZ/V38CJ+XSfwQ=', 'Content-Length': '0'} 2023-10-17T18:22:14.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4df6d22bd718de6-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.178 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.178 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.179 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.179 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.179 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.180 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.180 DEBUG:boto:Data: 2023-10-17T18:22:14.181 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.181 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.181 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.182 DEBUG:boto:Params: {} 2023-10-17T18:22:14.182 DEBUG:boto:Token: None 2023-10-17T18:22:14.183 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cvVGF7/wMxhBfTZ/V38CJ+XSfwQ= 2023-10-17T18:22:14.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cvVGF7/wMxhBfTZ/V38CJ+XSfwQ=', 'Content-Length': '0'} 2023-10-17T18:22:14.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f738c90a054e0a9c-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.188 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.189 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.189 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.190 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.190 DEBUG:boto:Method: GET 2023-10-17T18:22:14.191 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.191 DEBUG:boto:Data: 2023-10-17T18:22:14.192 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.192 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.193 DEBUG:boto:Params: {} 2023-10-17T18:22:14.193 DEBUG:boto:Token: None 2023-10-17T18:22:14.194 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY= 2023-10-17T18:22:14.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY=', 'Content-Length': '0'} 2023-10-17T18:22:14.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed3ae78c378757f0-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.238 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.239 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.239 DEBUG:boto:path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.240 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.240 DEBUG:boto:Method: GET 2023-10-17T18:22:14.241 DEBUG:boto:Path: /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.241 DEBUG:boto:Data: 2023-10-17T18:22:14.241 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.242 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.243 DEBUG:boto:Params: {} 2023-10-17T18:22:14.243 DEBUG:boto:Token: None 2023-10-17T18:22:14.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=.pPFaokTIv6RMPRkCpuToO8yY29mate 2023-10-17T18:22:14.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY= 2023-10-17T18:22:14.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xQPxXxShm41ZOAMFIXwbhn3dRY=', 'Content-Length': '0'} 2023-10-17T18:22:14.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '.pPFaokTIv6RMPRkCpuToO8yY29mate'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d461f77785ff0c7-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.247 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.248 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.248 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.249 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.249 DEBUG:boto:Method: GET 2023-10-17T18:22:14.250 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.250 DEBUG:boto:Data: 2023-10-17T18:22:14.251 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.251 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.252 DEBUG:boto:Params: {} 2023-10-17T18:22:14.252 DEBUG:boto:Token: None 2023-10-17T18:22:14.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb979c616f0e7b7-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.302 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.303 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.303 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.304 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.304 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.305 DEBUG:boto:Method: GET 2023-10-17T18:22:14.305 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.305 DEBUG:boto:Data: 2023-10-17T18:22:14.306 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.306 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.307 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.307 DEBUG:boto:Params: {} 2023-10-17T18:22:14.307 DEBUG:boto:Token: None 2023-10-17T18:22:14.308 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.308 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f535d6c36e8ef3d-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.312 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.313 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.313 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.313 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.314 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.314 DEBUG:boto:Data: 2023-10-17T18:22:14.315 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.315 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.315 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.316 DEBUG:boto:Params: {} 2023-10-17T18:22:14.316 DEBUG:boto:Token: None 2023-10-17T18:22:14.317 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.317 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaeb031c2a88f699-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.321 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.322 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.322 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.323 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.323 DEBUG:boto:Data: 2023-10-17T18:22:14.324 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.324 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.324 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.325 DEBUG:boto:Params: {} 2023-10-17T18:22:14.325 DEBUG:boto:Token: None 2023-10-17T18:22:14.326 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.326 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000761a219b02c1c930-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.331 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.331 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.332 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.332 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.332 DEBUG:boto:Method: GET 2023-10-17T18:22:14.333 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.333 DEBUG:boto:Data: 2023-10-17T18:22:14.334 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.334 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.335 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.335 DEBUG:boto:Params: {} 2023-10-17T18:22:14.336 DEBUG:boto:Token: None 2023-10-17T18:22:14.336 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3c63c94c850359a-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.382 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.383 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.383 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.384 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.384 DEBUG:boto:Method: GET 2023-10-17T18:22:14.385 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.385 DEBUG:boto:Data: 2023-10-17T18:22:14.386 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.386 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.390 DEBUG:boto:Params: {} 2023-10-17T18:22:14.391 DEBUG:boto:Token: None 2023-10-17T18:22:14.391 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004498f1bd97eb263c-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.395 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.396 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.396 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.396 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.397 DEBUG:boto:Method: GET 2023-10-17T18:22:14.397 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.398 DEBUG:boto:Data: 2023-10-17T18:22:14.398 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.399 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.400 DEBUG:boto:Params: {} 2023-10-17T18:22:14.401 DEBUG:boto:Token: None 2023-10-17T18:22:14.401 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd98e706aca9383-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.446 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.447 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.448 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.448 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.449 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.449 DEBUG:boto:Method: GET 2023-10-17T18:22:14.450 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.450 DEBUG:boto:Data: 2023-10-17T18:22:14.451 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.451 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.452 DEBUG:boto:Params: {} 2023-10-17T18:22:14.452 DEBUG:boto:Token: None 2023-10-17T18:22:14.453 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017060bf6bfdf33aa-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.457 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.459 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.459 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.459 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.460 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.460 DEBUG:boto:Method: GET 2023-10-17T18:22:14.461 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.461 DEBUG:boto:Data: 2023-10-17T18:22:14.461 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.462 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.463 DEBUG:boto:Params: {} 2023-10-17T18:22:14.463 DEBUG:boto:Token: None 2023-10-17T18:22:14.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU= 2023-10-17T18:22:14.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU=', 'Content-Length': '0'} 2023-10-17T18:22:14.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000920f0c66ebb4ac22-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.467 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.468 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.468 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.468 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.469 DEBUG:boto:Method: GET 2023-10-17T18:22:14.469 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.470 DEBUG:boto:Data: 2023-10-17T18:22:14.470 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.471 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.472 DEBUG:boto:Params: {} 2023-10-17T18:22:14.472 DEBUG:boto:Token: None 2023-10-17T18:22:14.473 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU= 2023-10-17T18:22:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU=', 'Content-Length': '0'} 2023-10-17T18:22:14.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000833b76e9dd0c2e25-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.476 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.477 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.477 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.478 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.478 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.478 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.479 DEBUG:boto:Data: 2023-10-17T18:22:14.479 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.480 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.480 DEBUG:boto:Params: {} 2023-10-17T18:22:14.481 DEBUG:boto:Token: None 2023-10-17T18:22:14.481 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ULiTNlMhICTXrjeqocvgkYVuk7o= 2023-10-17T18:22:14.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ULiTNlMhICTXrjeqocvgkYVuk7o=', 'Content-Length': '0'} 2023-10-17T18:22:14.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004450913279981cc4-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.484 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.485 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.485 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.486 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.486 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.487 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.487 DEBUG:boto:Data: 2023-10-17T18:22:14.487 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.488 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.488 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.488 DEBUG:boto:Params: {} 2023-10-17T18:22:14.489 DEBUG:boto:Token: None 2023-10-17T18:22:14.489 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ULiTNlMhICTXrjeqocvgkYVuk7o= 2023-10-17T18:22:14.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ULiTNlMhICTXrjeqocvgkYVuk7o=', 'Content-Length': '0'} 2023-10-17T18:22:14.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000667ec24a7e11fa69-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.493 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.494 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.494 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.495 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.495 DEBUG:boto:Method: GET 2023-10-17T18:22:14.495 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.496 DEBUG:boto:Data: 2023-10-17T18:22:14.496 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.497 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.497 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.498 DEBUG:boto:Params: {} 2023-10-17T18:22:14.498 DEBUG:boto:Token: None 2023-10-17T18:22:14.499 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.499 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU= 2023-10-17T18:22:14.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU=', 'Content-Length': '0'} 2023-10-17T18:22:14.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003268935a95a29ea9-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.502 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.503 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.503 DEBUG:boto:path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.504 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.504 DEBUG:boto:Method: GET 2023-10-17T18:22:14.504 DEBUG:boto:Path: /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.505 DEBUG:boto:Data: 2023-10-17T18:22:14.505 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.506 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.506 DEBUG:boto:Params: {} 2023-10-17T18:22:14.507 DEBUG:boto:Token: None 2023-10-17T18:22:14.507 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG 2023-10-17T18:22:14.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU= 2023-10-17T18:22:14.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iZnSTV5gnXQWB/YC2SpK3uP8zAU=', 'Content-Length': '0'} 2023-10-17T18:22:14.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'EpUZ.-OLSUZcByZDwOabzUO9OaVD.KG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005572ec0582006855-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.511 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.512 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.512 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.512 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.513 DEBUG:boto:Method: GET 2023-10-17T18:22:14.513 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.514 DEBUG:boto:Data: 2023-10-17T18:22:14.514 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.515 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.516 DEBUG:boto:Params: {} 2023-10-17T18:22:14.516 DEBUG:boto:Token: None 2023-10-17T18:22:14.517 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2c900efe9988e7c-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.521 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.522 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.522 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.523 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.523 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.524 DEBUG:boto:Method: GET 2023-10-17T18:22:14.524 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.525 DEBUG:boto:Data: 2023-10-17T18:22:14.525 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.526 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.526 DEBUG:boto:Params: {} 2023-10-17T18:22:14.527 DEBUG:boto:Token: None 2023-10-17T18:22:14.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048274ae2f6af1f39-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.574 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.575 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.575 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.576 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.576 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.577 DEBUG:boto:Data: 2023-10-17T18:22:14.577 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.578 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.578 DEBUG:boto:Params: {} 2023-10-17T18:22:14.579 DEBUG:boto:Token: None 2023-10-17T18:22:14.579 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b9bf05c67138044-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.584 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.585 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.585 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.586 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.586 DEBUG:boto:Data: 2023-10-17T18:22:14.586 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.587 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.588 DEBUG:boto:Params: {} 2023-10-17T18:22:14.588 DEBUG:boto:Token: None 2023-10-17T18:22:14.588 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011cc5e1139367977-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.594 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.594 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.595 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.595 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.596 DEBUG:boto:Method: GET 2023-10-17T18:22:14.596 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.597 DEBUG:boto:Data: 2023-10-17T18:22:14.597 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.598 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.599 DEBUG:boto:Params: {} 2023-10-17T18:22:14.599 DEBUG:boto:Token: None 2023-10-17T18:22:14.600 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0cad01dc321f325-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.646 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.647 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.648 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.648 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.648 DEBUG:boto:Method: GET 2023-10-17T18:22:14.649 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.649 DEBUG:boto:Data: 2023-10-17T18:22:14.650 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.651 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.651 DEBUG:boto:Params: {} 2023-10-17T18:22:14.652 DEBUG:boto:Token: None 2023-10-17T18:22:14.652 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000247af68ef7b71b0d-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.656 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.657 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.657 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.657 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.658 DEBUG:boto:Method: GET 2023-10-17T18:22:14.658 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.659 DEBUG:boto:Data: 2023-10-17T18:22:14.659 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.660 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.660 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.661 DEBUG:boto:Params: {} 2023-10-17T18:22:14.661 DEBUG:boto:Token: None 2023-10-17T18:22:14.662 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.662 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ab5b905b673b80-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.666 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.667 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.667 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.668 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.668 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.668 DEBUG:boto:Method: GET 2023-10-17T18:22:14.669 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.669 DEBUG:boto:Data: 2023-10-17T18:22:14.670 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.671 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.671 DEBUG:boto:Params: {} 2023-10-17T18:22:14.672 DEBUG:boto:Token: None 2023-10-17T18:22:14.672 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5dfdf85e21b9a9a-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.676 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.677 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.678 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.678 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.679 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.679 DEBUG:boto:Method: GET 2023-10-17T18:22:14.680 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.680 DEBUG:boto:Data: 2023-10-17T18:22:14.680 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.681 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.682 DEBUG:boto:Params: {} 2023-10-17T18:22:14.682 DEBUG:boto:Token: None 2023-10-17T18:22:14.683 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M= 2023-10-17T18:22:14.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M=', 'Content-Length': '0'} 2023-10-17T18:22:14.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063e9c1c2ffba7a18-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.686 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.687 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.687 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.688 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.688 DEBUG:boto:Method: GET 2023-10-17T18:22:14.689 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.689 DEBUG:boto:Data: 2023-10-17T18:22:14.690 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.690 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.691 DEBUG:boto:Params: {} 2023-10-17T18:22:14.691 DEBUG:boto:Token: None 2023-10-17T18:22:14.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M= 2023-10-17T18:22:14.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M=', 'Content-Length': '0'} 2023-10-17T18:22:14.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000729e603cad0884cc-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.696 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.696 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.696 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.697 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.697 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.698 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.698 DEBUG:boto:Data: 2023-10-17T18:22:14.698 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.699 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.700 DEBUG:boto:Params: {} 2023-10-17T18:22:14.700 DEBUG:boto:Token: None 2023-10-17T18:22:14.701 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CO7PLHQn1hrtV0pVAEtMYHnNVTE= 2023-10-17T18:22:14.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CO7PLHQn1hrtV0pVAEtMYHnNVTE=', 'Content-Length': '0'} 2023-10-17T18:22:14.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007f6154b3a07d27e-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.704 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.704 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.705 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.705 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.706 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.706 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.707 DEBUG:boto:Data: 2023-10-17T18:22:14.707 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.708 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.708 DEBUG:boto:Params: {} 2023-10-17T18:22:14.709 DEBUG:boto:Token: None 2023-10-17T18:22:14.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CO7PLHQn1hrtV0pVAEtMYHnNVTE= 2023-10-17T18:22:14.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CO7PLHQn1hrtV0pVAEtMYHnNVTE=', 'Content-Length': '0'} 2023-10-17T18:22:14.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063416bd1ab170d1c-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.713 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.714 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.714 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.714 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.715 DEBUG:boto:Method: GET 2023-10-17T18:22:14.715 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.716 DEBUG:boto:Data: 2023-10-17T18:22:14.716 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.717 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.717 DEBUG:boto:Params: {} 2023-10-17T18:22:14.718 DEBUG:boto:Token: None 2023-10-17T18:22:14.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M= 2023-10-17T18:22:14.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M=', 'Content-Length': '0'} 2023-10-17T18:22:14.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037e921b875a5e57a-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.722 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.722 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.723 DEBUG:boto:path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.723 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.724 DEBUG:boto:Method: GET 2023-10-17T18:22:14.724 DEBUG:boto:Path: /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.724 DEBUG:boto:Data: 2023-10-17T18:22:14.725 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.725 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.726 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.726 DEBUG:boto:Params: {} 2023-10-17T18:22:14.727 DEBUG:boto:Token: None 2023-10-17T18:22:14.727 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3 2023-10-17T18:22:14.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M= 2023-10-17T18:22:14.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DtkOv8t93NuIlJOD3Vhp0wF8L8M=', 'Content-Length': '0'} 2023-10-17T18:22:14.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '7ls-p2mgkRNuF.TFyuDyT03O.hWrxO3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69c6466509841a1-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.774 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.775 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.775 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.776 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.776 DEBUG:boto:Method: GET 2023-10-17T18:22:14.777 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.777 DEBUG:boto:Data: 2023-10-17T18:22:14.778 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.778 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.779 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.779 DEBUG:boto:Params: {} 2023-10-17T18:22:14.779 DEBUG:boto:Token: None 2023-10-17T18:22:14.780 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.780 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5d3c9064f19eb9c-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.784 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.785 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.785 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.786 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.786 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.786 DEBUG:boto:Method: GET 2023-10-17T18:22:14.787 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.787 DEBUG:boto:Data: 2023-10-17T18:22:14.788 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.789 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.789 DEBUG:boto:Params: {} 2023-10-17T18:22:14.789 DEBUG:boto:Token: None 2023-10-17T18:22:14.790 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f636e3a5425f4d07-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.794 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.795 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.795 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.796 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.796 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.797 DEBUG:boto:Data: 2023-10-17T18:22:14.797 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.798 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.798 DEBUG:boto:Params: {} 2023-10-17T18:22:14.799 DEBUG:boto:Token: None 2023-10-17T18:22:14.799 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a009d97d4d26867a-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.804 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.804 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.805 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.805 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:14.806 DEBUG:boto:Data: 2023-10-17T18:22:14.806 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.807 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.807 DEBUG:boto:Params: {} 2023-10-17T18:22:14.808 DEBUG:boto:Token: None 2023-10-17T18:22:14.808 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj 2023-10-17T18:22:14.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE= 2023-10-17T18:22:14.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AiDIH4qHMd+pXgZkZ9mEUwBtsNE=', 'Content-Length': '0'} 2023-10-17T18:22:14.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033d75c1fb0657813-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.813 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.814 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.814 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.814 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.815 DEBUG:boto:Method: GET 2023-10-17T18:22:14.815 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.816 DEBUG:boto:Data: 2023-10-17T18:22:14.816 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.816 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.817 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.817 DEBUG:boto:Params: {} 2023-10-17T18:22:14.818 DEBUG:boto:Token: None 2023-10-17T18:22:14.818 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000002bebdb8663c1c6-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.862 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.863 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.863 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.864 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.864 DEBUG:boto:Method: GET 2023-10-17T18:22:14.865 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.865 DEBUG:boto:Data: 2023-10-17T18:22:14.866 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.866 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.867 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.867 DEBUG:boto:Params: {} 2023-10-17T18:22:14.867 DEBUG:boto:Token: None 2023-10-17T18:22:14.868 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:14.868 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo= 2023-10-17T18:22:14.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jXJxSOCEwHVq+h0F/L5lXXHLPuo=', 'Content-Length': '0'} 2023-10-17T18:22:14.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d5bd8b49c72ba17-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.872 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.872 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.873 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.873 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.874 DEBUG:boto:Method: GET 2023-10-17T18:22:14.874 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.875 DEBUG:boto:Data: 2023-10-17T18:22:14.876 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.876 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.877 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.877 DEBUG:boto:Params: {} 2023-10-17T18:22:14.878 DEBUG:boto:Token: None 2023-10-17T18:22:14.878 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.879 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003fa528fbe9b91b5-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.883 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.884 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.885 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.885 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.885 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.886 DEBUG:boto:Method: GET 2023-10-17T18:22:14.886 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.887 DEBUG:boto:Data: 2023-10-17T18:22:14.887 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.888 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.888 DEBUG:boto:Params: {} 2023-10-17T18:22:14.889 DEBUG:boto:Token: None 2023-10-17T18:22:14.889 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:14.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058270af0f5e3ddae-00652ed0d6-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.894 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:14.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.895 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.895 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.896 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.896 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.897 DEBUG:boto:Method: GET 2023-10-17T18:22:14.897 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.897 DEBUG:boto:Data: 2023-10-17T18:22:14.898 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.898 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.899 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.899 DEBUG:boto:Params: {} 2023-10-17T18:22:14.900 DEBUG:boto:Token: None 2023-10-17T18:22:14.900 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o= 2023-10-17T18:22:14.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o=', 'Content-Length': '0'} 2023-10-17T18:22:14.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8de13eef467ca7d-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.904 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.905 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.905 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.905 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.906 DEBUG:boto:Method: GET 2023-10-17T18:22:14.906 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.907 DEBUG:boto:Data: 2023-10-17T18:22:14.907 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.908 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.908 DEBUG:boto:Params: {} 2023-10-17T18:22:14.909 DEBUG:boto:Token: None 2023-10-17T18:22:14.909 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o= 2023-10-17T18:22:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o=', 'Content-Length': '0'} 2023-10-17T18:22:14.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005f5fb02188cdc57-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.913 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.913 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.914 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.914 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.915 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.915 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.916 DEBUG:boto:Data: 2023-10-17T18:22:14.916 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.917 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.917 DEBUG:boto:Params: {} 2023-10-17T18:22:14.917 DEBUG:boto:Token: None 2023-10-17T18:22:14.918 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.918 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nWu1YrFVczTvb9njt7ueY75vvu0= 2023-10-17T18:22:14.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWu1YrFVczTvb9njt7ueY75vvu0=', 'Content-Length': '0'} 2023-10-17T18:22:14.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d927f157360ef907-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.921 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.922 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.922 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.923 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.923 DEBUG:boto:Method: HEAD 2023-10-17T18:22:14.923 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.924 DEBUG:boto:Data: 2023-10-17T18:22:14.924 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.924 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.925 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.925 DEBUG:boto:Params: {} 2023-10-17T18:22:14.926 DEBUG:boto:Token: None 2023-10-17T18:22:14.926 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.926 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nWu1YrFVczTvb9njt7ueY75vvu0= 2023-10-17T18:22:14.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWu1YrFVczTvb9njt7ueY75vvu0=', 'Content-Length': '0'} 2023-10-17T18:22:14.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050a418d7c8270fc0-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:14.930 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.930 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.931 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.931 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.932 DEBUG:boto:Method: GET 2023-10-17T18:22:14.932 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.933 DEBUG:boto:Data: 2023-10-17T18:22:14.933 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.934 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.935 DEBUG:boto:Params: {} 2023-10-17T18:22:14.935 DEBUG:boto:Token: None 2023-10-17T18:22:14.935 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o= 2023-10-17T18:22:14.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o=', 'Content-Length': '0'} 2023-10-17T18:22:14.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d41f1f05413865-00652ed0d6-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.982 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:14.983 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:14.983 DEBUG:boto:path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.984 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.984 DEBUG:boto:Method: GET 2023-10-17T18:22:14.985 DEBUG:boto:Path: /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.985 DEBUG:boto:Data: 2023-10-17T18:22:14.986 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:14.986 DEBUG:boto:Port: 8000 2023-10-17T18:22:14.987 DEBUG:boto:Params: {} 2023-10-17T18:22:14.987 DEBUG:boto:Token: None 2023-10-17T18:22:14.988 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/obj?versionId=w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9 2023-10-17T18:22:14.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o= 2023-10-17T18:22:14.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq0ndbNffCE1bJDcS/wIj9FIT8o=', 'Content-Length': '0'} 2023-10-17T18:22:14.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'w3lLb1lmq4GvkGFK0XRgyJqunVLVxB9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a57b38736383e12b-00652ed0d6-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:14.991 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:14.992 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:14.992 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.993 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:14.993 DEBUG:boto:Method: GET 2023-10-17T18:22:14.993 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:14.994 DEBUG:boto:Data: 2023-10-17T18:22:14.994 DEBUG:boto:Headers: {} 2023-10-17T18:22:14.995 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:14.995 DEBUG:boto:Port: 8001 2023-10-17T18:22:14.995 DEBUG:boto:Params: {} 2023-10-17T18:22:14.996 DEBUG:boto:Token: None 2023-10-17T18:22:14.996 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:14 GMT /hbmuog-13/ 2023-10-17T18:22:14.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY= 2023-10-17T18:22:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NC02ZBPDw/0OEprzEWb75C7azIY=', 'Content-Length': '0'} 2023-10-17T18:22:15.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e0f8be005487aa-00652ed0d6-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.002 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.003 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.003 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.004 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.004 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.005 DEBUG:boto:Method: GET 2023-10-17T18:22:15.005 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.005 DEBUG:boto:Data: 2023-10-17T18:22:15.006 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.006 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.007 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.007 DEBUG:boto:Params: {} 2023-10-17T18:22:15.008 DEBUG:boto:Token: None 2023-10-17T18:22:15.008 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.009 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000153746096ad4867-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.054 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.055 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.055 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.056 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.056 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.057 DEBUG:boto:Data: 2023-10-17T18:22:15.057 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.058 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.058 DEBUG:boto:Params: {} 2023-10-17T18:22:15.059 DEBUG:boto:Token: None 2023-10-17T18:22:15.059 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b32016d7330d64a-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.064 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.064 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.065 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.065 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.065 DEBUG:boto:Data: 2023-10-17T18:22:15.066 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.067 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.067 DEBUG:boto:Params: {} 2023-10-17T18:22:15.067 DEBUG:boto:Token: None 2023-10-17T18:22:15.068 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000528a9d564a9619c4-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.073 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.073 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.074 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.074 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.074 DEBUG:boto:Method: GET 2023-10-17T18:22:15.075 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.075 DEBUG:boto:Data: 2023-10-17T18:22:15.076 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.076 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.077 DEBUG:boto:Params: {} 2023-10-17T18:22:15.077 DEBUG:boto:Token: None 2023-10-17T18:22:15.078 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a308c6419dff87da-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.122 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.123 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.123 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.124 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.124 DEBUG:boto:Method: GET 2023-10-17T18:22:15.124 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.125 DEBUG:boto:Data: 2023-10-17T18:22:15.125 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.126 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.126 DEBUG:boto:Params: {} 2023-10-17T18:22:15.127 DEBUG:boto:Token: None 2023-10-17T18:22:15.127 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053a6c6af85e299a1-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.131 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.131 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.132 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.132 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.133 DEBUG:boto:Method: GET 2023-10-17T18:22:15.133 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.133 DEBUG:boto:Data: 2023-10-17T18:22:15.134 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.135 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.135 DEBUG:boto:Params: {} 2023-10-17T18:22:15.135 DEBUG:boto:Token: None 2023-10-17T18:22:15.136 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f620ff6d08426b1b-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.140 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.141 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.141 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.141 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.142 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.142 DEBUG:boto:Method: GET 2023-10-17T18:22:15.143 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.143 DEBUG:boto:Data: 2023-10-17T18:22:15.144 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.144 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.145 DEBUG:boto:Params: {} 2023-10-17T18:22:15.145 DEBUG:boto:Token: None 2023-10-17T18:22:15.146 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb733c56076deff-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.150 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.151 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.151 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.152 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.152 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.153 DEBUG:boto:Method: GET 2023-10-17T18:22:15.153 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.153 DEBUG:boto:Data: 2023-10-17T18:22:15.154 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.155 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.155 DEBUG:boto:Params: {} 2023-10-17T18:22:15.156 DEBUG:boto:Token: None 2023-10-17T18:22:15.156 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc= 2023-10-17T18:22:15.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc=', 'Content-Length': '0'} 2023-10-17T18:22:15.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0c361b2937c5e31-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.160 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.161 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.161 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.161 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.162 DEBUG:boto:Method: GET 2023-10-17T18:22:15.162 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.163 DEBUG:boto:Data: 2023-10-17T18:22:15.163 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.164 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.164 DEBUG:boto:Params: {} 2023-10-17T18:22:15.165 DEBUG:boto:Token: None 2023-10-17T18:22:15.165 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc= 2023-10-17T18:22:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc=', 'Content-Length': '0'} 2023-10-17T18:22:15.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b574e2fa43f2933a-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.169 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.170 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.170 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.171 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.171 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.172 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.172 DEBUG:boto:Data: 2023-10-17T18:22:15.172 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.173 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.173 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.174 DEBUG:boto:Params: {} 2023-10-17T18:22:15.174 DEBUG:boto:Token: None 2023-10-17T18:22:15.175 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.175 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EKUq+8fDQKjOmPZQKsH+mpZ2iME= 2023-10-17T18:22:15.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EKUq+8fDQKjOmPZQKsH+mpZ2iME=', 'Content-Length': '0'} 2023-10-17T18:22:15.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000983cdf45a7dbc3ef-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.178 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.178 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.179 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.179 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.180 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.180 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.180 DEBUG:boto:Data: 2023-10-17T18:22:15.181 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.181 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.181 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.182 DEBUG:boto:Params: {} 2023-10-17T18:22:15.182 DEBUG:boto:Token: None 2023-10-17T18:22:15.183 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EKUq+8fDQKjOmPZQKsH+mpZ2iME= 2023-10-17T18:22:15.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EKUq+8fDQKjOmPZQKsH+mpZ2iME=', 'Content-Length': '0'} 2023-10-17T18:22:15.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033ebf66a414e72d7-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.187 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.187 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.188 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.188 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.189 DEBUG:boto:Method: GET 2023-10-17T18:22:15.189 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.190 DEBUG:boto:Data: 2023-10-17T18:22:15.190 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.191 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.191 DEBUG:boto:Params: {} 2023-10-17T18:22:15.192 DEBUG:boto:Token: None 2023-10-17T18:22:15.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc= 2023-10-17T18:22:15.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc=', 'Content-Length': '0'} 2023-10-17T18:22:15.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000510d562340f06a7f-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.238 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.239 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.239 DEBUG:boto:path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.240 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.240 DEBUG:boto:Method: GET 2023-10-17T18:22:15.240 DEBUG:boto:Path: /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.241 DEBUG:boto:Data: 2023-10-17T18:22:15.241 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.242 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.243 DEBUG:boto:Params: {} 2023-10-17T18:22:15.243 DEBUG:boto:Token: None 2023-10-17T18:22:15.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=s3dLv7gwdXVgtTA6evp.aG64TtsPNjE 2023-10-17T18:22:15.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc= 2023-10-17T18:22:15.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o2rMjffj7PvmMbMKVGuiPrz2tAc=', 'Content-Length': '0'} 2023-10-17T18:22:15.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 's3dLv7gwdXVgtTA6evp.aG64TtsPNjE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019a7bf8c4182a014-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.247 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.248 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.248 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.249 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.249 DEBUG:boto:Method: GET 2023-10-17T18:22:15.250 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.250 DEBUG:boto:Data: 2023-10-17T18:22:15.250 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.251 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.252 DEBUG:boto:Params: {} 2023-10-17T18:22:15.252 DEBUG:boto:Token: None 2023-10-17T18:22:15.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae378a46c7dccc0-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.257 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.257 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.258 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.258 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.259 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.259 DEBUG:boto:Method: GET 2023-10-17T18:22:15.260 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.260 DEBUG:boto:Data: 2023-10-17T18:22:15.260 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.261 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.261 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.262 DEBUG:boto:Params: {} 2023-10-17T18:22:15.262 DEBUG:boto:Token: None 2023-10-17T18:22:15.263 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08cfb2ded4c8bb4-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.310 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.311 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.311 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.312 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.312 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.312 DEBUG:boto:Data: 2023-10-17T18:22:15.313 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.314 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.314 DEBUG:boto:Params: {} 2023-10-17T18:22:15.314 DEBUG:boto:Token: None 2023-10-17T18:22:15.315 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aaf38f980ced3b3-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.320 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.320 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.320 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.321 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.321 DEBUG:boto:Data: 2023-10-17T18:22:15.321 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.322 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.323 DEBUG:boto:Params: {} 2023-10-17T18:22:15.323 DEBUG:boto:Token: None 2023-10-17T18:22:15.323 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a5a2013dc79371d-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.328 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.329 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.329 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.330 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.330 DEBUG:boto:Method: GET 2023-10-17T18:22:15.330 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.331 DEBUG:boto:Data: 2023-10-17T18:22:15.331 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.332 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.332 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.332 DEBUG:boto:Params: {} 2023-10-17T18:22:15.333 DEBUG:boto:Token: None 2023-10-17T18:22:15.333 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7f17804dc3a5c60-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.378 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.379 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.379 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.379 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.380 DEBUG:boto:Method: GET 2023-10-17T18:22:15.380 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.381 DEBUG:boto:Data: 2023-10-17T18:22:15.381 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.382 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.382 DEBUG:boto:Params: {} 2023-10-17T18:22:15.383 DEBUG:boto:Token: None 2023-10-17T18:22:15.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.384 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f9f656f94fbb267-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.387 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.387 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.388 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.388 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.388 DEBUG:boto:Method: GET 2023-10-17T18:22:15.389 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.389 DEBUG:boto:Data: 2023-10-17T18:22:15.390 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.390 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.391 DEBUG:boto:Params: {} 2023-10-17T18:22:15.391 DEBUG:boto:Token: None 2023-10-17T18:22:15.392 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008713bb7f24a885ce-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.438 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.439 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.440 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.440 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.441 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.441 DEBUG:boto:Method: GET 2023-10-17T18:22:15.442 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.442 DEBUG:boto:Data: 2023-10-17T18:22:15.443 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.443 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.443 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.444 DEBUG:boto:Params: {} 2023-10-17T18:22:15.444 DEBUG:boto:Token: None 2023-10-17T18:22:15.445 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae3ad6d078fa835-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.449 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.450 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.450 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.451 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.451 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.452 DEBUG:boto:Method: GET 2023-10-17T18:22:15.452 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.453 DEBUG:boto:Data: 2023-10-17T18:22:15.453 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.454 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.454 DEBUG:boto:Params: {} 2023-10-17T18:22:15.455 DEBUG:boto:Token: None 2023-10-17T18:22:15.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ= 2023-10-17T18:22:15.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e87673cfcf85f2f-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.459 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.460 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.460 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.461 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.461 DEBUG:boto:Method: GET 2023-10-17T18:22:15.461 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.462 DEBUG:boto:Data: 2023-10-17T18:22:15.462 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.463 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.463 DEBUG:boto:Params: {} 2023-10-17T18:22:15.464 DEBUG:boto:Token: None 2023-10-17T18:22:15.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ= 2023-10-17T18:22:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b6aedc8049767c4-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.468 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.468 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.469 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.469 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.470 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.470 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.470 DEBUG:boto:Data: 2023-10-17T18:22:15.471 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.471 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.472 DEBUG:boto:Params: {} 2023-10-17T18:22:15.472 DEBUG:boto:Token: None 2023-10-17T18:22:15.473 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TtuaEkzF77xpUcY00IjruPF0BIA= 2023-10-17T18:22:15.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TtuaEkzF77xpUcY00IjruPF0BIA=', 'Content-Length': '0'} 2023-10-17T18:22:15.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8946e518f5f06b3-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.476 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.477 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.477 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.477 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.478 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.478 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.478 DEBUG:boto:Data: 2023-10-17T18:22:15.479 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.479 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.480 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.480 DEBUG:boto:Params: {} 2023-10-17T18:22:15.480 DEBUG:boto:Token: None 2023-10-17T18:22:15.481 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TtuaEkzF77xpUcY00IjruPF0BIA= 2023-10-17T18:22:15.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TtuaEkzF77xpUcY00IjruPF0BIA=', 'Content-Length': '0'} 2023-10-17T18:22:15.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a7dd84c0e0fc6ca-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.484 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.485 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.485 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.485 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.486 DEBUG:boto:Method: GET 2023-10-17T18:22:15.486 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.487 DEBUG:boto:Data: 2023-10-17T18:22:15.487 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.488 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.488 DEBUG:boto:Params: {} 2023-10-17T18:22:15.489 DEBUG:boto:Token: None 2023-10-17T18:22:15.489 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ= 2023-10-17T18:22:15.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000662fb528f38009e9-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.493 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.493 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.493 DEBUG:boto:path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.494 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.494 DEBUG:boto:Method: GET 2023-10-17T18:22:15.495 DEBUG:boto:Path: /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.495 DEBUG:boto:Data: 2023-10-17T18:22:15.496 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.496 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.496 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.497 DEBUG:boto:Params: {} 2023-10-17T18:22:15.497 DEBUG:boto:Token: None 2023-10-17T18:22:15.498 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw 2023-10-17T18:22:15.498 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ= 2023-10-17T18:22:15.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZAsE1+a01e+Ap8UaO8Fljfu91EQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'yPuJVmTAKnpTje-Aw-K6Cmd-G5nsqjw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000875ccead9e98dd0a-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.501 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.502 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.502 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.502 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.503 DEBUG:boto:Method: GET 2023-10-17T18:22:15.503 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.504 DEBUG:boto:Data: 2023-10-17T18:22:15.504 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.505 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.505 DEBUG:boto:Params: {} 2023-10-17T18:22:15.505 DEBUG:boto:Token: None 2023-10-17T18:22:15.506 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.506 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645f103a28ca0df6-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.510 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.510 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.511 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.511 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.512 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.512 DEBUG:boto:Method: GET 2023-10-17T18:22:15.512 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.513 DEBUG:boto:Data: 2023-10-17T18:22:15.513 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.514 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.514 DEBUG:boto:Params: {} 2023-10-17T18:22:15.515 DEBUG:boto:Token: None 2023-10-17T18:22:15.515 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.516 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096e7ae4cad2c2f97-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.519 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.520 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.520 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.520 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.521 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.521 DEBUG:boto:Data: 2023-10-17T18:22:15.521 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.522 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.523 DEBUG:boto:Params: {} 2023-10-17T18:22:15.523 DEBUG:boto:Token: None 2023-10-17T18:22:15.524 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016571f200a1cb7d0-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.528 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.529 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.529 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.529 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.530 DEBUG:boto:Data: 2023-10-17T18:22:15.530 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.531 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.531 DEBUG:boto:Params: {} 2023-10-17T18:22:15.532 DEBUG:boto:Token: None 2023-10-17T18:22:15.532 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000592cbe482c4f7c86-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.537 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.538 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.538 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.539 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.539 DEBUG:boto:Method: GET 2023-10-17T18:22:15.539 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.540 DEBUG:boto:Data: 2023-10-17T18:22:15.540 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.541 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.541 DEBUG:boto:Params: {} 2023-10-17T18:22:15.542 DEBUG:boto:Token: None 2023-10-17T18:22:15.542 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.543 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd171f429db17394-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.586 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.587 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.587 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.588 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.588 DEBUG:boto:Method: GET 2023-10-17T18:22:15.588 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.589 DEBUG:boto:Data: 2023-10-17T18:22:15.589 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.590 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.590 DEBUG:boto:Params: {} 2023-10-17T18:22:15.591 DEBUG:boto:Token: None 2023-10-17T18:22:15.591 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ead29f04a84507cd-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.595 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.596 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.596 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.596 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.597 DEBUG:boto:Method: GET 2023-10-17T18:22:15.597 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.597 DEBUG:boto:Data: 2023-10-17T18:22:15.598 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.599 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.599 DEBUG:boto:Params: {} 2023-10-17T18:22:15.600 DEBUG:boto:Token: None 2023-10-17T18:22:15.600 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4ed52b3be25284-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.604 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.605 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.605 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.605 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.606 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.606 DEBUG:boto:Method: GET 2023-10-17T18:22:15.607 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.607 DEBUG:boto:Data: 2023-10-17T18:22:15.607 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.608 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.608 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.608 DEBUG:boto:Params: {} 2023-10-17T18:22:15.609 DEBUG:boto:Token: None 2023-10-17T18:22:15.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.610 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729320766825c385-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.616 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.617 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.618 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.618 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.618 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.619 DEBUG:boto:Method: GET 2023-10-17T18:22:15.619 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.620 DEBUG:boto:Data: 2023-10-17T18:22:15.620 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.621 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.621 DEBUG:boto:Params: {} 2023-10-17T18:22:15.622 DEBUG:boto:Token: None 2023-10-17T18:22:15.622 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78= 2023-10-17T18:22:15.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78=', 'Content-Length': '0'} 2023-10-17T18:22:15.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079696f918c937956-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.626 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.626 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.627 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.627 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.627 DEBUG:boto:Method: GET 2023-10-17T18:22:15.628 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.628 DEBUG:boto:Data: 2023-10-17T18:22:15.629 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.629 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.629 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.630 DEBUG:boto:Params: {} 2023-10-17T18:22:15.630 DEBUG:boto:Token: None 2023-10-17T18:22:15.631 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78= 2023-10-17T18:22:15.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78=', 'Content-Length': '0'} 2023-10-17T18:22:15.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016a05b3d990b3122-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.634 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.635 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.635 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.635 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.636 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.636 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.636 DEBUG:boto:Data: 2023-10-17T18:22:15.637 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.637 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.637 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.638 DEBUG:boto:Params: {} 2023-10-17T18:22:15.638 DEBUG:boto:Token: None 2023-10-17T18:22:15.638 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jwryjp/qjsW56eR9zWPw3tzNkw4= 2023-10-17T18:22:15.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jwryjp/qjsW56eR9zWPw3tzNkw4=', 'Content-Length': '0'} 2023-10-17T18:22:15.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bec9c05ae3c2fc31-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.642 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.642 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.642 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.643 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.643 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.643 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.644 DEBUG:boto:Data: 2023-10-17T18:22:15.644 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.645 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.645 DEBUG:boto:Params: {} 2023-10-17T18:22:15.646 DEBUG:boto:Token: None 2023-10-17T18:22:15.646 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jwryjp/qjsW56eR9zWPw3tzNkw4= 2023-10-17T18:22:15.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jwryjp/qjsW56eR9zWPw3tzNkw4=', 'Content-Length': '0'} 2023-10-17T18:22:15.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b8fe28a9887084a-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.650 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.650 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.651 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.651 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.651 DEBUG:boto:Method: GET 2023-10-17T18:22:15.652 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.652 DEBUG:boto:Data: 2023-10-17T18:22:15.652 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.653 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.653 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.654 DEBUG:boto:Params: {} 2023-10-17T18:22:15.654 DEBUG:boto:Token: None 2023-10-17T18:22:15.655 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78= 2023-10-17T18:22:15.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78=', 'Content-Length': '0'} 2023-10-17T18:22:15.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ab8a3a1551b5ea7-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.658 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.658 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.659 DEBUG:boto:path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.659 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.660 DEBUG:boto:Method: GET 2023-10-17T18:22:15.660 DEBUG:boto:Path: /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.660 DEBUG:boto:Data: 2023-10-17T18:22:15.661 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.662 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.662 DEBUG:boto:Params: {} 2023-10-17T18:22:15.662 DEBUG:boto:Token: None 2023-10-17T18:22:15.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE 2023-10-17T18:22:15.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78= 2023-10-17T18:22:15.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isE/hFARG/LtW96BjAdilphit78=', 'Content-Length': '0'} 2023-10-17T18:22:15.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'LOSlfnG5Bvj0RF01qjrhE1AtM-Qj3vE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000472aeb1108b2f582-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.667 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.667 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.667 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.668 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.668 DEBUG:boto:Method: GET 2023-10-17T18:22:15.668 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.669 DEBUG:boto:Data: 2023-10-17T18:22:15.669 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.670 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.671 DEBUG:boto:Params: {} 2023-10-17T18:22:15.671 DEBUG:boto:Token: None 2023-10-17T18:22:15.671 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca8b5339d7e5c14-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.676 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.676 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.677 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.677 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.677 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.678 DEBUG:boto:Method: GET 2023-10-17T18:22:15.678 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.679 DEBUG:boto:Data: 2023-10-17T18:22:15.679 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.680 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.680 DEBUG:boto:Params: {} 2023-10-17T18:22:15.681 DEBUG:boto:Token: None 2023-10-17T18:22:15.681 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.681 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0d635a81b9c9df-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.685 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.685 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.686 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.686 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.687 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.687 DEBUG:boto:Data: 2023-10-17T18:22:15.687 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.688 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.689 DEBUG:boto:Params: {} 2023-10-17T18:22:15.689 DEBUG:boto:Token: None 2023-10-17T18:22:15.689 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bef5253c24ed8a0d-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.694 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.694 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.695 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.695 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.695 DEBUG:boto:Data: 2023-10-17T18:22:15.696 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.696 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.697 DEBUG:boto:Params: {} 2023-10-17T18:22:15.697 DEBUG:boto:Token: None 2023-10-17T18:22:15.698 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.698 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abd09c51ea4de321-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.702 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.703 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.703 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.704 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.704 DEBUG:boto:Method: GET 2023-10-17T18:22:15.704 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.705 DEBUG:boto:Data: 2023-10-17T18:22:15.705 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.706 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.706 DEBUG:boto:Params: {} 2023-10-17T18:22:15.707 DEBUG:boto:Token: None 2023-10-17T18:22:15.707 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.708 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abe507f3e348e60a-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.754 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.755 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.755 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.756 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.756 DEBUG:boto:Method: GET 2023-10-17T18:22:15.756 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.757 DEBUG:boto:Data: 2023-10-17T18:22:15.757 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.757 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.758 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.758 DEBUG:boto:Params: {} 2023-10-17T18:22:15.759 DEBUG:boto:Token: None 2023-10-17T18:22:15.759 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082fd4e81fc67897b-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.763 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.763 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.763 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.764 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.764 DEBUG:boto:Method: GET 2023-10-17T18:22:15.765 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.765 DEBUG:boto:Data: 2023-10-17T18:22:15.765 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.766 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.766 DEBUG:boto:Params: {} 2023-10-17T18:22:15.767 DEBUG:boto:Token: None 2023-10-17T18:22:15.767 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000974bda210c8e5745-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.772 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.772 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.773 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.773 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.773 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.774 DEBUG:boto:Method: GET 2023-10-17T18:22:15.774 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.775 DEBUG:boto:Data: 2023-10-17T18:22:15.775 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.776 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.776 DEBUG:boto:Params: {} 2023-10-17T18:22:15.777 DEBUG:boto:Token: None 2023-10-17T18:22:15.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd859e40a8a5c47-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.781 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.782 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.783 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.783 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.783 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.784 DEBUG:boto:Method: GET 2023-10-17T18:22:15.784 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.784 DEBUG:boto:Data: 2023-10-17T18:22:15.785 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.785 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.786 DEBUG:boto:Params: {} 2023-10-17T18:22:15.786 DEBUG:boto:Token: None 2023-10-17T18:22:15.787 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc= 2023-10-17T18:22:15.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc=', 'Content-Length': '0'} 2023-10-17T18:22:15.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000016dba188ff297b2-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.790 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.791 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.791 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.792 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.792 DEBUG:boto:Method: GET 2023-10-17T18:22:15.792 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.793 DEBUG:boto:Data: 2023-10-17T18:22:15.793 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.794 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.794 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.794 DEBUG:boto:Params: {} 2023-10-17T18:22:15.795 DEBUG:boto:Token: None 2023-10-17T18:22:15.795 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc= 2023-10-17T18:22:15.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc=', 'Content-Length': '0'} 2023-10-17T18:22:15.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091b14ecad89508aa-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.799 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.799 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.800 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.800 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.801 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.801 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.801 DEBUG:boto:Data: 2023-10-17T18:22:15.802 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.803 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.803 DEBUG:boto:Params: {} 2023-10-17T18:22:15.803 DEBUG:boto:Token: None 2023-10-17T18:22:15.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r49/e17Vyu/zdW4Cy8Yu7J7FdeQ= 2023-10-17T18:22:15.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r49/e17Vyu/zdW4Cy8Yu7J7FdeQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002997745586500f2c-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.807 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.808 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.808 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.808 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.809 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.809 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.810 DEBUG:boto:Data: 2023-10-17T18:22:15.810 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.811 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.811 DEBUG:boto:Params: {} 2023-10-17T18:22:15.812 DEBUG:boto:Token: None 2023-10-17T18:22:15.812 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r49/e17Vyu/zdW4Cy8Yu7J7FdeQ= 2023-10-17T18:22:15.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r49/e17Vyu/zdW4Cy8Yu7J7FdeQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008596b3293b40196b-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.816 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.816 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.817 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.817 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.817 DEBUG:boto:Method: GET 2023-10-17T18:22:15.818 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.818 DEBUG:boto:Data: 2023-10-17T18:22:15.819 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.819 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.819 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.820 DEBUG:boto:Params: {} 2023-10-17T18:22:15.820 DEBUG:boto:Token: None 2023-10-17T18:22:15.821 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc= 2023-10-17T18:22:15.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc=', 'Content-Length': '0'} 2023-10-17T18:22:15.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8b2d1f2c52281d3-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.866 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.867 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.867 DEBUG:boto:path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.868 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.868 DEBUG:boto:Method: GET 2023-10-17T18:22:15.868 DEBUG:boto:Path: /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.869 DEBUG:boto:Data: 2023-10-17T18:22:15.869 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.870 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.870 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.871 DEBUG:boto:Params: {} 2023-10-17T18:22:15.871 DEBUG:boto:Token: None 2023-10-17T18:22:15.872 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI 2023-10-17T18:22:15.872 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc= 2023-10-17T18:22:15.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jN0wX+c9TFuk2YmHL5pOat0ryRc=', 'Content-Length': '0'} 2023-10-17T18:22:15.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'bNYB8oGeDeoT3eNcfxSR3SPKav3E0PI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e4c0d37ccc13391-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.875 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.876 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.876 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.876 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.877 DEBUG:boto:Method: GET 2023-10-17T18:22:15.877 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.878 DEBUG:boto:Data: 2023-10-17T18:22:15.878 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.879 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.879 DEBUG:boto:Params: {} 2023-10-17T18:22:15.880 DEBUG:boto:Token: None 2023-10-17T18:22:15.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064dfdf4637f90bf2-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.885 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.885 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.886 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.886 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.886 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.887 DEBUG:boto:Method: GET 2023-10-17T18:22:15.887 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.888 DEBUG:boto:Data: 2023-10-17T18:22:15.888 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.889 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.889 DEBUG:boto:Params: {} 2023-10-17T18:22:15.890 DEBUG:boto:Token: None 2023-10-17T18:22:15.890 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.891 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271b98e46d37caf6-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.894 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.894 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.895 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.895 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.896 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.896 DEBUG:boto:Data: 2023-10-17T18:22:15.896 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.897 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.897 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.897 DEBUG:boto:Params: {} 2023-10-17T18:22:15.898 DEBUG:boto:Token: None 2023-10-17T18:22:15.898 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.899 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094c8533a84816290-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.903 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.903 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.904 DEBUG:boto:Method: HEAD 2023-10-17T18:22:15.904 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:15.904 DEBUG:boto:Data: 2023-10-17T18:22:15.905 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.905 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.906 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.906 DEBUG:boto:Params: {} 2023-10-17T18:22:15.907 DEBUG:boto:Token: None 2023-10-17T18:22:15.907 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj 2023-10-17T18:22:15.907 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE= 2023-10-17T18:22:15.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0anEFz9AjLuhBWok2t3lJzUVoE=', 'Content-Length': '0'} 2023-10-17T18:22:15.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe47e09648b7c5f7-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.912 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.912 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.913 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.913 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.913 DEBUG:boto:Method: GET 2023-10-17T18:22:15.914 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.914 DEBUG:boto:Data: 2023-10-17T18:22:15.914 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.915 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.916 DEBUG:boto:Params: {} 2023-10-17T18:22:15.916 DEBUG:boto:Token: None 2023-10-17T18:22:15.917 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b71869f3eb494587-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.962 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.963 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.963 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.963 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.964 DEBUG:boto:Method: GET 2023-10-17T18:22:15.964 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.965 DEBUG:boto:Data: 2023-10-17T18:22:15.965 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.966 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.966 DEBUG:boto:Params: {} 2023-10-17T18:22:15.967 DEBUG:boto:Token: None 2023-10-17T18:22:15.967 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:15.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ= 2023-10-17T18:22:15.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OT1l/vUMT54ut4tXpbTPVAo/AQQ=', 'Content-Length': '0'} 2023-10-17T18:22:15.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000105991130f20fae9-00652ed0d7-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.970 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.971 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.971 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.972 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.972 DEBUG:boto:Method: GET 2023-10-17T18:22:15.972 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.973 DEBUG:boto:Data: 2023-10-17T18:22:15.973 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.974 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.974 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.974 DEBUG:boto:Params: {} 2023-10-17T18:22:15.975 DEBUG:boto:Token: None 2023-10-17T18:22:15.975 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.976 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0779956d60cec69-00652ed0d7-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.979 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.980 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:15.980 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:15.980 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.981 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:15.981 DEBUG:boto:Method: GET 2023-10-17T18:22:15.982 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:15.982 DEBUG:boto:Data: 2023-10-17T18:22:15.982 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.983 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:15.983 DEBUG:boto:Port: 8000 2023-10-17T18:22:15.984 DEBUG:boto:Params: {} 2023-10-17T18:22:15.984 DEBUG:boto:Token: None 2023-10-17T18:22:15.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/ 2023-10-17T18:22:15.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w= 2023-10-17T18:22:15.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LC+Q6NppH2btQ5c73YegRFLiJ8w=', 'Content-Length': '0'} 2023-10-17T18:22:15.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7d4501f486bcff-00652ed0d7-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.989 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:15.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:15.990 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.990 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.991 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:15.991 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:15.991 DEBUG:boto:Method: GET 2023-10-17T18:22:15.992 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:15.992 DEBUG:boto:Data: 2023-10-17T18:22:15.993 DEBUG:boto:Headers: {} 2023-10-17T18:22:15.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:15.993 DEBUG:boto:Port: 8001 2023-10-17T18:22:15.994 DEBUG:boto:Params: {} 2023-10-17T18:22:15.994 DEBUG:boto:Token: None 2023-10-17T18:22:15.995 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:15 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:15.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BJNho4WyA/m26H32S854gbaC3IU= 2023-10-17T18:22:15.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BJNho4WyA/m26H32S854gbaC3IU=', 'Content-Length': '0'} 2023-10-17T18:22:15.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000815882ae2e772083-00652ed0d7-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:15.998 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:15.999 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:15.999 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:15.999 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.000 DEBUG:boto:Method: GET 2023-10-17T18:22:16.000 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.001 DEBUG:boto:Data: 2023-10-17T18:22:16.001 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.001 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.002 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.002 DEBUG:boto:Params: {} 2023-10-17T18:22:16.003 DEBUG:boto:Token: None 2023-10-17T18:22:16.003 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg= 2023-10-17T18:22:16.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg=', 'Content-Length': '0'} 2023-10-17T18:22:16.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a777e585c31e620e-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.007 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.007 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.007 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.008 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.008 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.009 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.009 DEBUG:boto:Data: 2023-10-17T18:22:16.009 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.010 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.010 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.011 DEBUG:boto:Params: {} 2023-10-17T18:22:16.011 DEBUG:boto:Token: None 2023-10-17T18:22:16.011 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.012 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XJdX8UOFfCc40TYji9F4wNCM4Ec= 2023-10-17T18:22:16.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XJdX8UOFfCc40TYji9F4wNCM4Ec=', 'Content-Length': '0'} 2023-10-17T18:22:16.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aee12559c33ac22-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.015 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.015 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.016 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.016 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.017 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.017 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.017 DEBUG:boto:Data: 2023-10-17T18:22:16.018 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.021 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.021 DEBUG:boto:Params: {} 2023-10-17T18:22:16.022 DEBUG:boto:Token: None 2023-10-17T18:22:16.022 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XJdX8UOFfCc40TYji9F4wNCM4Ec= 2023-10-17T18:22:16.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XJdX8UOFfCc40TYji9F4wNCM4Ec=', 'Content-Length': '0'} 2023-10-17T18:22:16.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aec559a401ed95dd-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.026 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.027 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.027 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.028 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.028 DEBUG:boto:Method: GET 2023-10-17T18:22:16.029 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.029 DEBUG:boto:Data: 2023-10-17T18:22:16.029 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.030 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.031 DEBUG:boto:Params: {} 2023-10-17T18:22:16.031 DEBUG:boto:Token: None 2023-10-17T18:22:16.032 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg= 2023-10-17T18:22:16.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg=', 'Content-Length': '0'} 2023-10-17T18:22:16.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003908752a50675b95-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.078 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.079 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.079 DEBUG:boto:path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.080 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.080 DEBUG:boto:Method: GET 2023-10-17T18:22:16.080 DEBUG:boto:Path: /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.081 DEBUG:boto:Data: 2023-10-17T18:22:16.081 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.082 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.083 DEBUG:boto:Params: {} 2023-10-17T18:22:16.083 DEBUG:boto:Token: None 2023-10-17T18:22:16.083 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=TsUld4HsC0SqKDxbDEGOA493mZHoSCz 2023-10-17T18:22:16.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg= 2023-10-17T18:22:16.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B6e352m5rUC+t6cKSEnAm7E1cqg=', 'Content-Length': '0'} 2023-10-17T18:22:16.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'TsUld4HsC0SqKDxbDEGOA493mZHoSCz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c801e158c6f7aa9-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.087 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.088 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.088 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.088 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.089 DEBUG:boto:Method: GET 2023-10-17T18:22:16.089 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.090 DEBUG:boto:Data: 2023-10-17T18:22:16.090 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.091 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.091 DEBUG:boto:Params: {} 2023-10-17T18:22:16.092 DEBUG:boto:Token: None 2023-10-17T18:22:16.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc4870c67ea8fb32-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.096 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.097 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.097 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.098 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.098 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.099 DEBUG:boto:Method: GET 2023-10-17T18:22:16.099 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.100 DEBUG:boto:Data: 2023-10-17T18:22:16.100 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.101 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.101 DEBUG:boto:Params: {} 2023-10-17T18:22:16.102 DEBUG:boto:Token: None 2023-10-17T18:22:16.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5da56858b66beef-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.106 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.106 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.107 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.107 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.107 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.108 DEBUG:boto:Data: 2023-10-17T18:22:16.108 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.109 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.110 DEBUG:boto:Params: {} 2023-10-17T18:22:16.110 DEBUG:boto:Token: None 2023-10-17T18:22:16.110 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d1fcc5630c1b407-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.115 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.116 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.116 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.117 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.117 DEBUG:boto:Data: 2023-10-17T18:22:16.117 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.118 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.118 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.119 DEBUG:boto:Params: {} 2023-10-17T18:22:16.119 DEBUG:boto:Token: None 2023-10-17T18:22:16.120 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d12af5bb7238689-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.124 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.125 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.125 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.126 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.126 DEBUG:boto:Method: GET 2023-10-17T18:22:16.127 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.127 DEBUG:boto:Data: 2023-10-17T18:22:16.127 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.128 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.128 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.129 DEBUG:boto:Params: {} 2023-10-17T18:22:16.129 DEBUG:boto:Token: None 2023-10-17T18:22:16.130 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.130 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0ab20aa0359224-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.174 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.175 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.175 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.176 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.176 DEBUG:boto:Method: GET 2023-10-17T18:22:16.177 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.177 DEBUG:boto:Data: 2023-10-17T18:22:16.177 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.178 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.179 DEBUG:boto:Params: {} 2023-10-17T18:22:16.179 DEBUG:boto:Token: None 2023-10-17T18:22:16.180 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030efacea4ac4b0c7-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.183 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.183 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.184 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.184 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.184 DEBUG:boto:Method: GET 2023-10-17T18:22:16.185 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.185 DEBUG:boto:Data: 2023-10-17T18:22:16.186 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.186 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.186 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.187 DEBUG:boto:Params: {} 2023-10-17T18:22:16.187 DEBUG:boto:Token: None 2023-10-17T18:22:16.188 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.188 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b797a364aa74b3b4-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.192 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.193 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.193 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.194 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.194 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.195 DEBUG:boto:Method: GET 2023-10-17T18:22:16.195 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.195 DEBUG:boto:Data: 2023-10-17T18:22:16.196 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.196 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.197 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.197 DEBUG:boto:Params: {} 2023-10-17T18:22:16.197 DEBUG:boto:Token: None 2023-10-17T18:22:16.198 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.198 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ead073f28cede5-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.202 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.203 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.204 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.204 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.204 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.205 DEBUG:boto:Method: GET 2023-10-17T18:22:16.205 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.205 DEBUG:boto:Data: 2023-10-17T18:22:16.206 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.207 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.207 DEBUG:boto:Params: {} 2023-10-17T18:22:16.207 DEBUG:boto:Token: None 2023-10-17T18:22:16.208 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8= 2023-10-17T18:22:16.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8=', 'Content-Length': '0'} 2023-10-17T18:22:16.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043b85af369d5ef4c-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.211 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.212 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.212 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.213 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.213 DEBUG:boto:Method: GET 2023-10-17T18:22:16.214 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.214 DEBUG:boto:Data: 2023-10-17T18:22:16.214 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.215 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.216 DEBUG:boto:Params: {} 2023-10-17T18:22:16.216 DEBUG:boto:Token: None 2023-10-17T18:22:16.216 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.217 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8= 2023-10-17T18:22:16.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8=', 'Content-Length': '0'} 2023-10-17T18:22:16.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001144c3182db6004c-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.222 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.222 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.223 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.223 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.223 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.224 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.224 DEBUG:boto:Data: 2023-10-17T18:22:16.225 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.225 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.226 DEBUG:boto:Params: {} 2023-10-17T18:22:16.226 DEBUG:boto:Token: None 2023-10-17T18:22:16.227 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.227 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLHMRmke69+/u7/5uBi7iiAmTuE= 2023-10-17T18:22:16.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLHMRmke69+/u7/5uBi7iiAmTuE=', 'Content-Length': '0'} 2023-10-17T18:22:16.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b826f1ad28f9ebb-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.230 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.230 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.231 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.231 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.231 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.232 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.232 DEBUG:boto:Data: 2023-10-17T18:22:16.233 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.233 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.234 DEBUG:boto:Params: {} 2023-10-17T18:22:16.234 DEBUG:boto:Token: None 2023-10-17T18:22:16.235 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLHMRmke69+/u7/5uBi7iiAmTuE= 2023-10-17T18:22:16.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLHMRmke69+/u7/5uBi7iiAmTuE=', 'Content-Length': '0'} 2023-10-17T18:22:16.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000879e173b19983eb4-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.238 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.239 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.239 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.240 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.240 DEBUG:boto:Method: GET 2023-10-17T18:22:16.240 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.241 DEBUG:boto:Data: 2023-10-17T18:22:16.241 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.242 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.242 DEBUG:boto:Params: {} 2023-10-17T18:22:16.243 DEBUG:boto:Token: None 2023-10-17T18:22:16.243 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8= 2023-10-17T18:22:16.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8=', 'Content-Length': '0'} 2023-10-17T18:22:16.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f34a52f3dbf2a238-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.286 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.287 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.287 DEBUG:boto:path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.287 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.288 DEBUG:boto:Method: GET 2023-10-17T18:22:16.288 DEBUG:boto:Path: /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.288 DEBUG:boto:Data: 2023-10-17T18:22:16.289 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.290 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.290 DEBUG:boto:Params: {} 2023-10-17T18:22:16.290 DEBUG:boto:Token: None 2023-10-17T18:22:16.291 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL 2023-10-17T18:22:16.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8= 2023-10-17T18:22:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9MyD5NIN2OszipxBKPkiEGeTrG8=', 'Content-Length': '0'} 2023-10-17T18:22:16.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'dXQC0aagQwy9HHQaDPPZUv4y6-4ewTL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b0cba6715119849-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.295 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.295 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.295 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.296 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.296 DEBUG:boto:Method: GET 2023-10-17T18:22:16.296 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.297 DEBUG:boto:Data: 2023-10-17T18:22:16.297 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.298 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.298 DEBUG:boto:Params: {} 2023-10-17T18:22:16.299 DEBUG:boto:Token: None 2023-10-17T18:22:16.299 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f013b946fd86abea-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.303 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.304 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.304 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.305 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.305 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.305 DEBUG:boto:Method: GET 2023-10-17T18:22:16.306 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.306 DEBUG:boto:Data: 2023-10-17T18:22:16.307 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.307 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.307 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.308 DEBUG:boto:Params: {} 2023-10-17T18:22:16.308 DEBUG:boto:Token: None 2023-10-17T18:22:16.309 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.309 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b47ac69cff2d4be-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.312 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.313 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.313 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.314 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.314 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.315 DEBUG:boto:Data: 2023-10-17T18:22:16.315 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.315 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.316 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.316 DEBUG:boto:Params: {} 2023-10-17T18:22:16.316 DEBUG:boto:Token: None 2023-10-17T18:22:16.317 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.317 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f2b903c0d6871ad-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.321 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.321 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.322 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.322 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.323 DEBUG:boto:Data: 2023-10-17T18:22:16.323 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.323 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.324 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.324 DEBUG:boto:Params: {} 2023-10-17T18:22:16.325 DEBUG:boto:Token: None 2023-10-17T18:22:16.325 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.325 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000446509c37936720-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.330 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.330 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.331 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.331 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.332 DEBUG:boto:Method: GET 2023-10-17T18:22:16.332 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.332 DEBUG:boto:Data: 2023-10-17T18:22:16.333 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.333 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.334 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.334 DEBUG:boto:Params: {} 2023-10-17T18:22:16.334 DEBUG:boto:Token: None 2023-10-17T18:22:16.335 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.335 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054c173cb7e2df8db-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.378 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.379 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.379 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.379 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.380 DEBUG:boto:Method: GET 2023-10-17T18:22:16.380 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.381 DEBUG:boto:Data: 2023-10-17T18:22:16.381 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.382 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.382 DEBUG:boto:Params: {} 2023-10-17T18:22:16.383 DEBUG:boto:Token: None 2023-10-17T18:22:16.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.384 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000960b36279bf6435d-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.387 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.387 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.388 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.388 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.388 DEBUG:boto:Method: GET 2023-10-17T18:22:16.389 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.389 DEBUG:boto:Data: 2023-10-17T18:22:16.389 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.390 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.391 DEBUG:boto:Params: {} 2023-10-17T18:22:16.391 DEBUG:boto:Token: None 2023-10-17T18:22:16.391 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1367d4dfc0d2ba-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.438 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.439 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.439 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.440 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.440 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.441 DEBUG:boto:Method: GET 2023-10-17T18:22:16.441 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.441 DEBUG:boto:Data: 2023-10-17T18:22:16.442 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.442 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.443 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.443 DEBUG:boto:Params: {} 2023-10-17T18:22:16.444 DEBUG:boto:Token: None 2023-10-17T18:22:16.444 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.444 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8d30ee00dd6183-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.448 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.449 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.449 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.450 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.450 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.451 DEBUG:boto:Method: GET 2023-10-17T18:22:16.451 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.451 DEBUG:boto:Data: 2023-10-17T18:22:16.452 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.452 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.453 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.453 DEBUG:boto:Params: {} 2023-10-17T18:22:16.453 DEBUG:boto:Token: None 2023-10-17T18:22:16.454 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.454 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0= 2023-10-17T18:22:16.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0=', 'Content-Length': '0'} 2023-10-17T18:22:16.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e85ca4d4934910c-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.457 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.458 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.458 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.459 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.459 DEBUG:boto:Method: GET 2023-10-17T18:22:16.460 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.460 DEBUG:boto:Data: 2023-10-17T18:22:16.460 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.461 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.461 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.462 DEBUG:boto:Params: {} 2023-10-17T18:22:16.462 DEBUG:boto:Token: None 2023-10-17T18:22:16.462 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.463 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0= 2023-10-17T18:22:16.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0=', 'Content-Length': '0'} 2023-10-17T18:22:16.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cfc8725fa29df24-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.466 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.467 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.467 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.467 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.468 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.468 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.468 DEBUG:boto:Data: 2023-10-17T18:22:16.469 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.470 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.470 DEBUG:boto:Params: {} 2023-10-17T18:22:16.470 DEBUG:boto:Token: None 2023-10-17T18:22:16.471 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvjhrgkO/XUz5pHsAiWzPftyR1E= 2023-10-17T18:22:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvjhrgkO/XUz5pHsAiWzPftyR1E=', 'Content-Length': '0'} 2023-10-17T18:22:16.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004acb4bde2697db7c-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.474 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.475 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.475 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.476 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.476 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.476 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.477 DEBUG:boto:Data: 2023-10-17T18:22:16.477 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.478 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.478 DEBUG:boto:Params: {} 2023-10-17T18:22:16.478 DEBUG:boto:Token: None 2023-10-17T18:22:16.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IvjhrgkO/XUz5pHsAiWzPftyR1E= 2023-10-17T18:22:16.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IvjhrgkO/XUz5pHsAiWzPftyR1E=', 'Content-Length': '0'} 2023-10-17T18:22:16.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000639bd8648ac9a8f7-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.483 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.483 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.484 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.484 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.484 DEBUG:boto:Method: GET 2023-10-17T18:22:16.485 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.485 DEBUG:boto:Data: 2023-10-17T18:22:16.485 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.486 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.487 DEBUG:boto:Params: {} 2023-10-17T18:22:16.487 DEBUG:boto:Token: None 2023-10-17T18:22:16.487 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0= 2023-10-17T18:22:16.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0=', 'Content-Length': '0'} 2023-10-17T18:22:16.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a00affd4e41a6246-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.491 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.491 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.491 DEBUG:boto:path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.492 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.492 DEBUG:boto:Method: GET 2023-10-17T18:22:16.493 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.493 DEBUG:boto:Data: 2023-10-17T18:22:16.493 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.494 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.494 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.495 DEBUG:boto:Params: {} 2023-10-17T18:22:16.495 DEBUG:boto:Token: None 2023-10-17T18:22:16.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd 2023-10-17T18:22:16.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0= 2023-10-17T18:22:16.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FXp2jiWy+Rm/U+T/+n+yYJ3J+O0=', 'Content-Length': '0'} 2023-10-17T18:22:16.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', 'Ci1VxMdlhJnaYDdbpQem.RUfluTWjXd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9db4302ecaeb3b2-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.499 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.499 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.500 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.500 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.501 DEBUG:boto:Method: GET 2023-10-17T18:22:16.501 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.501 DEBUG:boto:Data: 2023-10-17T18:22:16.502 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.503 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.503 DEBUG:boto:Params: {} 2023-10-17T18:22:16.503 DEBUG:boto:Token: None 2023-10-17T18:22:16.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4e4c92dfd20c3c-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.508 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.508 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.509 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.509 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.510 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.510 DEBUG:boto:Method: GET 2023-10-17T18:22:16.510 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.511 DEBUG:boto:Data: 2023-10-17T18:22:16.511 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.512 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.513 DEBUG:boto:Params: {} 2023-10-17T18:22:16.513 DEBUG:boto:Token: None 2023-10-17T18:22:16.513 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064122891b803dd72-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.517 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.518 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.518 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.519 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.519 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.520 DEBUG:boto:Data: 2023-10-17T18:22:16.520 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.521 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.521 DEBUG:boto:Params: {} 2023-10-17T18:22:16.521 DEBUG:boto:Token: None 2023-10-17T18:22:16.522 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.522 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a7d08eb03b72e86-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.526 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.527 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.527 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.528 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.528 DEBUG:boto:Data: 2023-10-17T18:22:16.528 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.529 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.529 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.529 DEBUG:boto:Params: {} 2023-10-17T18:22:16.530 DEBUG:boto:Token: None 2023-10-17T18:22:16.530 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000076660b80716666d-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.536 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.536 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.536 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.537 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.537 DEBUG:boto:Method: GET 2023-10-17T18:22:16.538 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.538 DEBUG:boto:Data: 2023-10-17T18:22:16.538 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.539 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.540 DEBUG:boto:Params: {} 2023-10-17T18:22:16.540 DEBUG:boto:Token: None 2023-10-17T18:22:16.541 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbda50df4481311b-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.586 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.587 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.587 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.587 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.588 DEBUG:boto:Method: GET 2023-10-17T18:22:16.588 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.589 DEBUG:boto:Data: 2023-10-17T18:22:16.589 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.589 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.590 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.590 DEBUG:boto:Params: {} 2023-10-17T18:22:16.591 DEBUG:boto:Token: None 2023-10-17T18:22:16.591 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007010242ba72e7243-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.594 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.595 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.595 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.595 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.596 DEBUG:boto:Method: GET 2023-10-17T18:22:16.596 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.597 DEBUG:boto:Data: 2023-10-17T18:22:16.597 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.598 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.598 DEBUG:boto:Params: {} 2023-10-17T18:22:16.599 DEBUG:boto:Token: None 2023-10-17T18:22:16.599 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000101470d91ba7fbde-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.603 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.604 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.604 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.604 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.605 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.605 DEBUG:boto:Method: GET 2023-10-17T18:22:16.605 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.606 DEBUG:boto:Data: 2023-10-17T18:22:16.606 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.607 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.607 DEBUG:boto:Params: {} 2023-10-17T18:22:16.608 DEBUG:boto:Token: None 2023-10-17T18:22:16.608 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006248b4311d3ad638-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.613 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.613 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.614 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.614 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.615 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.615 DEBUG:boto:Method: GET 2023-10-17T18:22:16.615 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.616 DEBUG:boto:Data: 2023-10-17T18:22:16.616 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.617 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.617 DEBUG:boto:Params: {} 2023-10-17T18:22:16.618 DEBUG:boto:Token: None 2023-10-17T18:22:16.618 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.619 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0= 2023-10-17T18:22:16.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0=', 'Content-Length': '0'} 2023-10-17T18:22:16.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdd22afa6ac27266-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.622 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.623 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.623 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.623 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.624 DEBUG:boto:Method: GET 2023-10-17T18:22:16.624 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.625 DEBUG:boto:Data: 2023-10-17T18:22:16.625 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.626 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.626 DEBUG:boto:Params: {} 2023-10-17T18:22:16.627 DEBUG:boto:Token: None 2023-10-17T18:22:16.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0= 2023-10-17T18:22:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0=', 'Content-Length': '0'} 2023-10-17T18:22:16.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a3f157fbe3dc81b-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.674 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.675 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.675 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.675 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.676 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.676 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.676 DEBUG:boto:Data: 2023-10-17T18:22:16.677 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.678 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.678 DEBUG:boto:Params: {} 2023-10-17T18:22:16.678 DEBUG:boto:Token: None 2023-10-17T18:22:16.679 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.679 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gsZhaeU5svm+Ly20gBv3UjT+6n8= 2023-10-17T18:22:16.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gsZhaeU5svm+Ly20gBv3UjT+6n8=', 'Content-Length': '0'} 2023-10-17T18:22:16.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020cb581a88ddac18-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.682 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.683 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.683 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.684 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.684 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.684 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.685 DEBUG:boto:Data: 2023-10-17T18:22:16.685 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.686 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.686 DEBUG:boto:Params: {} 2023-10-17T18:22:16.687 DEBUG:boto:Token: None 2023-10-17T18:22:16.687 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.688 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gsZhaeU5svm+Ly20gBv3UjT+6n8= 2023-10-17T18:22:16.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gsZhaeU5svm+Ly20gBv3UjT+6n8=', 'Content-Length': '0'} 2023-10-17T18:22:16.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024acdafd8e07aeb1-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.691 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.691 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.692 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.692 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.692 DEBUG:boto:Method: GET 2023-10-17T18:22:16.693 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.693 DEBUG:boto:Data: 2023-10-17T18:22:16.694 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.694 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.694 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.695 DEBUG:boto:Params: {} 2023-10-17T18:22:16.695 DEBUG:boto:Token: None 2023-10-17T18:22:16.696 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0= 2023-10-17T18:22:16.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0=', 'Content-Length': '0'} 2023-10-17T18:22:16.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b889a451c6ee8f4-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.742 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.743 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.743 DEBUG:boto:path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.743 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.744 DEBUG:boto:Method: GET 2023-10-17T18:22:16.744 DEBUG:boto:Path: /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.745 DEBUG:boto:Data: 2023-10-17T18:22:16.745 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.745 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.746 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.746 DEBUG:boto:Params: {} 2023-10-17T18:22:16.747 DEBUG:boto:Token: None 2023-10-17T18:22:16.747 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb 2023-10-17T18:22:16.747 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0= 2023-10-17T18:22:16.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:en0LDaFjgDwzfRJOvdl/EtkK5g0=', 'Content-Length': '0'} 2023-10-17T18:22:16.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'lEy4.5Re5Y30Nm4WX-WCLD8xH-XO5cb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072153b82d798fa21-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.750 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.751 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.751 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.751 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.752 DEBUG:boto:Method: GET 2023-10-17T18:22:16.752 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.753 DEBUG:boto:Data: 2023-10-17T18:22:16.753 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.753 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.754 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.754 DEBUG:boto:Params: {} 2023-10-17T18:22:16.755 DEBUG:boto:Token: None 2023-10-17T18:22:16.755 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.755 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d521b6addfeb6b-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.802 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.803 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.803 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.804 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.804 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.805 DEBUG:boto:Method: GET 2023-10-17T18:22:16.805 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.805 DEBUG:boto:Data: 2023-10-17T18:22:16.806 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.806 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.807 DEBUG:boto:Params: {} 2023-10-17T18:22:16.807 DEBUG:boto:Token: None 2023-10-17T18:22:16.808 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.808 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d251d14ca9e41be-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.812 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.812 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.813 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.813 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.813 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.814 DEBUG:boto:Data: 2023-10-17T18:22:16.814 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.815 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.815 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.815 DEBUG:boto:Params: {} 2023-10-17T18:22:16.816 DEBUG:boto:Token: None 2023-10-17T18:22:16.816 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4dff6ebc41b9f79-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.821 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.821 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.821 DEBUG:boto:Method: HEAD 2023-10-17T18:22:16.822 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:16.822 DEBUG:boto:Data: 2023-10-17T18:22:16.822 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.823 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.823 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.823 DEBUG:boto:Params: {} 2023-10-17T18:22:16.824 DEBUG:boto:Token: None 2023-10-17T18:22:16.824 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj 2023-10-17T18:22:16.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0= 2023-10-17T18:22:16.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:whXfjpd1kPPU0br6YMxV51hzrf0=', 'Content-Length': '0'} 2023-10-17T18:22:16.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4df0af16b158506-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.829 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.830 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.830 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.831 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.831 DEBUG:boto:Method: GET 2023-10-17T18:22:16.832 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.832 DEBUG:boto:Data: 2023-10-17T18:22:16.832 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.833 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.833 DEBUG:boto:Params: {} 2023-10-17T18:22:16.834 DEBUG:boto:Token: None 2023-10-17T18:22:16.834 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da92bde5c59b072c-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.878 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.879 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.879 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.879 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.880 DEBUG:boto:Method: GET 2023-10-17T18:22:16.880 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.881 DEBUG:boto:Data: 2023-10-17T18:22:16.881 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.882 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.882 DEBUG:boto:Params: {} 2023-10-17T18:22:16.883 DEBUG:boto:Token: None 2023-10-17T18:22:16.883 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:16.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I= 2023-10-17T18:22:16.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgNLPWkLQ0ykROj3e/fFd79Ah8I=', 'Content-Length': '0'} 2023-10-17T18:22:16.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eea3b1f53f6596bc-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.887 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.887 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.888 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.888 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.889 DEBUG:boto:Method: GET 2023-10-17T18:22:16.889 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.890 DEBUG:boto:Data: 2023-10-17T18:22:16.890 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.891 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.891 DEBUG:boto:Params: {} 2023-10-17T18:22:16.892 DEBUG:boto:Token: None 2023-10-17T18:22:16.892 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc82447326b2b908-00652ed0d8-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.938 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.939 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:16.939 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:16.940 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.940 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:16.941 DEBUG:boto:Method: GET 2023-10-17T18:22:16.941 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:16.941 DEBUG:boto:Data: 2023-10-17T18:22:16.942 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.943 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.943 DEBUG:boto:Params: {} 2023-10-17T18:22:16.943 DEBUG:boto:Token: None 2023-10-17T18:22:16.944 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/ 2023-10-17T18:22:16.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8= 2023-10-17T18:22:16.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V7aQH4ouzPdVGpyRHLt+vzoCcJ8=', 'Content-Length': '0'} 2023-10-17T18:22:16.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024aa02d9989ac846-00652ed0d8-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.948 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:16.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:16.949 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.949 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.949 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.950 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.950 DEBUG:boto:Method: GET 2023-10-17T18:22:16.951 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.951 DEBUG:boto:Data: 2023-10-17T18:22:16.951 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:16.952 DEBUG:boto:Port: 8001 2023-10-17T18:22:16.952 DEBUG:boto:Params: {} 2023-10-17T18:22:16.953 DEBUG:boto:Token: None 2023-10-17T18:22:16.953 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T2Rz1r6k9Am/kV19RVZ6DIqoEcE= 2023-10-17T18:22:16.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T2Rz1r6k9Am/kV19RVZ6DIqoEcE=', 'Content-Length': '0'} 2023-10-17T18:22:16.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ff86c1f3d247695-00652ed0d8-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:16.957 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:16.957 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:16.958 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.958 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.959 DEBUG:boto:Method: GET 2023-10-17T18:22:16.959 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.959 DEBUG:boto:Data: 2023-10-17T18:22:16.960 DEBUG:boto:Headers: {} 2023-10-17T18:22:16.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:16.961 DEBUG:boto:Port: 8000 2023-10-17T18:22:16.961 DEBUG:boto:Params: {} 2023-10-17T18:22:16.961 DEBUG:boto:Token: None 2023-10-17T18:22:16.962 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:16 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:16.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T2Rz1r6k9Am/kV19RVZ6DIqoEcE= 2023-10-17T18:22:16.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T2Rz1r6k9Am/kV19RVZ6DIqoEcE=', 'Content-Length': '0'} 2023-10-17T18:22:16.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f84649b0bc0ab0f7-00652ed0d8-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.006 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.007 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.007 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.007 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.008 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.008 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.009 DEBUG:boto:Data: 2023-10-17T18:22:17.009 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.010 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.010 DEBUG:boto:Params: {} 2023-10-17T18:22:17.011 DEBUG:boto:Token: None 2023-10-17T18:22:17.011 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bTfOlAkYT9LmNFLfOTwJPQAufcw= 2023-10-17T18:22:17.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bTfOlAkYT9LmNFLfOTwJPQAufcw=', 'Content-Length': '0'} 2023-10-17T18:22:17.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2016eb6d014bbbd-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.014 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.015 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.015 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.015 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.016 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.016 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.017 DEBUG:boto:Data: 2023-10-17T18:22:17.017 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.018 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.018 DEBUG:boto:Params: {} 2023-10-17T18:22:17.018 DEBUG:boto:Token: None 2023-10-17T18:22:17.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bTfOlAkYT9LmNFLfOTwJPQAufcw= 2023-10-17T18:22:17.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bTfOlAkYT9LmNFLfOTwJPQAufcw=', 'Content-Length': '0'} 2023-10-17T18:22:17.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c21af9cce9a33233-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.023 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.023 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.023 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.024 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.024 DEBUG:boto:Method: GET 2023-10-17T18:22:17.024 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.025 DEBUG:boto:Data: 2023-10-17T18:22:17.025 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.025 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.026 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.026 DEBUG:boto:Params: {} 2023-10-17T18:22:17.027 DEBUG:boto:Token: None 2023-10-17T18:22:17.027 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.027 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dE3Xh5T6YOibdF9CIuIWDa3gUOc= 2023-10-17T18:22:17.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dE3Xh5T6YOibdF9CIuIWDa3gUOc=', 'Content-Length': '0'} 2023-10-17T18:22:17.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000292e37ae8ddb532-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.070 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.071 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.071 DEBUG:boto:path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.072 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.072 DEBUG:boto:Method: GET 2023-10-17T18:22:17.072 DEBUG:boto:Path: /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.073 DEBUG:boto:Data: 2023-10-17T18:22:17.073 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.074 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.074 DEBUG:boto:Params: {} 2023-10-17T18:22:17.075 DEBUG:boto:Token: None 2023-10-17T18:22:17.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9 2023-10-17T18:22:17.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dE3Xh5T6YOibdF9CIuIWDa3gUOc= 2023-10-17T18:22:17.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dE3Xh5T6YOibdF9CIuIWDa3gUOc=', 'Content-Length': '0'} 2023-10-17T18:22:17.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', '3VcXEFx4xA.tdjo0ll8Ot8BcYlZntR9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec3086c39b00c36a-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.079 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.079 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.080 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.080 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.080 DEBUG:boto:Method: GET 2023-10-17T18:22:17.081 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.081 DEBUG:boto:Data: 2023-10-17T18:22:17.081 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.082 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.083 DEBUG:boto:Params: {} 2023-10-17T18:22:17.083 DEBUG:boto:Token: None 2023-10-17T18:22:17.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aaf8a106a64a6e1-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.087 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.088 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.088 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.089 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.089 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.090 DEBUG:boto:Method: GET 2023-10-17T18:22:17.090 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.090 DEBUG:boto:Data: 2023-10-17T18:22:17.091 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.092 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.092 DEBUG:boto:Params: {} 2023-10-17T18:22:17.092 DEBUG:boto:Token: None 2023-10-17T18:22:17.093 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027934d6c5133511b-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.097 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.097 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.098 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.098 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.098 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.099 DEBUG:boto:Data: 2023-10-17T18:22:17.099 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.100 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.100 DEBUG:boto:Params: {} 2023-10-17T18:22:17.101 DEBUG:boto:Token: None 2023-10-17T18:22:17.101 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a6c10b70fcc2f4d-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.106 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.106 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.107 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.107 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.107 DEBUG:boto:Data: 2023-10-17T18:22:17.108 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.108 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.109 DEBUG:boto:Params: {} 2023-10-17T18:22:17.109 DEBUG:boto:Token: None 2023-10-17T18:22:17.110 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.110 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ac5c3abb04d59e5-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.115 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.115 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.116 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.116 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.116 DEBUG:boto:Method: GET 2023-10-17T18:22:17.117 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.117 DEBUG:boto:Data: 2023-10-17T18:22:17.117 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.118 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.118 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.119 DEBUG:boto:Params: {} 2023-10-17T18:22:17.119 DEBUG:boto:Token: None 2023-10-17T18:22:17.120 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062622679aa10f8c7-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.166 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.167 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.167 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.167 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.168 DEBUG:boto:Method: GET 2023-10-17T18:22:17.168 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.169 DEBUG:boto:Data: 2023-10-17T18:22:17.169 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.169 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.170 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.170 DEBUG:boto:Params: {} 2023-10-17T18:22:17.171 DEBUG:boto:Token: None 2023-10-17T18:22:17.171 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000de1b31681e297e4-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.175 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.175 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.176 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.176 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.176 DEBUG:boto:Method: GET 2023-10-17T18:22:17.177 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.177 DEBUG:boto:Data: 2023-10-17T18:22:17.177 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.178 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.179 DEBUG:boto:Params: {} 2023-10-17T18:22:17.179 DEBUG:boto:Token: None 2023-10-17T18:22:17.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000743949f2e96878b3-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.183 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.184 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.184 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.185 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.185 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.185 DEBUG:boto:Method: GET 2023-10-17T18:22:17.186 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.186 DEBUG:boto:Data: 2023-10-17T18:22:17.187 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.187 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.187 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.188 DEBUG:boto:Params: {} 2023-10-17T18:22:17.188 DEBUG:boto:Token: None 2023-10-17T18:22:17.189 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.189 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7130a56a9cf51b8-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.234 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.235 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.235 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.236 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.236 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.237 DEBUG:boto:Method: GET 2023-10-17T18:22:17.237 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.237 DEBUG:boto:Data: 2023-10-17T18:22:17.238 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.238 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.239 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.239 DEBUG:boto:Params: {} 2023-10-17T18:22:17.240 DEBUG:boto:Token: None 2023-10-17T18:22:17.240 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.240 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0= 2023-10-17T18:22:17.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0=', 'Content-Length': '0'} 2023-10-17T18:22:17.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ea6cf2f183d4143-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.244 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.244 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.244 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.245 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.245 DEBUG:boto:Method: GET 2023-10-17T18:22:17.246 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.246 DEBUG:boto:Data: 2023-10-17T18:22:17.246 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.247 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.248 DEBUG:boto:Params: {} 2023-10-17T18:22:17.248 DEBUG:boto:Token: None 2023-10-17T18:22:17.248 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.249 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0= 2023-10-17T18:22:17.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0=', 'Content-Length': '0'} 2023-10-17T18:22:17.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c322077cd1c2fda-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.252 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.252 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.253 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.253 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.253 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.254 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.254 DEBUG:boto:Data: 2023-10-17T18:22:17.254 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.255 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.256 DEBUG:boto:Params: {} 2023-10-17T18:22:17.256 DEBUG:boto:Token: None 2023-10-17T18:22:17.256 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EQau1hT0v+cZVrQsB6+og69Pg/g= 2023-10-17T18:22:17.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EQau1hT0v+cZVrQsB6+og69Pg/g=', 'Content-Length': '0'} 2023-10-17T18:22:17.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c944ac405203735-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.260 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.260 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.260 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.261 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.261 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.261 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.262 DEBUG:boto:Data: 2023-10-17T18:22:17.262 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.263 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.263 DEBUG:boto:Params: {} 2023-10-17T18:22:17.264 DEBUG:boto:Token: None 2023-10-17T18:22:17.264 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.265 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EQau1hT0v+cZVrQsB6+og69Pg/g= 2023-10-17T18:22:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EQau1hT0v+cZVrQsB6+og69Pg/g=', 'Content-Length': '0'} 2023-10-17T18:22:17.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d000b1687aff278d-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.268 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.268 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.269 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.269 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.269 DEBUG:boto:Method: GET 2023-10-17T18:22:17.270 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.270 DEBUG:boto:Data: 2023-10-17T18:22:17.271 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.271 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.271 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.272 DEBUG:boto:Params: {} 2023-10-17T18:22:17.272 DEBUG:boto:Token: None 2023-10-17T18:22:17.273 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0= 2023-10-17T18:22:17.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0=', 'Content-Length': '0'} 2023-10-17T18:22:17.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022923b5fd2a7aedc-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.318 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.319 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.319 DEBUG:boto:path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.320 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.320 DEBUG:boto:Method: GET 2023-10-17T18:22:17.321 DEBUG:boto:Path: /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.321 DEBUG:boto:Data: 2023-10-17T18:22:17.322 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.323 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.323 DEBUG:boto:Params: {} 2023-10-17T18:22:17.324 DEBUG:boto:Token: None 2023-10-17T18:22:17.324 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u 2023-10-17T18:22:17.325 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0= 2023-10-17T18:22:17.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YWl9JyC2xO/2EXExME4QVbp7So0=', 'Content-Length': '0'} 2023-10-17T18:22:17.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'OKEhtnlf6XQGZFxQtWGUXOIJqaWWH-u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e831ec5faa66f269-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.370 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.371 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.371 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.372 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.372 DEBUG:boto:Method: GET 2023-10-17T18:22:17.373 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.373 DEBUG:boto:Data: 2023-10-17T18:22:17.374 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.375 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.375 DEBUG:boto:Params: {} 2023-10-17T18:22:17.376 DEBUG:boto:Token: None 2023-10-17T18:22:17.376 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcce81403fd82d00-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.380 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.381 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.382 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.382 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.383 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.383 DEBUG:boto:Method: GET 2023-10-17T18:22:17.384 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.384 DEBUG:boto:Data: 2023-10-17T18:22:17.385 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.385 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.386 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.386 DEBUG:boto:Params: {} 2023-10-17T18:22:17.387 DEBUG:boto:Token: None 2023-10-17T18:22:17.387 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001414ac0fe20642df-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.434 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.435 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.435 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.436 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.436 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.436 DEBUG:boto:Data: 2023-10-17T18:22:17.437 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.437 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.438 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.438 DEBUG:boto:Params: {} 2023-10-17T18:22:17.439 DEBUG:boto:Token: None 2023-10-17T18:22:17.439 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003859a03e26e94c5-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.444 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.444 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.445 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.445 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.445 DEBUG:boto:Data: 2023-10-17T18:22:17.446 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.447 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.447 DEBUG:boto:Params: {} 2023-10-17T18:22:17.447 DEBUG:boto:Token: None 2023-10-17T18:22:17.448 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bc5c28959b84f09-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.453 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.453 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.454 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.454 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.455 DEBUG:boto:Method: GET 2023-10-17T18:22:17.455 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.455 DEBUG:boto:Data: 2023-10-17T18:22:17.456 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.457 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.457 DEBUG:boto:Params: {} 2023-10-17T18:22:17.458 DEBUG:boto:Token: None 2023-10-17T18:22:17.458 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.459 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d301f8e100f07cf7-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.502 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.503 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.503 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.504 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.504 DEBUG:boto:Method: GET 2023-10-17T18:22:17.504 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.505 DEBUG:boto:Data: 2023-10-17T18:22:17.505 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.506 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.507 DEBUG:boto:Params: {} 2023-10-17T18:22:17.507 DEBUG:boto:Token: None 2023-10-17T18:22:17.508 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000926c7766d4769a82-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.511 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.512 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.512 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.513 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.513 DEBUG:boto:Method: GET 2023-10-17T18:22:17.513 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.514 DEBUG:boto:Data: 2023-10-17T18:22:17.514 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.515 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.515 DEBUG:boto:Params: {} 2023-10-17T18:22:17.516 DEBUG:boto:Token: None 2023-10-17T18:22:17.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab1b4e216ee276e2-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.520 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.521 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.522 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.522 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.523 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.523 DEBUG:boto:Method: GET 2023-10-17T18:22:17.523 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.524 DEBUG:boto:Data: 2023-10-17T18:22:17.524 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.525 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.525 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.525 DEBUG:boto:Params: {} 2023-10-17T18:22:17.526 DEBUG:boto:Token: None 2023-10-17T18:22:17.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.527 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae786d6b0ed02214-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.531 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.532 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.532 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.533 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.533 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.534 DEBUG:boto:Method: GET 2023-10-17T18:22:17.534 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.535 DEBUG:boto:Data: 2023-10-17T18:22:17.535 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.536 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.536 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.536 DEBUG:boto:Params: {} 2023-10-17T18:22:17.537 DEBUG:boto:Token: None 2023-10-17T18:22:17.538 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.538 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo= 2023-10-17T18:22:17.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo=', 'Content-Length': '0'} 2023-10-17T18:22:17.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065857ed6a8801de1-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.582 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.583 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.583 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.584 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.584 DEBUG:boto:Method: GET 2023-10-17T18:22:17.585 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.585 DEBUG:boto:Data: 2023-10-17T18:22:17.586 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.586 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.587 DEBUG:boto:Params: {} 2023-10-17T18:22:17.587 DEBUG:boto:Token: None 2023-10-17T18:22:17.588 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo= 2023-10-17T18:22:17.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo=', 'Content-Length': '0'} 2023-10-17T18:22:17.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff400df10faa7444-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.638 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.639 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.639 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.640 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.640 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.641 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.641 DEBUG:boto:Data: 2023-10-17T18:22:17.641 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.642 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.643 DEBUG:boto:Params: {} 2023-10-17T18:22:17.643 DEBUG:boto:Token: None 2023-10-17T18:22:17.644 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IOyI+LbDrd/UTlwbYOooUorEj8E= 2023-10-17T18:22:17.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IOyI+LbDrd/UTlwbYOooUorEj8E=', 'Content-Length': '0'} 2023-10-17T18:22:17.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe27765fa8ff2e95-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.647 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.648 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.648 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.648 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.649 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.649 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.650 DEBUG:boto:Data: 2023-10-17T18:22:17.650 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.651 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.651 DEBUG:boto:Params: {} 2023-10-17T18:22:17.653 DEBUG:boto:Token: None 2023-10-17T18:22:17.654 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.654 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IOyI+LbDrd/UTlwbYOooUorEj8E= 2023-10-17T18:22:17.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IOyI+LbDrd/UTlwbYOooUorEj8E=', 'Content-Length': '0'} 2023-10-17T18:22:17.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a620ba8144e8a3e1-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.658 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.658 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.658 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.659 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.659 DEBUG:boto:Method: GET 2023-10-17T18:22:17.659 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.660 DEBUG:boto:Data: 2023-10-17T18:22:17.660 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.661 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.662 DEBUG:boto:Params: {} 2023-10-17T18:22:17.662 DEBUG:boto:Token: None 2023-10-17T18:22:17.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo= 2023-10-17T18:22:17.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo=', 'Content-Length': '0'} 2023-10-17T18:22:17.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009af51227b324220d-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.667 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.667 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.667 DEBUG:boto:path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.668 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.668 DEBUG:boto:Method: GET 2023-10-17T18:22:17.669 DEBUG:boto:Path: /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.669 DEBUG:boto:Data: 2023-10-17T18:22:17.670 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.670 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.671 DEBUG:boto:Params: {} 2023-10-17T18:22:17.671 DEBUG:boto:Token: None 2023-10-17T18:22:17.672 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30 2023-10-17T18:22:17.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo= 2023-10-17T18:22:17.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iHwvF1Se6wDnmLYsfF910GeQeqo=', 'Content-Length': '0'} 2023-10-17T18:22:17.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'Z4w5c4hF6pTYCv5j4hB3P5t7CC-Nr30'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025e5e459984e788a-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.718 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.719 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.719 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.720 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.720 DEBUG:boto:Method: GET 2023-10-17T18:22:17.721 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.721 DEBUG:boto:Data: 2023-10-17T18:22:17.721 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.722 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.723 DEBUG:boto:Params: {} 2023-10-17T18:22:17.723 DEBUG:boto:Token: None 2023-10-17T18:22:17.723 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678aea1803f3a563-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.728 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.728 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.729 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.729 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.729 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.730 DEBUG:boto:Method: GET 2023-10-17T18:22:17.730 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.731 DEBUG:boto:Data: 2023-10-17T18:22:17.731 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.732 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.732 DEBUG:boto:Params: {} 2023-10-17T18:22:17.733 DEBUG:boto:Token: None 2023-10-17T18:22:17.733 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.734 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad15d8a5362fcd6c-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.737 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.738 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.738 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.739 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.739 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.740 DEBUG:boto:Data: 2023-10-17T18:22:17.740 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.740 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.741 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.741 DEBUG:boto:Params: {} 2023-10-17T18:22:17.742 DEBUG:boto:Token: None 2023-10-17T18:22:17.742 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.742 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024e3438975418b61-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.746 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.747 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.747 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.747 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:17.748 DEBUG:boto:Data: 2023-10-17T18:22:17.748 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.749 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.749 DEBUG:boto:Params: {} 2023-10-17T18:22:17.750 DEBUG:boto:Token: None 2023-10-17T18:22:17.750 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj 2023-10-17T18:22:17.751 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI= 2023-10-17T18:22:17.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qLdH0OxXy/ISIHbeVqUk+o+iTaI=', 'Content-Length': '0'} 2023-10-17T18:22:17.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c404bbb6956fffce-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.755 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.755 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.756 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.756 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.757 DEBUG:boto:Method: GET 2023-10-17T18:22:17.757 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.757 DEBUG:boto:Data: 2023-10-17T18:22:17.758 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.759 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.759 DEBUG:boto:Params: {} 2023-10-17T18:22:17.759 DEBUG:boto:Token: None 2023-10-17T18:22:17.760 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2988a2b0ecf28b-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.810 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.811 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.811 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.811 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.812 DEBUG:boto:Method: GET 2023-10-17T18:22:17.812 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.813 DEBUG:boto:Data: 2023-10-17T18:22:17.813 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.814 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.814 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.814 DEBUG:boto:Params: {} 2023-10-17T18:22:17.815 DEBUG:boto:Token: None 2023-10-17T18:22:17.815 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:17.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA= 2023-10-17T18:22:17.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YHllZreyPr2c8DN0zfeSRf4zyxA=', 'Content-Length': '0'} 2023-10-17T18:22:17.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bf84c2ba4c1eda7-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.819 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.819 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.820 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.820 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.821 DEBUG:boto:Method: GET 2023-10-17T18:22:17.821 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.821 DEBUG:boto:Data: 2023-10-17T18:22:17.822 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.823 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.823 DEBUG:boto:Params: {} 2023-10-17T18:22:17.823 DEBUG:boto:Token: None 2023-10-17T18:22:17.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de897b62ade9e95-00652ed0d9-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.870 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.871 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:17.871 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:17.872 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.872 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:17.873 DEBUG:boto:Method: GET 2023-10-17T18:22:17.873 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:17.873 DEBUG:boto:Data: 2023-10-17T18:22:17.874 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.874 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.875 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.875 DEBUG:boto:Params: {} 2023-10-17T18:22:17.876 DEBUG:boto:Token: None 2023-10-17T18:22:17.876 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/ 2023-10-17T18:22:17.877 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI= 2023-10-17T18:22:17.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T+sEA3CRltE28xAJKEwk8378RZI=', 'Content-Length': '0'} 2023-10-17T18:22:17.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe8ae79e28eb1cb-00652ed0d9-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.880 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:17.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.881 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.882 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.882 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.883 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.883 DEBUG:boto:Method: GET 2023-10-17T18:22:17.883 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.884 DEBUG:boto:Data: 2023-10-17T18:22:17.884 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.885 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.885 DEBUG:boto:Params: {} 2023-10-17T18:22:17.886 DEBUG:boto:Token: None 2023-10-17T18:22:17.886 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.887 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w= 2023-10-17T18:22:17.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w=', 'Content-Length': '0'} 2023-10-17T18:22:17.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000181e8056147d753f-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.890 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.891 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.891 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.892 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.892 DEBUG:boto:Method: GET 2023-10-17T18:22:17.892 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.893 DEBUG:boto:Data: 2023-10-17T18:22:17.893 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.894 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.894 DEBUG:boto:Params: {} 2023-10-17T18:22:17.895 DEBUG:boto:Token: None 2023-10-17T18:22:17.895 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w= 2023-10-17T18:22:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w=', 'Content-Length': '0'} 2023-10-17T18:22:17.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035e8af70a70a029c-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.938 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.939 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.939 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.940 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.940 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.940 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.941 DEBUG:boto:Data: 2023-10-17T18:22:17.941 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.941 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.942 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.942 DEBUG:boto:Params: {} 2023-10-17T18:22:17.943 DEBUG:boto:Token: None 2023-10-17T18:22:17.943 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RNCfjngfY2gMlFacsaL3cgJtTOw= 2023-10-17T18:22:17.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RNCfjngfY2gMlFacsaL3cgJtTOw=', 'Content-Length': '0'} 2023-10-17T18:22:17.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc1c3d3fd385493c-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.947 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.947 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.948 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.948 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.949 DEBUG:boto:Method: HEAD 2023-10-17T18:22:17.949 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.949 DEBUG:boto:Data: 2023-10-17T18:22:17.950 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.950 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:17.951 DEBUG:boto:Port: 8000 2023-10-17T18:22:17.951 DEBUG:boto:Params: {} 2023-10-17T18:22:17.951 DEBUG:boto:Token: None 2023-10-17T18:22:17.952 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.952 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RNCfjngfY2gMlFacsaL3cgJtTOw= 2023-10-17T18:22:17.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RNCfjngfY2gMlFacsaL3cgJtTOw=', 'Content-Length': '0'} 2023-10-17T18:22:17.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085d4c6569101ed52-00652ed0d9-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:17.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:17.955 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:17.956 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:17.956 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.957 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.957 DEBUG:boto:Method: GET 2023-10-17T18:22:17.957 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.958 DEBUG:boto:Data: 2023-10-17T18:22:17.958 DEBUG:boto:Headers: {} 2023-10-17T18:22:17.959 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:17.959 DEBUG:boto:Port: 8001 2023-10-17T18:22:17.960 DEBUG:boto:Params: {} 2023-10-17T18:22:17.960 DEBUG:boto:Token: None 2023-10-17T18:22:17.960 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:17 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:17.961 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w= 2023-10-17T18:22:17.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b0S9K3zSzKKjLPguUzT7l7ysB8w=', 'Content-Length': '0'} 2023-10-17T18:22:17.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fc14a6016f04a36-00652ed0d9-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.006 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:18.007 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:18.007 DEBUG:boto:path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:18.008 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:18.008 DEBUG:boto:Method: GET 2023-10-17T18:22:18.008 DEBUG:boto:Path: /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:18.009 DEBUG:boto:Data: 2023-10-17T18:22:18.009 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.010 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.010 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.011 DEBUG:boto:Params: {} 2023-10-17T18:22:18.011 DEBUG:boto:Token: None 2023-10-17T18:22:18.011 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/obj?versionId=pHuNXlITqWHRXWomQb1bjamN0YoudhQ 2023-10-17T18:22:18.012 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r68WcgvYo5yeInUFZM7ne5xOXhg= 2023-10-17T18:22:18.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r68WcgvYo5yeInUFZM7ne5xOXhg=', 'Content-Length': '0'} 2023-10-17T18:22:18.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:11 GMT'), ('x-amz-version-id', 'pHuNXlITqWHRXWomQb1bjamN0YoudhQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000331245881cdd68fe-00652ed0da-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.015 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:18.016 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:18.016 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.016 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.017 DEBUG:boto:Method: GET 2023-10-17T18:22:18.017 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:18.018 DEBUG:boto:Data: 2023-10-17T18:22:18.018 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:18.019 DEBUG:boto:Port: 8001 2023-10-17T18:22:18.019 DEBUG:boto:Params: {} 2023-10-17T18:22:18.020 DEBUG:boto:Token: None 2023-10-17T18:22:18.020 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/ 2023-10-17T18:22:18.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8= 2023-10-17T18:22:18.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8=', 'Content-Length': '0'} 2023-10-17T18:22:18.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895fa9d7a992b73e-00652ed0da-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.066 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:18.067 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:18.067 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:18.068 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.068 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.068 DEBUG:boto:Method: GET 2023-10-17T18:22:18.069 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:18.069 DEBUG:boto:Data: 2023-10-17T18:22:18.070 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.070 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.071 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.071 DEBUG:boto:Params: {} 2023-10-17T18:22:18.071 DEBUG:boto:Token: None 2023-10-17T18:22:18.072 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/ 2023-10-17T18:22:18.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8= 2023-10-17T18:22:18.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8=', 'Content-Length': '0'} 2023-10-17T18:22:18.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042df07d6757c52b1-00652ed0da-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.076 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:18.076 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:18.077 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:18.077 DEBUG:boto:Method: HEAD 2023-10-17T18:22:18.078 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:18.078 DEBUG:boto:Data: 2023-10-17T18:22:18.078 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:18.079 DEBUG:boto:Port: 8001 2023-10-17T18:22:18.080 DEBUG:boto:Params: {} 2023-10-17T18:22:18.080 DEBUG:boto:Token: None 2023-10-17T18:22:18.080 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/obj 2023-10-17T18:22:18.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hXdkQHqnJNN/sEsylfKpw7iCkrI= 2023-10-17T18:22:18.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hXdkQHqnJNN/sEsylfKpw7iCkrI=', 'Content-Length': '0'} 2023-10-17T18:22:18.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002694dc2953b149b8-00652ed0da-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.085 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:18.085 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:18.086 DEBUG:boto:Method: HEAD 2023-10-17T18:22:18.086 DEBUG:boto:Path: /hbmuog-13/obj 2023-10-17T18:22:18.087 DEBUG:boto:Data: 2023-10-17T18:22:18.087 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.087 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.088 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.088 DEBUG:boto:Params: {} 2023-10-17T18:22:18.088 DEBUG:boto:Token: None 2023-10-17T18:22:18.089 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/obj 2023-10-17T18:22:18.089 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hXdkQHqnJNN/sEsylfKpw7iCkrI= 2023-10-17T18:22:18.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hXdkQHqnJNN/sEsylfKpw7iCkrI=', 'Content-Length': '0'} 2023-10-17T18:22:18.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae88fa178b951182-00652ed0da-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:18.094 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:18.094 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:18.095 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.095 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.095 DEBUG:boto:Method: GET 2023-10-17T18:22:18.096 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.096 DEBUG:boto:Data: 2023-10-17T18:22:18.097 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:18.097 DEBUG:boto:Port: 8001 2023-10-17T18:22:18.098 DEBUG:boto:Params: {} 2023-10-17T18:22:18.098 DEBUG:boto:Token: None 2023-10-17T18:22:18.099 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KoerwsJSW2DUT4y5HmcaJYWI3wM= 2023-10-17T18:22:18.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KoerwsJSW2DUT4y5HmcaJYWI3wM=', 'Content-Length': '0'} 2023-10-17T18:22:18.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000357d168ff8b4efc9-00652ed0da-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.142 DEBUG:boto:path=/hbmuog-13/obj 2023-10-17T18:22:18.143 DEBUG:boto:auth_path=/hbmuog-13/obj 2023-10-17T18:22:18.143 DEBUG:boto:path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.144 DEBUG:boto:auth_path=/hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.144 DEBUG:boto:Method: GET 2023-10-17T18:22:18.145 DEBUG:boto:Path: /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.145 DEBUG:boto:Data: 2023-10-17T18:22:18.145 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.146 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.147 DEBUG:boto:Params: {} 2023-10-17T18:22:18.147 DEBUG:boto:Token: None 2023-10-17T18:22:18.148 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/obj?versionId=9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh 2023-10-17T18:22:18.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KoerwsJSW2DUT4y5HmcaJYWI3wM= 2023-10-17T18:22:18.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KoerwsJSW2DUT4y5HmcaJYWI3wM=', 'Content-Length': '0'} 2023-10-17T18:22:18.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:21:12 GMT'), ('x-amz-version-id', '9fzmmSUjW9dPgBqBIxdsPXb9EQlcKMh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000323ad0e09de3f936-00652ed0da-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.151 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:18.152 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:18.152 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.153 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.153 DEBUG:boto:Method: GET 2023-10-17T18:22:18.153 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:18.154 DEBUG:boto:Data: 2023-10-17T18:22:18.154 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:18.155 DEBUG:boto:Port: 8001 2023-10-17T18:22:18.155 DEBUG:boto:Params: {} 2023-10-17T18:22:18.156 DEBUG:boto:Token: None 2023-10-17T18:22:18.156 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/ 2023-10-17T18:22:18.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8= 2023-10-17T18:22:18.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8=', 'Content-Length': '0'} 2023-10-17T18:22:18.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac25f49d2f4e416-00652ed0da-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.202 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:18.203 DEBUG:boto:path=/hbmuog-13/ 2023-10-17T18:22:18.203 DEBUG:boto:auth_path=/hbmuog-13/ 2023-10-17T18:22:18.204 DEBUG:boto:path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.204 DEBUG:boto:auth_path=/hbmuog-13/?prefix=obj 2023-10-17T18:22:18.204 DEBUG:boto:Method: GET 2023-10-17T18:22:18.205 DEBUG:boto:Path: /hbmuog-13/?prefix=obj 2023-10-17T18:22:18.205 DEBUG:boto:Data: 2023-10-17T18:22:18.206 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.207 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.207 DEBUG:boto:Params: {} 2023-10-17T18:22:18.207 DEBUG:boto:Token: None 2023-10-17T18:22:18.208 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-13/ 2023-10-17T18:22:18.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8= 2023-10-17T18:22:18.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rA2deuax9IpAnl9Un9svDwmnzr8=', 'Content-Length': '0'} 2023-10-17T18:22:18.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd59804c28895d17-00652ed0da-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.212 DEBUG:boto:b'hbmuog-13obj1000falseobj2023-10-17T18:21:12.472Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:18.213 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-13 zones={a2, a1} 2023-10-17T18:22:18.213 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T18:22:18.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.215 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.321 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:22:18.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.322 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:22:18.322 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:22:18.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.323 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-17T18:22:18.323 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:29.380239Z" 2023-10-17T18:22:18.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-10-17T18:22:18.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:21:40.173076Z" 2023-10-17T18:22:18.325 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.325 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:22:18.326 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-17T18:22:18.326 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:22:18.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.327 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.380 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.380 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.422 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:22:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:22:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:22:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:22:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:22:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:22:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:22:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:22:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:22:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-17T18:22:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.380239Z" 2023-10-17T18:22:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:22:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:22:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:22:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-10-17T18:22:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:21:40.173076Z" 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:22:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:22:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:22:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:22:18Z" 2023-10-17T18:22:18.433 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:22:18.434 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:21:09.376621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.380239Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:21:40.173076Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:22:18Z" } 2023-10-17T18:22:18.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:21:09.376621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.380239Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:21:40.173076Z'}}] 2023-10-17T18:22:18.435 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-17T18:22:18.435 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-17T18:22:18.436 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:22:18.436 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T18:22:18.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.437 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.486 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.487 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.532 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:22:18.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.532 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:22:18.532 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:22:18.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.533 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-17T18:22:18.533 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:15:29.385966Z" 2023-10-17T18:22:18.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.534 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:22:18.534 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:22:18.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.535 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-10-17T18:22:18.535 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:22:02.131498Z" 2023-10-17T18:22:18.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.535 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:22:18.536 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-10-17T18:22:18.537 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:22:18.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.538 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.590 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.591 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.632 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:22:18.632 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:22:18.632 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:22:18.633 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:22:18.633 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:22:18.633 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:22:18.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.634 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:22:18.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.634 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:22:18.634 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.634 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:22:18.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.637 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-17T18:22:18.637 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.637 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.637 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:15:29.385966Z" 2023-10-17T18:22:18.638 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.638 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:22:18.638 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.639 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:22:18.639 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.639 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:22:18.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.641 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:22:18.641 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.641 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:22:18.641 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-10-17T18:22:18.641 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.642 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.642 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.642 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:22:02.131498Z" 2023-10-17T18:22:18.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:22:18.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.644 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:22:18Z" 2023-10-17T18:22:18.644 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:22:18.645 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:15:29.385966Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002337", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:22:02.131498Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:22:18Z" } 2023-10-17T18:22:18.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:15:29.385966Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002337', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:22:02.131498Z'}}] 2023-10-17T18:22:18.646 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-10-17T18:22:18.646 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002337'} 2023-10-17T18:22:18.646 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:22:18.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-10-17T18:22:18.648 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:18.649 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:18.649 DEBUG:boto:Method: PUT 2023-10-17T18:22:18.649 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:22:18.650 DEBUG:boto:Data: 2023-10-17T18:22:18.650 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.651 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.651 DEBUG:boto:Params: {} 2023-10-17T18:22:18.652 DEBUG:boto:Token: None 2023-10-17T18:22:18.652 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-14/ 2023-10-17T18:22:18.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o17JXL0uQYUgeSyp5BGHB8LuG2o= 2023-10-17T18:22:18.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o17JXL0uQYUgeSyp5BGHB8LuG2o=', 'Content-Length': '0'} 2023-10-17T18:22:18.665 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e95539520efb8ad-00652ed0da-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.665 DEBUG:rgw_multi.tests:created bucket=hbmuog-14 2023-10-17T18:22:18.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.667 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.716 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.716 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:22:18.759 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:22:18.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.759 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566938.664260_180.1", 2023-10-17T18:22:18.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:22:18.664260Z" 2023-10-17T18:22:18.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.760 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:22:18.760 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:22:18.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.761 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:18.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:22:18.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.762 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:18.762 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:22:18.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.762 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:22:18.763 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566938.664260_180.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:18.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:22:18.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:18.765 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:18.817 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.817 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:18.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:18.844+0000 7f4a9810a780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:22:18.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:18.845+0000 7f4a9810a780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:22:18.858 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:22:18.858 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:22:18.858 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:22:18.858 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:22:18.859 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:22:18.859 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:22:18.859 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:18.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.859 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:22:18.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.860 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:22:18.860 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.860 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:18.860 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566938.664260_180.1", 2023-10-17T18:22:18.861 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.861 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:22:18.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.861 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:22:18.664260Z", 2023-10-17T18:22:18.862 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:18.862 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.862 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:22:18.863 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.863 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:18.863 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:22:18.863 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.863 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.864 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.864 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:22:18.864 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:18.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.865 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:22:18.865 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.865 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:18.866 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:18.866 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.866 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.866 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.866 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:22:18.867 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:18.867 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:18.868 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:22:18.868 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:18.868 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:18.868 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:18.868 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.870 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:18.870 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:22:18.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.870 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:22:18.870 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:22:18.871 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:22:18.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:18.871 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:22:18Z" 2023-10-17T18:22:18.871 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:22:18.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566938.664260_180.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:22:18.664260Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:22:18Z" } 2023-10-17T18:22:18.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566938.664260_180.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:22:18.664260Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:22:18.873 DEBUG:rgw_multi.tests:log_status={0: '1_1697566938.664260_180.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:18.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566938.664260_180.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:18.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:22:18.875 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:18.875 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:18.875 DEBUG:boto:Method: HEAD 2023-10-17T18:22:18.876 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:22:18.876 DEBUG:boto:Data: 2023-10-17T18:22:18.877 DEBUG:boto:Headers: {} 2023-10-17T18:22:18.877 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.877 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.878 DEBUG:boto:Params: {} 2023-10-17T18:22:18.878 DEBUG:boto:Token: None 2023-10-17T18:22:18.879 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-14/ 2023-10-17T18:22:18.879 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVkXzxCwbHSCyO3OsH+I8TA38/w= 2023-10-17T18:22:18.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVkXzxCwbHSCyO3OsH+I8TA38/w=', 'Content-Length': '0'} 2023-10-17T18:22:18.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a462c7a010db30d-00652ed0da-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.883 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:18.883 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:18.884 DEBUG:boto:Method: PUT 2023-10-17T18:22:18.884 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:18.885 DEBUG:boto:Data: 2023-10-17T18:22:18.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:22:18.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:18.886 DEBUG:boto:Port: 8000 2023-10-17T18:22:18.886 DEBUG:boto:Params: {} 2023-10-17T18:22:18.887 DEBUG:boto:Token: None 2023-10-17T18:22:18.887 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:22:18 GMT /hbmuog-14/obj 2023-10-17T18:22:18.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BQhg10cbonazkxTPow0wlJBkigE= 2023-10-17T18:22:18.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:22:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BQhg10cbonazkxTPow0wlJBkigE='} 2023-10-17T18:22:18.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000db57fc356d848b9-00652ed0da-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:18.931 DEBUG:rgw_multi.tests:created initial version id=None 2023-10-17T18:22:18.932 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-14 2023-10-17T18:22:18.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:18.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:18.933 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:22:18.991 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:19.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:19.024+0000 7f8d5500e780 1 waiting to reach incremental sync.. 2023-10-17T18:22:24.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:24.025+0000 7f8d5500e780 1 waiting to reach incremental sync.. 2023-10-17T18:22:29.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:29.026+0000 7f8d5500e780 1 waiting to reach incremental sync.. 2023-10-17T18:22:34.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:34.028+0000 7f8d5500e780 1 bucket sync caught up with source: 2023-10-17T18:22:34.029 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-10-17T18:22:34.029 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.5.6, , ] 2023-10-17T18:22:34.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:34.028+0000 7f8d5500e780 0 bucket checkpoint complete 2023-10-17T18:22:34.044 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-14 2023-10-17T18:22:34.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:34.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:34.046 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:22:34.112 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:34.112 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:34.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:34.148+0000 7f13ffa55780 1 waiting to reach incremental sync.. 2023-10-17T18:22:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:39.148+0000 7f13ffa55780 1 waiting to reach incremental sync.. 2023-10-17T18:22:44.150 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:44.149+0000 7f13ffa55780 1 waiting to reach incremental sync.. 2023-10-17T18:22:49.150 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:49.149+0000 7f13ffa55780 1 waiting to reach incremental sync.. 2023-10-17T18:22:54.152 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:54.151+0000 7f13ffa55780 1 bucket sync caught up with source: 2023-10-17T18:22:54.152 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:22:54.153 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:22:54.153 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:54.151+0000 7f13ffa55780 0 bucket checkpoint complete 2023-10-17T18:22:54.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:22:54.169 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.169 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.170 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.170 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:22:54.171 DEBUG:boto:Data: 2023-10-17T18:22:54.171 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.172 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.172 DEBUG:boto:Params: {} 2023-10-17T18:22:54.173 DEBUG:boto:Token: None 2023-10-17T18:22:54.173 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.174 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg= 2023-10-17T18:22:54.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg=', 'Content-Length': '0'} 2023-10-17T18:22:54.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005df4f4bf820e24d3-00652ed0fe-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.178 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.178 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.178 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.179 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:22:54.179 DEBUG:boto:Data: 2023-10-17T18:22:54.180 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.180 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.181 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.181 DEBUG:boto:Params: {} 2023-10-17T18:22:54.181 DEBUG:boto:Token: None 2023-10-17T18:22:54.182 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.182 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg= 2023-10-17T18:22:54.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg=', 'Content-Length': '0'} 2023-10-17T18:22:54.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3d9afcf01e71837-00652ed0fe-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:22:54.187 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.188 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.188 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.188 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.189 DEBUG:boto:Method: GET 2023-10-17T18:22:54.189 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.190 DEBUG:boto:Data: 2023-10-17T18:22:54.190 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.191 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.191 DEBUG:boto:Params: {} 2023-10-17T18:22:54.192 DEBUG:boto:Token: None 2023-10-17T18:22:54.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/?versions 2023-10-17T18:22:54.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A= 2023-10-17T18:22:54.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A=', 'Content-Length': '0'} 2023-10-17T18:22:54.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8bc238ceca11e7-00652ed0fe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.197 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:54.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:22:54.198 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.199 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.199 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.200 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.200 DEBUG:boto:Method: GET 2023-10-17T18:22:54.201 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.201 DEBUG:boto:Data: 2023-10-17T18:22:54.201 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.202 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.203 DEBUG:boto:Params: {} 2023-10-17T18:22:54.203 DEBUG:boto:Token: None 2023-10-17T18:22:54.203 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/?versions 2023-10-17T18:22:54.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A= 2023-10-17T18:22:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A=', 'Content-Length': '0'} 2023-10-17T18:22:54.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16a603d1801bb64-00652ed0fe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.208 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:22:54.209 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.209 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.210 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.210 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.211 DEBUG:boto:Method: GET 2023-10-17T18:22:54.212 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.212 DEBUG:boto:Data: 2023-10-17T18:22:54.212 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.213 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.214 DEBUG:boto:Params: {} 2023-10-17T18:22:54.214 DEBUG:boto:Token: None 2023-10-17T18:22:54.215 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/?versions 2023-10-17T18:22:54.216 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A= 2023-10-17T18:22:54.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A=', 'Content-Length': '0'} 2023-10-17T18:22:54.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eebca4e60aed2380-00652ed0fe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.219 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.220 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.220 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.221 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.221 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.222 DEBUG:boto:Method: GET 2023-10-17T18:22:54.222 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:22:54.223 DEBUG:boto:Data: 2023-10-17T18:22:54.223 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.223 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.224 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.224 DEBUG:boto:Params: {} 2023-10-17T18:22:54.225 DEBUG:boto:Token: None 2023-10-17T18:22:54.225 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/?versions 2023-10-17T18:22:54.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A= 2023-10-17T18:22:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rumunlRDmFfyYF3l+4Vok7WDH6A=', 'Content-Length': '0'} 2023-10-17T18:22:54.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c77aeab38fb0165-00652ed0fe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.229 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:54.230 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.231 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.231 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.232 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.232 DEBUG:boto:Method: GET 2023-10-17T18:22:54.233 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.233 DEBUG:boto:Data: 2023-10-17T18:22:54.233 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.234 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.234 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.235 DEBUG:boto:Params: {} 2023-10-17T18:22:54.235 DEBUG:boto:Token: None 2023-10-17T18:22:54.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.236 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE= 2023-10-17T18:22:54.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE=', 'Content-Length': '0'} 2023-10-17T18:22:54.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000011b38013f8a4b8-00652ed0fe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.239 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.239 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.240 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.240 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.241 DEBUG:boto:Method: GET 2023-10-17T18:22:54.241 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.242 DEBUG:boto:Data: 2023-10-17T18:22:54.242 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.243 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.243 DEBUG:boto:Params: {} 2023-10-17T18:22:54.244 DEBUG:boto:Token: None 2023-10-17T18:22:54.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE= 2023-10-17T18:22:54.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE=', 'Content-Length': '0'} 2023-10-17T18:22:54.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000139584caac2c57c5-00652ed0fe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.247 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.248 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.248 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.249 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.249 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.250 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.250 DEBUG:boto:Data: 2023-10-17T18:22:54.251 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.251 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.252 DEBUG:boto:Params: {} 2023-10-17T18:22:54.252 DEBUG:boto:Token: None 2023-10-17T18:22:54.253 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xsAShiEd6N6HtmE3m8jkYHMeOpM= 2023-10-17T18:22:54.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xsAShiEd6N6HtmE3m8jkYHMeOpM=', 'Content-Length': '0'} 2023-10-17T18:22:54.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f80951b677a9b99-00652ed0fe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.256 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.257 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.257 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.257 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.258 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.258 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.259 DEBUG:boto:Data: 2023-10-17T18:22:54.259 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.260 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.260 DEBUG:boto:Params: {} 2023-10-17T18:22:54.261 DEBUG:boto:Token: None 2023-10-17T18:22:54.261 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.262 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xsAShiEd6N6HtmE3m8jkYHMeOpM= 2023-10-17T18:22:54.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xsAShiEd6N6HtmE3m8jkYHMeOpM=', 'Content-Length': '0'} 2023-10-17T18:22:54.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5de56eb38c31ed9-00652ed0fe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:54.265 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.266 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.266 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.267 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.267 DEBUG:boto:Method: GET 2023-10-17T18:22:54.268 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.268 DEBUG:boto:Data: 2023-10-17T18:22:54.268 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.269 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.269 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.270 DEBUG:boto:Params: {} 2023-10-17T18:22:54.270 DEBUG:boto:Token: None 2023-10-17T18:22:54.271 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.271 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE= 2023-10-17T18:22:54.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE=', 'Content-Length': '0'} 2023-10-17T18:22:54.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077be02395811d430-00652ed0fe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.274 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.275 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.275 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.275 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.276 DEBUG:boto:Method: GET 2023-10-17T18:22:54.276 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.276 DEBUG:boto:Data: 2023-10-17T18:22:54.277 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.277 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.278 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.279 DEBUG:boto:Params: {} 2023-10-17T18:22:54.279 DEBUG:boto:Token: None 2023-10-17T18:22:54.280 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:22:54.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE= 2023-10-17T18:22:54.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFmBDu2RWwMsqbB6X1S8u9S8dRE=', 'Content-Length': '0'} 2023-10-17T18:22:54.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065cdb517b1230ff5-00652ed0fe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.284 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.284 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.284 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.285 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.285 DEBUG:boto:Method: GET 2023-10-17T18:22:54.286 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:22:54.286 DEBUG:boto:Data: 2023-10-17T18:22:54.287 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.288 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.288 DEBUG:boto:Params: {} 2023-10-17T18:22:54.288 DEBUG:boto:Token: None 2023-10-17T18:22:54.289 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0= 2023-10-17T18:22:54.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0=', 'Content-Length': '0'} 2023-10-17T18:22:54.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2a51eaa426f3348-00652ed0fe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.293 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.294 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.294 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.295 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.295 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.295 DEBUG:boto:Method: GET 2023-10-17T18:22:54.296 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:22:54.296 DEBUG:boto:Data: 2023-10-17T18:22:54.297 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.297 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.297 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.298 DEBUG:boto:Params: {} 2023-10-17T18:22:54.298 DEBUG:boto:Token: None 2023-10-17T18:22:54.299 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.299 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0= 2023-10-17T18:22:54.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0=', 'Content-Length': '0'} 2023-10-17T18:22:54.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681c3fbc8cc99505-00652ed0fe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.346 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.347 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.347 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.348 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.348 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:54.349 DEBUG:boto:Data: 2023-10-17T18:22:54.349 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.350 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.350 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.350 DEBUG:boto:Params: {} 2023-10-17T18:22:54.351 DEBUG:boto:Token: None 2023-10-17T18:22:54.351 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj 2023-10-17T18:22:54.352 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DjM2V8Msi7djyskFMP7l13wlIgQ= 2023-10-17T18:22:54.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DjM2V8Msi7djyskFMP7l13wlIgQ=', 'Content-Length': '0'} 2023-10-17T18:22:54.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023a82088573fa72e-00652ed0fe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.355 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.355 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.356 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.356 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:54.357 DEBUG:boto:Data: 2023-10-17T18:22:54.357 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.358 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.358 DEBUG:boto:Params: {} 2023-10-17T18:22:54.359 DEBUG:boto:Token: None 2023-10-17T18:22:54.359 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj 2023-10-17T18:22:54.360 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DjM2V8Msi7djyskFMP7l13wlIgQ= 2023-10-17T18:22:54.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DjM2V8Msi7djyskFMP7l13wlIgQ=', 'Content-Length': '0'} 2023-10-17T18:22:54.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092f6314bf6668641-00652ed0fe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:22:54.363 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.363 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.364 DEBUG:boto:Method: GET 2023-10-17T18:22:54.364 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:54.365 DEBUG:boto:Data: 2023-10-17T18:22:54.365 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.366 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.367 DEBUG:boto:Params: {} 2023-10-17T18:22:54.367 DEBUG:boto:Token: None 2023-10-17T18:22:54.367 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj 2023-10-17T18:22:54.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kyzx6S0+bV2rXeEKvHuWQ8FdEhU= 2023-10-17T18:22:54.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kyzx6S0+bV2rXeEKvHuWQ8FdEhU=', 'Content-Length': '0'} 2023-10-17T18:22:54.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049515ee0d513a031-00652ed0fe-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.371 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.371 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.372 DEBUG:boto:Method: GET 2023-10-17T18:22:54.372 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:54.373 DEBUG:boto:Data: 2023-10-17T18:22:54.373 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.374 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.375 DEBUG:boto:Params: {} 2023-10-17T18:22:54.375 DEBUG:boto:Token: None 2023-10-17T18:22:54.376 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj 2023-10-17T18:22:54.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kyzx6S0+bV2rXeEKvHuWQ8FdEhU= 2023-10-17T18:22:54.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kyzx6S0+bV2rXeEKvHuWQ8FdEhU=', 'Content-Length': '0'} 2023-10-17T18:22:54.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5680f9dd0157700-00652ed0fe-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.379 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.380 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.380 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.381 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.382 DEBUG:boto:Method: GET 2023-10-17T18:22:54.382 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:22:54.383 DEBUG:boto:Data: 2023-10-17T18:22:54.383 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:22:54.384 DEBUG:boto:Port: 8001 2023-10-17T18:22:54.384 DEBUG:boto:Params: {} 2023-10-17T18:22:54.385 DEBUG:boto:Token: None 2023-10-17T18:22:54.385 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.386 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0= 2023-10-17T18:22:54.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0=', 'Content-Length': '0'} 2023-10-17T18:22:54.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017af9a69b4c73a96-00652ed0fe-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.430 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.431 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.432 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.432 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.433 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:22:54.433 DEBUG:boto:Method: GET 2023-10-17T18:22:54.433 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:22:54.434 DEBUG:boto:Data: 2023-10-17T18:22:54.434 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.435 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.435 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.436 DEBUG:boto:Params: {} 2023-10-17T18:22:54.436 DEBUG:boto:Token: None 2023-10-17T18:22:54.437 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.438 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0= 2023-10-17T18:22:54.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qh30CdWhR9+qQvK6yXdLYNF59d0=', 'Content-Length': '0'} 2023-10-17T18:22:54.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006150f1138cc45633-00652ed0fe-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.441 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:22:54.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:22:54.443 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.443 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.444 DEBUG:boto:path=/hbmuog-14/?versioning 2023-10-17T18:22:54.444 DEBUG:boto:auth_path=/hbmuog-14/?versioning 2023-10-17T18:22:54.445 DEBUG:boto:Method: PUT 2023-10-17T18:22:54.445 DEBUG:boto:Path: /hbmuog-14/?versioning 2023-10-17T18:22:54.445 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:22:54.446 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.447 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.447 DEBUG:boto:Params: {} 2023-10-17T18:22:54.448 DEBUG:boto:Token: None 2023-10-17T18:22:54.448 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/?versioning 2023-10-17T18:22:54.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WhYEOqVQhUt+YvhRTVAVdYdZkuA= 2023-10-17T18:22:54.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WhYEOqVQhUt+YvhRTVAVdYdZkuA=', 'Content-Length': '228'} 2023-10-17T18:22:54.455 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a23eb40a7c441e25-00652ed0fe-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:54.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:54.457 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:54.545 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:22:54.557 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:22:54.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.558 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566974.454939_184.1", 2023-10-17T18:22:54.558 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:22:54.454939Z" 2023-10-17T18:22:54.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.559 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:22:54.559 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:22:54.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.560 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:54.560 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:22:54.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.561 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:54.561 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:22:54.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.561 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:22:54.563 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697566974.454939_184.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:54.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:22:54.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:54.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:22:54.564 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:22:54.620 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:54.620 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:54.648 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:54.647+0000 7f6f36e99780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:22:54.649 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:54.648+0000 7f6f36e99780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:22:54.660 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:22:54.661 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:22:54.661 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:22:54.661 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:22:54.661 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:22:54.662 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:22:54.662 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:54.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.662 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:22:54.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.663 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:22:54.663 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:54.663 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:54.664 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566974.454939_184.1", 2023-10-17T18:22:54.664 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:54.664 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:22:54.664 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:54.665 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:22:54.454939Z", 2023-10-17T18:22:54.665 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:54.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.666 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:22:54.666 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:54.666 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:54.667 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:22:54.667 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:54.667 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:54.667 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:54.667 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:22:54.668 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:54.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.668 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:22:54.669 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:54.669 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:54.669 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:54.669 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:54.669 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:54.670 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:54.670 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:22:54.670 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:54.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:22:54.671 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:22:54.671 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:22:54.671 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:22:54.671 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:22:54.672 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:22:54.672 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:22:54.672 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:22:54.672 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:22:54.672 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:22:54.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:22:54.673 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:22:54.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.674 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:22:54.674 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:22:54.674 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:22:54.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:22:54.674 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:22:54Z" 2023-10-17T18:22:54.675 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:22:54.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697566974.454939_184.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:22:54.454939Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:22:54Z" } 2023-10-17T18:22:54.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697566974.454939_184.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:22:54.454939Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:22:54.677 DEBUG:rgw_multi.tests:log_status={0: '1_1697566974.454939_184.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:54.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1697566974.454939_184.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:22:54.677 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:22:54.678 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:22:54.678 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:22:54.679 DEBUG:boto:Method: HEAD 2023-10-17T18:22:54.679 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:22:54.680 DEBUG:boto:Data: 2023-10-17T18:22:54.680 DEBUG:boto:Headers: {} 2023-10-17T18:22:54.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.681 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.682 DEBUG:boto:Params: {} 2023-10-17T18:22:54.682 DEBUG:boto:Token: None 2023-10-17T18:22:54.683 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/ 2023-10-17T18:22:54.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg= 2023-10-17T18:22:54.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aFfPkt05zFMN2+vkNxxXU0CGgIg=', 'Content-Length': '0'} 2023-10-17T18:22:54.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019f608d567a42a7e-00652ed0fe-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.686 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:22:54.687 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:22:54.687 DEBUG:boto:Method: PUT 2023-10-17T18:22:54.688 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:22:54.688 DEBUG:boto:Data: 2023-10-17T18:22:54.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:22:54.689 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:22:54.690 DEBUG:boto:Port: 8000 2023-10-17T18:22:54.690 DEBUG:boto:Params: {} 2023-10-17T18:22:54.691 DEBUG:boto:Token: None 2023-10-17T18:22:54.692 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:22:54 GMT /hbmuog-14/obj 2023-10-17T18:22:54.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ezR2yKeDNCXds1M12ayxEsLudGA= 2023-10-17T18:22:54.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:22:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ezR2yKeDNCXds1M12ayxEsLudGA='} 2023-10-17T18:22:54.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-amz-request-id', 'tx000008e4136b37dda07d8-00652ed0fe-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:22:54.704 DEBUG:rgw_multi.tests:created new version id=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:22:54.705 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-14 2023-10-17T18:22:54.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:54.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:22:54.706 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:22:54.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:22:54.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:22:54.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:54.798+0000 7f9ce3ea5780 1 waiting for incremental sync to catch up: 2023-10-17T18:22:54.800 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-10-17T18:22:54.800 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:22:59.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:22:59.799+0000 7f9ce3ea5780 1 waiting for incremental sync to catch up: 2023-10-17T18:22:59.801 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-10-17T18:22:59.801 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:04.802+0000 7f9ce3ea5780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:04.802 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-10-17T18:23:04.803 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:09.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:09.803+0000 7f9ce3ea5780 1 bucket sync caught up with source: 2023-10-17T18:23:09.803 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:09.804 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:09.803+0000 7f9ce3ea5780 0 bucket checkpoint complete 2023-10-17T18:23:09.820 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-14 2023-10-17T18:23:09.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:09.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:09.822 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:23:09.887 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:23:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:23:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:09.925+0000 7f074691c780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:09.925 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:23:09.926 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:14.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:14.926+0000 7f074691c780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:14.927 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:23:14.927 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:19.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:19.926+0000 7f074691c780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:19.928 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:23:19.928 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:24.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:24.928+0000 7f074691c780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:24.930 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.19.6, , ] 2023-10-17T18:23:24.930 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:29.931 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:29.929+0000 7f074691c780 1 bucket sync caught up with source: 2023-10-17T18:23:29.931 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:29.931 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:29.931 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:29.929+0000 7f074691c780 0 bucket checkpoint complete 2023-10-17T18:23:29.947 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:23:29.948 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:29.948 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:29.949 DEBUG:boto:Method: HEAD 2023-10-17T18:23:29.949 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:23:29.950 DEBUG:boto:Data: 2023-10-17T18:23:29.950 DEBUG:boto:Headers: {} 2023-10-17T18:23:29.951 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:29.951 DEBUG:boto:Port: 8001 2023-10-17T18:23:29.951 DEBUG:boto:Params: {} 2023-10-17T18:23:29.952 DEBUG:boto:Token: None 2023-10-17T18:23:29.952 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:29 GMT /hbmuog-14/ 2023-10-17T18:23:29.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XYi7Z7J8u1feg0VdIPR1hBkzsMs= 2023-10-17T18:23:29.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XYi7Z7J8u1feg0VdIPR1hBkzsMs=', 'Content-Length': '0'} 2023-10-17T18:23:29.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000738c0bcd10870f05-00652ed121-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:23:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:29.957 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:29.958 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:29.959 DEBUG:boto:Method: HEAD 2023-10-17T18:23:29.959 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:23:29.959 DEBUG:boto:Data: 2023-10-17T18:23:29.960 DEBUG:boto:Headers: {} 2023-10-17T18:23:29.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:29.961 DEBUG:boto:Port: 8000 2023-10-17T18:23:29.961 DEBUG:boto:Params: {} 2023-10-17T18:23:29.961 DEBUG:boto:Token: None 2023-10-17T18:23:29.962 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:29 GMT /hbmuog-14/ 2023-10-17T18:23:29.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XYi7Z7J8u1feg0VdIPR1hBkzsMs= 2023-10-17T18:23:29.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XYi7Z7J8u1feg0VdIPR1hBkzsMs=', 'Content-Length': '0'} 2023-10-17T18:23:29.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009462bcadeb19f80b-00652ed121-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:23:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:29.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:23:29.966 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:29.967 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:29.967 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.968 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.969 DEBUG:boto:Method: GET 2023-10-17T18:23:29.969 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.970 DEBUG:boto:Data: 2023-10-17T18:23:29.970 DEBUG:boto:Headers: {} 2023-10-17T18:23:29.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:29.971 DEBUG:boto:Port: 8001 2023-10-17T18:23:29.972 DEBUG:boto:Params: {} 2023-10-17T18:23:29.972 DEBUG:boto:Token: None 2023-10-17T18:23:29.973 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:29 GMT /hbmuog-14/?versions 2023-10-17T18:23:29.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s= 2023-10-17T18:23:29.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s=', 'Content-Length': '0'} 2023-10-17T18:23:29.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10ebf211a14db86-00652ed121-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:29.977 DEBUG:boto:b'hbmuog-14999falseobj.o-ObChRTfMWwza2w93bufGTDeHYNFbtrue2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:29.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:23:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:23:29.978 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:23:29.979 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:29.979 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:29.980 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.980 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.980 DEBUG:boto:Method: GET 2023-10-17T18:23:29.981 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.981 DEBUG:boto:Data: 2023-10-17T18:23:29.982 DEBUG:boto:Headers: {} 2023-10-17T18:23:29.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:29.983 DEBUG:boto:Port: 8000 2023-10-17T18:23:29.983 DEBUG:boto:Params: {} 2023-10-17T18:23:29.983 DEBUG:boto:Token: None 2023-10-17T18:23:29.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:29 GMT /hbmuog-14/?versions 2023-10-17T18:23:29.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s= 2023-10-17T18:23:29.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s=', 'Content-Length': '0'} 2023-10-17T18:23:29.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f99e718a3bf5cc3-00652ed121-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:29.987 DEBUG:boto:b'hbmuog-14999falseobj.o-ObChRTfMWwza2w93bufGTDeHYNFbtrue2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:29.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:23:29.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:23:29.989 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:29.989 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:29.990 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.990 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.991 DEBUG:boto:Method: GET 2023-10-17T18:23:29.991 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:29.992 DEBUG:boto:Data: 2023-10-17T18:23:29.992 DEBUG:boto:Headers: {} 2023-10-17T18:23:29.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:29.993 DEBUG:boto:Port: 8001 2023-10-17T18:23:29.993 DEBUG:boto:Params: {} 2023-10-17T18:23:29.994 DEBUG:boto:Token: None 2023-10-17T18:23:29.994 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:29 GMT /hbmuog-14/?versions 2023-10-17T18:23:29.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s= 2023-10-17T18:23:29.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UzFPO+kun+qcTiEP29ouW+ElE+s=', 'Content-Length': '0'} 2023-10-17T18:23:29.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ae391f1f980bf5-00652ed121-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:29.998 DEBUG:boto:b'hbmuog-14999falseobj.o-ObChRTfMWwza2w93bufGTDeHYNFbtrue2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:29.999 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.000 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.000 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:30.001 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:30.001 DEBUG:boto:Method: GET 2023-10-17T18:23:30.001 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:23:30.002 DEBUG:boto:Data: 2023-10-17T18:23:30.002 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.003 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.003 DEBUG:boto:Params: {} 2023-10-17T18:23:30.004 DEBUG:boto:Token: None 2023-10-17T18:23:30.004 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/?versions 2023-10-17T18:23:30.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iu6rV7soVC4H+YZY4jwTRfE6qJg= 2023-10-17T18:23:30.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iu6rV7soVC4H+YZY4jwTRfE6qJg=', 'Content-Length': '0'} 2023-10-17T18:23:30.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004ffac1546f6c3d8-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.008 DEBUG:boto:b'hbmuog-14999falseobj.o-ObChRTfMWwza2w93bufGTDeHYNFbtrue2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-17T18:22:18.890Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.010 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.010 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.011 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.011 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.011 DEBUG:boto:Method: GET 2023-10-17T18:23:30.012 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.012 DEBUG:boto:Data: 2023-10-17T18:23:30.013 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.013 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.013 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.014 DEBUG:boto:Params: {} 2023-10-17T18:23:30.014 DEBUG:boto:Token: None 2023-10-17T18:23:30.015 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.015 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000217ed6759669891-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.019 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.019 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.019 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.020 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.020 DEBUG:boto:Method: GET 2023-10-17T18:23:30.021 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.021 DEBUG:boto:Data: 2023-10-17T18:23:30.021 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.022 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.023 DEBUG:boto:Params: {} 2023-10-17T18:23:30.023 DEBUG:boto:Token: None 2023-10-17T18:23:30.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009354ee0f8c6121a5-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.027 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.028 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.028 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.028 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.029 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.029 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.030 DEBUG:boto:Data: 2023-10-17T18:23:30.030 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.031 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.031 DEBUG:boto:Params: {} 2023-10-17T18:23:30.032 DEBUG:boto:Token: None 2023-10-17T18:23:30.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TwmftJwuHe3o3WtngZ60jpubJh8= 2023-10-17T18:23:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TwmftJwuHe3o3WtngZ60jpubJh8=', 'Content-Length': '0'} 2023-10-17T18:23:30.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034a06d714e617096-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.035 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.036 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.036 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.037 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.037 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.037 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.038 DEBUG:boto:Data: 2023-10-17T18:23:30.038 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.039 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.039 DEBUG:boto:Params: {} 2023-10-17T18:23:30.040 DEBUG:boto:Token: None 2023-10-17T18:23:30.040 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TwmftJwuHe3o3WtngZ60jpubJh8= 2023-10-17T18:23:30.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TwmftJwuHe3o3WtngZ60jpubJh8=', 'Content-Length': '0'} 2023-10-17T18:23:30.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c70b45464ddc9ef-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.044 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.045 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.045 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.045 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.046 DEBUG:boto:Method: GET 2023-10-17T18:23:30.046 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.047 DEBUG:boto:Data: 2023-10-17T18:23:30.047 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.048 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.048 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.049 DEBUG:boto:Params: {} 2023-10-17T18:23:30.049 DEBUG:boto:Token: None 2023-10-17T18:23:30.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c84814a9c76c136b-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.053 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.053 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.054 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.054 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.055 DEBUG:boto:Method: GET 2023-10-17T18:23:30.055 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.056 DEBUG:boto:Data: 2023-10-17T18:23:30.056 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.057 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.057 DEBUG:boto:Params: {} 2023-10-17T18:23:30.058 DEBUG:boto:Token: None 2023-10-17T18:23:30.058 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b68bcaedcd433b3b-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.062 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.062 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.063 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.063 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.064 DEBUG:boto:Method: GET 2023-10-17T18:23:30.064 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.064 DEBUG:boto:Data: 2023-10-17T18:23:30.065 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.066 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.066 DEBUG:boto:Params: {} 2023-10-17T18:23:30.067 DEBUG:boto:Token: None 2023-10-17T18:23:30.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005babb93fdf70b120-00652ed122-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.071 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.072 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.072 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.073 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.073 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.074 DEBUG:boto:Method: GET 2023-10-17T18:23:30.074 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.074 DEBUG:boto:Data: 2023-10-17T18:23:30.075 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.075 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.076 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.076 DEBUG:boto:Params: {} 2023-10-17T18:23:30.077 DEBUG:boto:Token: None 2023-10-17T18:23:30.077 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.077 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af25ad42e8901f36-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.122 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.123 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.124 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.124 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.124 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:23:30.125 DEBUG:boto:Data: 2023-10-17T18:23:30.125 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.126 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.126 DEBUG:boto:Params: {} 2023-10-17T18:23:30.127 DEBUG:boto:Token: None 2023-10-17T18:23:30.127 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj 2023-10-17T18:23:30.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I= 2023-10-17T18:23:30.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I=', 'Content-Length': '0'} 2023-10-17T18:23:30.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7b2fc9d9104573b-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.131 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.132 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.132 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.133 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:23:30.133 DEBUG:boto:Data: 2023-10-17T18:23:30.133 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.134 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.135 DEBUG:boto:Params: {} 2023-10-17T18:23:30.135 DEBUG:boto:Token: None 2023-10-17T18:23:30.135 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj 2023-10-17T18:23:30.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I= 2023-10-17T18:23:30.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I=', 'Content-Length': '0'} 2023-10-17T18:23:30.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8adf4b046828bad-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.140 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.140 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.140 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.141 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.141 DEBUG:boto:Method: GET 2023-10-17T18:23:30.142 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.142 DEBUG:boto:Data: 2023-10-17T18:23:30.143 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.143 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.144 DEBUG:boto:Params: {} 2023-10-17T18:23:30.144 DEBUG:boto:Token: None 2023-10-17T18:23:30.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f46e8c6339ef9bd-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.148 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.149 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.149 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.150 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.150 DEBUG:boto:Method: GET 2023-10-17T18:23:30.151 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.151 DEBUG:boto:Data: 2023-10-17T18:23:30.151 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.152 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.153 DEBUG:boto:Params: {} 2023-10-17T18:23:30.153 DEBUG:boto:Token: None 2023-10-17T18:23:30.154 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d659c26fe498543-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.157 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.157 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.158 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.158 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.159 DEBUG:boto:Method: GET 2023-10-17T18:23:30.159 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.160 DEBUG:boto:Data: 2023-10-17T18:23:30.160 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.161 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.161 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.161 DEBUG:boto:Params: {} 2023-10-17T18:23:30.162 DEBUG:boto:Token: None 2023-10-17T18:23:30.162 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.163 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf74e8cdcaf01ad5-00652ed122-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.167 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.168 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.168 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.169 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.169 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.170 DEBUG:boto:Method: GET 2023-10-17T18:23:30.170 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.171 DEBUG:boto:Data: 2023-10-17T18:23:30.171 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.172 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.172 DEBUG:boto:Params: {} 2023-10-17T18:23:30.173 DEBUG:boto:Token: None 2023-10-17T18:23:30.173 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.174 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a69070480f50a31-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.177 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.179 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.179 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.179 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.180 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.180 DEBUG:boto:Method: GET 2023-10-17T18:23:30.181 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.181 DEBUG:boto:Data: 2023-10-17T18:23:30.181 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.182 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.182 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.183 DEBUG:boto:Params: {} 2023-10-17T18:23:30.183 DEBUG:boto:Token: None 2023-10-17T18:23:30.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.184 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE= 2023-10-17T18:23:30.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE=', 'Content-Length': '0'} 2023-10-17T18:23:30.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000417abf1cbd481d7f-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.187 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.188 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.188 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.188 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.189 DEBUG:boto:Method: GET 2023-10-17T18:23:30.189 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.190 DEBUG:boto:Data: 2023-10-17T18:23:30.190 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.191 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.191 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.191 DEBUG:boto:Params: {} 2023-10-17T18:23:30.192 DEBUG:boto:Token: None 2023-10-17T18:23:30.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE= 2023-10-17T18:23:30.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE=', 'Content-Length': '0'} 2023-10-17T18:23:30.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000589420446755c2c1-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.196 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.196 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.197 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.197 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.197 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.198 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.199 DEBUG:boto:Data: 2023-10-17T18:23:30.199 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.200 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.200 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.201 DEBUG:boto:Params: {} 2023-10-17T18:23:30.201 DEBUG:boto:Token: None 2023-10-17T18:23:30.201 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.202 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vR5ojcOgGe1b2sQuzmU2Wc4ucZw= 2023-10-17T18:23:30.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vR5ojcOgGe1b2sQuzmU2Wc4ucZw=', 'Content-Length': '0'} 2023-10-17T18:23:30.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000581d7ac00e571c7a-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.205 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.206 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.206 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.206 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.207 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.207 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.208 DEBUG:boto:Data: 2023-10-17T18:23:30.208 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.209 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.209 DEBUG:boto:Params: {} 2023-10-17T18:23:30.210 DEBUG:boto:Token: None 2023-10-17T18:23:30.210 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vR5ojcOgGe1b2sQuzmU2Wc4ucZw= 2023-10-17T18:23:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vR5ojcOgGe1b2sQuzmU2Wc4ucZw=', 'Content-Length': '0'} 2023-10-17T18:23:30.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0c65e8cfbbbec36-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.214 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.214 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.214 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.215 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.215 DEBUG:boto:Method: GET 2023-10-17T18:23:30.216 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.216 DEBUG:boto:Data: 2023-10-17T18:23:30.216 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.217 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.218 DEBUG:boto:Params: {} 2023-10-17T18:23:30.218 DEBUG:boto:Token: None 2023-10-17T18:23:30.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE= 2023-10-17T18:23:30.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE=', 'Content-Length': '0'} 2023-10-17T18:23:30.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001278506067e8f1fd-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.222 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.222 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.223 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.223 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.224 DEBUG:boto:Method: GET 2023-10-17T18:23:30.224 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.224 DEBUG:boto:Data: 2023-10-17T18:23:30.225 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.226 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.226 DEBUG:boto:Params: {} 2023-10-17T18:23:30.226 DEBUG:boto:Token: None 2023-10-17T18:23:30.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:23:30.227 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE= 2023-10-17T18:23:30.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TeoW1GQYXVytHxGy/yUgKXvysBE=', 'Content-Length': '0'} 2023-10-17T18:23:30.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b0f4c17665dcbc3-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.230 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.231 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.231 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.232 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.232 DEBUG:boto:Method: GET 2023-10-17T18:23:30.232 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.233 DEBUG:boto:Data: 2023-10-17T18:23:30.233 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.234 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.234 DEBUG:boto:Params: {} 2023-10-17T18:23:30.235 DEBUG:boto:Token: None 2023-10-17T18:23:30.235 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee0923f97c2d0e3-00652ed122-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.282 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.283 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.283 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.284 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.284 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.285 DEBUG:boto:Method: GET 2023-10-17T18:23:30.285 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.285 DEBUG:boto:Data: 2023-10-17T18:23:30.286 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.287 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.287 DEBUG:boto:Params: {} 2023-10-17T18:23:30.287 DEBUG:boto:Token: None 2023-10-17T18:23:30.288 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3574e59a457cb3a-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.292 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.293 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.293 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.294 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.294 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:23:30.294 DEBUG:boto:Data: 2023-10-17T18:23:30.295 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.295 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.296 DEBUG:boto:Params: {} 2023-10-17T18:23:30.296 DEBUG:boto:Token: None 2023-10-17T18:23:30.297 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj 2023-10-17T18:23:30.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I= 2023-10-17T18:23:30.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I=', 'Content-Length': '0'} 2023-10-17T18:23:30.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002456ce856cc66a03-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.300 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.301 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.301 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.302 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:23:30.302 DEBUG:boto:Data: 2023-10-17T18:23:30.302 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.303 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.303 DEBUG:boto:Params: {} 2023-10-17T18:23:30.304 DEBUG:boto:Token: None 2023-10-17T18:23:30.304 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj 2023-10-17T18:23:30.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I= 2023-10-17T18:23:30.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eGZWaRRxkog1VIlmfZk2I+lkU6I=', 'Content-Length': '0'} 2023-10-17T18:23:30.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3cf5fded8043d36-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:23:30.309 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.309 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.309 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.310 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.310 DEBUG:boto:Method: GET 2023-10-17T18:23:30.311 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.311 DEBUG:boto:Data: 2023-10-17T18:23:30.311 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.312 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.312 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.313 DEBUG:boto:Params: {} 2023-10-17T18:23:30.313 DEBUG:boto:Token: None 2023-10-17T18:23:30.314 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.314 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000647289552563e0d8-00652ed122-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.317 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.317 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.318 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.318 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.319 DEBUG:boto:Method: GET 2023-10-17T18:23:30.319 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.320 DEBUG:boto:Data: 2023-10-17T18:23:30.320 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.320 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.321 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.321 DEBUG:boto:Params: {} 2023-10-17T18:23:30.322 DEBUG:boto:Token: None 2023-10-17T18:23:30.322 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:23:30.322 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE= 2023-10-17T18:23:30.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RfbeCksgZwRgRTknGYN8Y2g4jlE=', 'Content-Length': '0'} 2023-10-17T18:23:30.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000645f31b4b9f56adf-00652ed122-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.325 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.326 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.326 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.327 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.327 DEBUG:boto:Method: GET 2023-10-17T18:23:30.327 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.328 DEBUG:boto:Data: 2023-10-17T18:23:30.328 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.329 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:23:30.329 DEBUG:boto:Port: 8001 2023-10-17T18:23:30.329 DEBUG:boto:Params: {} 2023-10-17T18:23:30.330 DEBUG:boto:Token: None 2023-10-17T18:23:30.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.331 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a6de5ddc09cbc9-00652ed122-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.334 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.335 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.335 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.336 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.336 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:23:30.337 DEBUG:boto:Method: GET 2023-10-17T18:23:30.337 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:23:30.337 DEBUG:boto:Data: 2023-10-17T18:23:30.338 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.339 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.339 DEBUG:boto:Params: {} 2023-10-17T18:23:30.340 DEBUG:boto:Token: None 2023-10-17T18:23:30.340 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o= 2023-10-17T18:23:30.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l+QU2n4yoN6BHYm7C+0V9k4HU5o=', 'Content-Length': '0'} 2023-10-17T18:23:30.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005275686eea625843-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.344 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:23:30.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:23:30.345 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.345 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.346 DEBUG:boto:path=/hbmuog-14/?versioning 2023-10-17T18:23:30.346 DEBUG:boto:auth_path=/hbmuog-14/?versioning 2023-10-17T18:23:30.346 DEBUG:boto:Method: PUT 2023-10-17T18:23:30.347 DEBUG:boto:Path: /hbmuog-14/?versioning 2023-10-17T18:23:30.347 DEBUG:boto:Data: Suspended Disabled 2023-10-17T18:23:30.347 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.348 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.348 DEBUG:boto:Params: {} 2023-10-17T18:23:30.349 DEBUG:boto:Token: None 2023-10-17T18:23:30.349 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/?versioning 2023-10-17T18:23:30.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DHxkL/+9bxQCMgga72p8KsbeJwA= 2023-10-17T18:23:30.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DHxkL/+9bxQCMgga72p8KsbeJwA=', 'Content-Length': '230'} 2023-10-17T18:23:30.357 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd6381fbaa4a564c-00652ed122-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:23:30.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:23:30.358 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:23:30.415 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:23:30.415 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:23:30.450 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:23:30.463 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:23:30.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.463 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567010.356509_188.1", 2023-10-17T18:23:30.463 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:23:30.356509Z" 2023-10-17T18:23:30.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.465 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:23:30.465 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:23:30.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.466 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:23:30.466 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:23:30.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.467 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:23:30.467 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:23:30.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.467 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:23:30.468 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567010.356509_188.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:23:30.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:23:30.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:23:30.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:23:30.470 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:23:30.521 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:23:30.522 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:23:30.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:30.548+0000 7f2c001a4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:23:30.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:30.549+0000 7f2c001a4780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:23:30.562 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:23:30.562 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:23:30.562 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:23:30.563 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:23:30.563 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:23:30.563 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:23:30.564 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:23:30.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.564 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:23:30.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.564 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:23:30.565 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:23:30.565 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:23:30.565 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567010.356509_188.1", 2023-10-17T18:23:30.565 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:23:30.566 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:23:30.566 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:23:30.566 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:23:30.356509Z", 2023-10-17T18:23:30.566 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:23:30.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.567 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:23:30.567 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:23:30.568 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:23:30.568 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:23:30.568 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:23:30.568 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:23:30.569 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:23:30.569 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:23:30.569 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:23:30.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.570 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:23:30.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:23:30.570 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:23:30.571 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:23:30.571 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:23:30.571 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:23:30.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:23:30.572 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:23:30.572 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:23:30.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:23:30.573 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:23:30.573 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:23:30.573 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:23:30.573 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:23:30.573 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:23:30.574 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:23:30.574 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:23:30.574 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:23:30.574 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:23:30.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:23:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:23:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:23:30.576 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:23:30.576 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:23:30.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:23:30.576 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:23:30Z" 2023-10-17T18:23:30.576 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:23:30.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567010.356509_188.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:23:30.356509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:23:30Z" } 2023-10-17T18:23:30.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567010.356509_188.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:23:30.356509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:23:30.578 DEBUG:rgw_multi.tests:log_status={0: '1_1697567010.356509_188.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:23:30.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567010.356509_188.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:23:30.579 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:23:30.580 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:23:30.580 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:23:30.580 DEBUG:boto:Method: HEAD 2023-10-17T18:23:30.581 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:23:30.581 DEBUG:boto:Data: 2023-10-17T18:23:30.582 DEBUG:boto:Headers: {} 2023-10-17T18:23:30.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.582 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.583 DEBUG:boto:Params: {} 2023-10-17T18:23:30.583 DEBUG:boto:Token: None 2023-10-17T18:23:30.584 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/ 2023-10-17T18:23:30.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xSCW2JSSPADCrFKecIM22bkmycg= 2023-10-17T18:23:30.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xSCW2JSSPADCrFKecIM22bkmycg=', 'Content-Length': '0'} 2023-10-17T18:23:30.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc6c415cdcc04a58-00652ed122-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.587 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:23:30.588 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:23:30.588 DEBUG:boto:Method: PUT 2023-10-17T18:23:30.589 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:23:30.589 DEBUG:boto:Data: 2023-10-17T18:23:30.590 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:23:30.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:23:30.590 DEBUG:boto:Port: 8000 2023-10-17T18:23:30.591 DEBUG:boto:Params: {} 2023-10-17T18:23:30.591 DEBUG:boto:Token: None 2023-10-17T18:23:30.592 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:23:30 GMT /hbmuog-14/obj 2023-10-17T18:23:30.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KAgsMsb36umAf4mscAon4tAtanQ= 2023-10-17T18:23:30.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:23:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KAgsMsb36umAf4mscAon4tAtanQ='} 2023-10-17T18:23:30.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bc2cef73638d82b-00652ed122-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:23:30.603 DEBUG:rgw_multi.tests:created null version id=None 2023-10-17T18:23:30.604 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-14 2023-10-17T18:23:30.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:30.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:30.605 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:23:30.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:23:30.665 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:23:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:30.697+0000 7f47ab3e7780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:30.699 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:30.699 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:35.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:35.699+0000 7f47ab3e7780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:35.700 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:35.701 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:40.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:40.701+0000 7f47ab3e7780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:40.702 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:40.702 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:45.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:45.702+0000 7f47ab3e7780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:45.703 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-10-17T18:23:45.703 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:50.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:50.704+0000 7f47ab3e7780 1 bucket sync caught up with source: 2023-10-17T18:23:50.705 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:50.705 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-10-17T18:23:50.705 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:50.704+0000 7f47ab3e7780 0 bucket checkpoint complete 2023-10-17T18:23:50.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-14 2023-10-17T18:23:50.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:50.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:23:50.722 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:23:50.786 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:23:50.786 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:23:50.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:50.823+0000 7fd8cd7d5780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:50.824 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:50.824 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:23:55.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:23:55.824+0000 7fd8cd7d5780 1 waiting for incremental sync to catch up: 2023-10-17T18:23:55.825 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:23:55.826 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:24:00.826 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:00.825+0000 7fd8cd7d5780 1 waiting for incremental sync to catch up: 2023-10-17T18:24:00.827 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:24:00.827 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:24:05.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:05.826+0000 7fd8cd7d5780 1 waiting for incremental sync to catch up: 2023-10-17T18:24:05.828 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.23.13, , ] 2023-10-17T18:24:05.828 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:24:10.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:10.828+0000 7fd8cd7d5780 1 bucket sync caught up with source: 2023-10-17T18:24:10.829 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:24:10.830 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000005.28.18, , ] 2023-10-17T18:24:10.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:10.828+0000 7fd8cd7d5780 0 bucket checkpoint complete 2023-10-17T18:24:10.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:24:10.846 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.846 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.847 DEBUG:boto:Method: HEAD 2023-10-17T18:24:10.847 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:24:10.848 DEBUG:boto:Data: 2023-10-17T18:24:10.848 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.849 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.849 DEBUG:boto:Params: {} 2023-10-17T18:24:10.850 DEBUG:boto:Token: None 2023-10-17T18:24:10.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/ 2023-10-17T18:24:10.851 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:95PluhFJRG/pVoL1TpOQhJau++E= 2023-10-17T18:24:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:95PluhFJRG/pVoL1TpOQhJau++E=', 'Content-Length': '0'} 2023-10-17T18:24:10.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b072f797d5f42fe-00652ed14a-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.855 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.856 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.856 DEBUG:boto:Method: HEAD 2023-10-17T18:24:10.857 DEBUG:boto:Path: /hbmuog-14/ 2023-10-17T18:24:10.857 DEBUG:boto:Data: 2023-10-17T18:24:10.858 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.859 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.859 DEBUG:boto:Params: {} 2023-10-17T18:24:10.860 DEBUG:boto:Token: None 2023-10-17T18:24:10.860 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/ 2023-10-17T18:24:10.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:95PluhFJRG/pVoL1TpOQhJau++E= 2023-10-17T18:24:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:95PluhFJRG/pVoL1TpOQhJau++E=', 'Content-Length': '0'} 2023-10-17T18:24:10.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be75271d80e43793-00652ed14a-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.864 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:24:10.865 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.865 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.866 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.866 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.867 DEBUG:boto:Method: GET 2023-10-17T18:24:10.867 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.868 DEBUG:boto:Data: 2023-10-17T18:24:10.868 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.869 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.869 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.869 DEBUG:boto:Params: {} 2023-10-17T18:24:10.870 DEBUG:boto:Token: None 2023-10-17T18:24:10.870 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/?versions 2023-10-17T18:24:10.871 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4= 2023-10-17T18:24:10.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4=', 'Content-Length': '0'} 2023-10-17T18:24:10.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f20abd7b77c831-00652ed14a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.874 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.o-ObChRTfMWwza2w93bufGTDeHYNFbfalse2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:10.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:10.876 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:24:10.876 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.877 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.877 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.877 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.878 DEBUG:boto:Method: GET 2023-10-17T18:24:10.878 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.879 DEBUG:boto:Data: 2023-10-17T18:24:10.879 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.880 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.880 DEBUG:boto:Params: {} 2023-10-17T18:24:10.881 DEBUG:boto:Token: None 2023-10-17T18:24:10.881 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/?versions 2023-10-17T18:24:10.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4= 2023-10-17T18:24:10.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4=', 'Content-Length': '0'} 2023-10-17T18:24:10.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b9ae4defde505e5-00652ed14a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.885 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.o-ObChRTfMWwza2w93bufGTDeHYNFbfalse2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:10.887 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.887 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.887 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.888 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.888 DEBUG:boto:Method: GET 2023-10-17T18:24:10.889 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.889 DEBUG:boto:Data: 2023-10-17T18:24:10.889 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.890 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.891 DEBUG:boto:Params: {} 2023-10-17T18:24:10.891 DEBUG:boto:Token: None 2023-10-17T18:24:10.892 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/?versions 2023-10-17T18:24:10.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4= 2023-10-17T18:24:10.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4=', 'Content-Length': '0'} 2023-10-17T18:24:10.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ce25d5aaf68b99-00652ed14a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.896 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.o-ObChRTfMWwza2w93bufGTDeHYNFbfalse2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:10.897 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.897 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.898 DEBUG:boto:path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.898 DEBUG:boto:auth_path=/hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.899 DEBUG:boto:Method: GET 2023-10-17T18:24:10.899 DEBUG:boto:Path: /hbmuog-14/?versions&max-keys=999 2023-10-17T18:24:10.900 DEBUG:boto:Data: 2023-10-17T18:24:10.900 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.901 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.901 DEBUG:boto:Params: {} 2023-10-17T18:24:10.902 DEBUG:boto:Token: None 2023-10-17T18:24:10.903 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/?versions 2023-10-17T18:24:10.903 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4= 2023-10-17T18:24:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ACTj64OCockhKSkgt3ndVnWslg4=', 'Content-Length': '0'} 2023-10-17T18:24:10.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bab815432c1a7cba-00652ed14a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.907 DEBUG:boto:b'hbmuog-14999falseobjnulltrue2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.o-ObChRTfMWwza2w93bufGTDeHYNFbfalse2023-10-17T18:22:54.696Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:10.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:10.908 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.908 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.909 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.909 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.910 DEBUG:boto:Method: GET 2023-10-17T18:24:10.910 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.911 DEBUG:boto:Data: 2023-10-17T18:24:10.911 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.912 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.912 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.913 DEBUG:boto:Params: {} 2023-10-17T18:24:10.913 DEBUG:boto:Token: None 2023-10-17T18:24:10.914 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.914 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI= 2023-10-17T18:24:10.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI=', 'Content-Length': '0'} 2023-10-17T18:24:10.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1da820739b6304a-00652ed14a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.917 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.918 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.918 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.919 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.919 DEBUG:boto:Method: GET 2023-10-17T18:24:10.920 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.920 DEBUG:boto:Data: 2023-10-17T18:24:10.920 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.921 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.922 DEBUG:boto:Params: {} 2023-10-17T18:24:10.922 DEBUG:boto:Token: None 2023-10-17T18:24:10.923 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.924 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI= 2023-10-17T18:24:10.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI=', 'Content-Length': '0'} 2023-10-17T18:24:10.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006006397cf138fe39-00652ed14a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.926 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.927 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.927 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.928 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.928 DEBUG:boto:Method: HEAD 2023-10-17T18:24:10.928 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.929 DEBUG:boto:Data: 2023-10-17T18:24:10.929 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.930 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.930 DEBUG:boto:Params: {} 2023-10-17T18:24:10.931 DEBUG:boto:Token: None 2023-10-17T18:24:10.931 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QFU1vJgESiBk58zAF65QdO5HHQ= 2023-10-17T18:24:10.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QFU1vJgESiBk58zAF65QdO5HHQ=', 'Content-Length': '0'} 2023-10-17T18:24:10.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c572956135932c0e-00652ed14a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.935 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.935 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.936 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.936 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.937 DEBUG:boto:Method: HEAD 2023-10-17T18:24:10.937 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.938 DEBUG:boto:Data: 2023-10-17T18:24:10.938 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.939 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.939 DEBUG:boto:Params: {} 2023-10-17T18:24:10.940 DEBUG:boto:Token: None 2023-10-17T18:24:10.940 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.940 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QFU1vJgESiBk58zAF65QdO5HHQ= 2023-10-17T18:24:10.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QFU1vJgESiBk58zAF65QdO5HHQ=', 'Content-Length': '0'} 2023-10-17T18:24:10.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000037c81813842b783-00652ed14a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:10.944 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.944 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.945 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.945 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.946 DEBUG:boto:Method: GET 2023-10-17T18:24:10.946 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.947 DEBUG:boto:Data: 2023-10-17T18:24:10.947 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.948 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.948 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.949 DEBUG:boto:Params: {} 2023-10-17T18:24:10.949 DEBUG:boto:Token: None 2023-10-17T18:24:10.950 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.950 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI= 2023-10-17T18:24:10.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI=', 'Content-Length': '0'} 2023-10-17T18:24:10.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000450cc446fe225ef2-00652ed14a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.953 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:10.954 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:10.954 DEBUG:boto:path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.955 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.955 DEBUG:boto:Method: GET 2023-10-17T18:24:10.955 DEBUG:boto:Path: /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.956 DEBUG:boto:Data: 2023-10-17T18:24:10.956 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.957 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:10.957 DEBUG:boto:Port: 8000 2023-10-17T18:24:10.958 DEBUG:boto:Params: {} 2023-10-17T18:24:10.958 DEBUG:boto:Token: None 2023-10-17T18:24:10.959 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/obj?versionId=null 2023-10-17T18:24:10.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI= 2023-10-17T18:24:10.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KRj45UkX1cm9/OsMLsz+ygNYYxI=', 'Content-Length': '0'} 2023-10-17T18:24:10.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bce74e2277db0c6d-00652ed14a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:10.963 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:10.963 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:10.963 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:10.964 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:10.964 DEBUG:boto:Method: GET 2023-10-17T18:24:10.965 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:10.965 DEBUG:boto:Data: 2023-10-17T18:24:10.966 DEBUG:boto:Headers: {} 2023-10-17T18:24:10.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:10.967 DEBUG:boto:Port: 8001 2023-10-17T18:24:10.967 DEBUG:boto:Params: {} 2023-10-17T18:24:10.967 DEBUG:boto:Token: None 2023-10-17T18:24:10.968 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:10 GMT /hbmuog-14/ 2023-10-17T18:24:10.968 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xhflUqq37W8rNE8Tv/B9XUd2Lao= 2023-10-17T18:24:10.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:10 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xhflUqq37W8rNE8Tv/B9XUd2Lao=', 'Content-Length': '0'} 2023-10-17T18:24:10.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a261c4fef8142276-00652ed14a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.014 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.015 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.016 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.016 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.016 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.017 DEBUG:boto:Method: GET 2023-10-17T18:24:11.017 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.018 DEBUG:boto:Data: 2023-10-17T18:24:11.018 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.019 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.019 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.020 DEBUG:boto:Params: {} 2023-10-17T18:24:11.020 DEBUG:boto:Token: None 2023-10-17T18:24:11.021 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272e151de8212028-00652ed14b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.025 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.025 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.026 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.026 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.027 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.027 DEBUG:boto:Data: 2023-10-17T18:24:11.028 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.028 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.028 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.029 DEBUG:boto:Params: {} 2023-10-17T18:24:11.029 DEBUG:boto:Token: None 2023-10-17T18:24:11.030 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.030 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w= 2023-10-17T18:24:11.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w=', 'Content-Length': '0'} 2023-10-17T18:24:11.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066b926bef05aa1d2-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.034 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.034 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.035 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.035 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.035 DEBUG:boto:Data: 2023-10-17T18:24:11.036 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.036 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.037 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.037 DEBUG:boto:Params: {} 2023-10-17T18:24:11.037 DEBUG:boto:Token: None 2023-10-17T18:24:11.038 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.038 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w= 2023-10-17T18:24:11.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w=', 'Content-Length': '0'} 2023-10-17T18:24:11.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffd97d91c023dac6-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:11.042 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.043 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.043 DEBUG:boto:Method: GET 2023-10-17T18:24:11.044 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.044 DEBUG:boto:Data: 2023-10-17T18:24:11.045 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.045 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.046 DEBUG:boto:Params: {} 2023-10-17T18:24:11.046 DEBUG:boto:Token: None 2023-10-17T18:24:11.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.047 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g= 2023-10-17T18:24:11.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g=', 'Content-Length': '0'} 2023-10-17T18:24:11.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e17db8a683fc33d-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.051 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.051 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.052 DEBUG:boto:Method: GET 2023-10-17T18:24:11.052 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.053 DEBUG:boto:Data: 2023-10-17T18:24:11.053 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.054 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.054 DEBUG:boto:Params: {} 2023-10-17T18:24:11.055 DEBUG:boto:Token: None 2023-10-17T18:24:11.055 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g= 2023-10-17T18:24:11.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g=', 'Content-Length': '0'} 2023-10-17T18:24:11.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bab13dc8a9dc83ca-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.059 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.060 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.060 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.060 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.061 DEBUG:boto:Method: GET 2023-10-17T18:24:11.061 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.062 DEBUG:boto:Data: 2023-10-17T18:24:11.062 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.063 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.063 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.064 DEBUG:boto:Params: {} 2023-10-17T18:24:11.064 DEBUG:boto:Token: None 2023-10-17T18:24:11.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1393fdf865777f-00652ed14b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.069 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.069 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.070 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.070 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.071 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.071 DEBUG:boto:Method: GET 2023-10-17T18:24:11.072 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.072 DEBUG:boto:Data: 2023-10-17T18:24:11.073 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.074 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.074 DEBUG:boto:Params: {} 2023-10-17T18:24:11.074 DEBUG:boto:Token: None 2023-10-17T18:24:11.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.075 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b64829c71e3fb0af-00652ed14b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.079 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:11.080 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.080 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.081 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.081 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.082 DEBUG:boto:Method: GET 2023-10-17T18:24:11.082 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.083 DEBUG:boto:Data: 2023-10-17T18:24:11.083 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.084 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.084 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.084 DEBUG:boto:Params: {} 2023-10-17T18:24:11.085 DEBUG:boto:Token: None 2023-10-17T18:24:11.085 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.086 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4= 2023-10-17T18:24:11.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4=', 'Content-Length': '0'} 2023-10-17T18:24:11.089 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5890953d96619cd-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.089 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.090 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.090 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.091 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.091 DEBUG:boto:Method: GET 2023-10-17T18:24:11.092 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.092 DEBUG:boto:Data: 2023-10-17T18:24:11.092 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.093 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.093 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.094 DEBUG:boto:Params: {} 2023-10-17T18:24:11.094 DEBUG:boto:Token: None 2023-10-17T18:24:11.095 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.095 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4= 2023-10-17T18:24:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4=', 'Content-Length': '0'} 2023-10-17T18:24:11.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011ecaad6c9d5bb2f-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.098 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.099 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.099 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.100 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.100 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.101 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.101 DEBUG:boto:Data: 2023-10-17T18:24:11.102 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.102 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.103 DEBUG:boto:Params: {} 2023-10-17T18:24:11.103 DEBUG:boto:Token: None 2023-10-17T18:24:11.104 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Sz+Lt1U+3w9BCTkJ6YRjAjnaNFc= 2023-10-17T18:24:11.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Sz+Lt1U+3w9BCTkJ6YRjAjnaNFc=', 'Content-Length': '0'} 2023-10-17T18:24:11.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4d5222df61cc5af-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.108 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.108 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.109 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.109 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.109 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.110 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.110 DEBUG:boto:Data: 2023-10-17T18:24:11.111 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.112 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.112 DEBUG:boto:Params: {} 2023-10-17T18:24:11.113 DEBUG:boto:Token: None 2023-10-17T18:24:11.113 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.114 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Sz+Lt1U+3w9BCTkJ6YRjAjnaNFc= 2023-10-17T18:24:11.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Sz+Lt1U+3w9BCTkJ6YRjAjnaNFc=', 'Content-Length': '0'} 2023-10-17T18:24:11.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b076d491d149e4f-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:11.117 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.117 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.118 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.118 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.119 DEBUG:boto:Method: GET 2023-10-17T18:24:11.119 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.120 DEBUG:boto:Data: 2023-10-17T18:24:11.120 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.121 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.121 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.122 DEBUG:boto:Params: {} 2023-10-17T18:24:11.122 DEBUG:boto:Token: None 2023-10-17T18:24:11.123 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.123 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4= 2023-10-17T18:24:11.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4=', 'Content-Length': '0'} 2023-10-17T18:24:11.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7a0941b20386087-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.126 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.127 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.127 DEBUG:boto:path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.128 DEBUG:boto:auth_path=/hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.128 DEBUG:boto:Method: GET 2023-10-17T18:24:11.129 DEBUG:boto:Path: /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.129 DEBUG:boto:Data: 2023-10-17T18:24:11.130 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.131 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.131 DEBUG:boto:Params: {} 2023-10-17T18:24:11.132 DEBUG:boto:Token: None 2023-10-17T18:24:11.132 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj?versionId=.o-ObChRTfMWwza2w93bufGTDeHYNFb 2023-10-17T18:24:11.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4= 2023-10-17T18:24:11.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmYi1h+qkYTCK9AkP4JvDtOU9O4=', 'Content-Length': '0'} 2023-10-17T18:24:11.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:22:54 GMT'), ('x-amz-version-id', '.o-ObChRTfMWwza2w93bufGTDeHYNFb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f8770a38bcaedcf-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.136 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.137 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.137 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.137 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.138 DEBUG:boto:Method: GET 2023-10-17T18:24:11.138 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.139 DEBUG:boto:Data: 2023-10-17T18:24:11.139 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.140 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.140 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.141 DEBUG:boto:Params: {} 2023-10-17T18:24:11.141 DEBUG:boto:Token: None 2023-10-17T18:24:11.142 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.142 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf2dd71f5858b29b-00652ed14b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.146 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.147 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.147 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.148 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.148 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.148 DEBUG:boto:Method: GET 2023-10-17T18:24:11.149 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.149 DEBUG:boto:Data: 2023-10-17T18:24:11.150 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.150 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.150 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.151 DEBUG:boto:Params: {} 2023-10-17T18:24:11.151 DEBUG:boto:Token: None 2023-10-17T18:24:11.152 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.152 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d48caa18ebb760bc-00652ed14b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.198 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.199 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.200 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.200 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.201 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.201 DEBUG:boto:Data: 2023-10-17T18:24:11.201 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.202 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.203 DEBUG:boto:Params: {} 2023-10-17T18:24:11.203 DEBUG:boto:Token: None 2023-10-17T18:24:11.204 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w= 2023-10-17T18:24:11.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w=', 'Content-Length': '0'} 2023-10-17T18:24:11.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d576de0c07ffbeea-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.207 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.208 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.208 DEBUG:boto:Method: HEAD 2023-10-17T18:24:11.208 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.209 DEBUG:boto:Data: 2023-10-17T18:24:11.209 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.210 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.210 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.210 DEBUG:boto:Params: {} 2023-10-17T18:24:11.211 DEBUG:boto:Token: None 2023-10-17T18:24:11.211 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w= 2023-10-17T18:24:11.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SDwQwy1w0KA7TFsors9Vnzcsp0w=', 'Content-Length': '0'} 2023-10-17T18:24:11.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7d6c470ca9e098c-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:11.215 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.216 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.216 DEBUG:boto:Method: GET 2023-10-17T18:24:11.216 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.217 DEBUG:boto:Data: 2023-10-17T18:24:11.217 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.218 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.219 DEBUG:boto:Params: {} 2023-10-17T18:24:11.219 DEBUG:boto:Token: None 2023-10-17T18:24:11.220 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g= 2023-10-17T18:24:11.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g=', 'Content-Length': '0'} 2023-10-17T18:24:11.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ce3a978b988fbcb-00652ed14b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.223 DEBUG:boto:path=/hbmuog-14/obj 2023-10-17T18:24:11.224 DEBUG:boto:auth_path=/hbmuog-14/obj 2023-10-17T18:24:11.224 DEBUG:boto:Method: GET 2023-10-17T18:24:11.225 DEBUG:boto:Path: /hbmuog-14/obj 2023-10-17T18:24:11.225 DEBUG:boto:Data: 2023-10-17T18:24:11.225 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.226 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.227 DEBUG:boto:Params: {} 2023-10-17T18:24:11.227 DEBUG:boto:Token: None 2023-10-17T18:24:11.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/obj 2023-10-17T18:24:11.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g= 2023-10-17T18:24:11.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pTMuhv7dSefFaY3KGH0rOYOWj/g=', 'Content-Length': '0'} 2023-10-17T18:24:11.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:23:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dffaa8e082e36a46-00652ed14b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.231 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.232 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.232 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.232 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.233 DEBUG:boto:Method: GET 2023-10-17T18:24:11.233 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.234 DEBUG:boto:Data: 2023-10-17T18:24:11.234 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.235 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.235 DEBUG:boto:Params: {} 2023-10-17T18:24:11.236 DEBUG:boto:Token: None 2023-10-17T18:24:11.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f81846982477ae-00652ed14b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.282 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.283 DEBUG:boto:path=/hbmuog-14/ 2023-10-17T18:24:11.283 DEBUG:boto:auth_path=/hbmuog-14/ 2023-10-17T18:24:11.284 DEBUG:boto:path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.284 DEBUG:boto:auth_path=/hbmuog-14/?prefix=obj 2023-10-17T18:24:11.285 DEBUG:boto:Method: GET 2023-10-17T18:24:11.285 DEBUG:boto:Path: /hbmuog-14/?prefix=obj 2023-10-17T18:24:11.286 DEBUG:boto:Data: 2023-10-17T18:24:11.286 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.287 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.288 DEBUG:boto:Params: {} 2023-10-17T18:24:11.288 DEBUG:boto:Token: None 2023-10-17T18:24:11.288 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-14/ 2023-10-17T18:24:11.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM= 2023-10-17T18:24:11.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0BKRACHf9jNWz09apOUrKmqZCFM=', 'Content-Length': '0'} 2023-10-17T18:24:11.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fbb39e1fabdc386-00652ed14b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.293 DEBUG:boto:b'hbmuog-14obj1000falseobj2023-10-17T18:23:30.594Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:11.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-14 zones={a2, a1} 2023-10-17T18:24:11.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-10-17T18:24:11.295 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-15 2023-10-17T18:24:11.296 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:11.296 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:11.296 DEBUG:boto:Method: PUT 2023-10-17T18:24:11.297 DEBUG:boto:Path: /hbmuog-15/ 2023-10-17T18:24:11.297 DEBUG:boto:Data: 2023-10-17T18:24:11.298 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.299 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.299 DEBUG:boto:Params: {} 2023-10-17T18:24:11.299 DEBUG:boto:Token: None 2023-10-17T18:24:11.300 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-15/ 2023-10-17T18:24:11.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SRiNLtJC38mdDtA2wy7hQZXFC9s= 2023-10-17T18:24:11.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SRiNLtJC38mdDtA2wy7hQZXFC9s=', 'Content-Length': '0'} 2023-10-17T18:24:11.317 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc33b1a7dade2764-00652ed14b-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.317 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-16 2023-10-17T18:24:11.318 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:11.318 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:11.319 DEBUG:boto:Method: PUT 2023-10-17T18:24:11.319 DEBUG:boto:Path: /hbmuog-16/ 2023-10-17T18:24:11.319 DEBUG:boto:Data: 2023-10-17T18:24:11.320 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.320 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.321 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.321 DEBUG:boto:Params: {} 2023-10-17T18:24:11.322 DEBUG:boto:Token: None 2023-10-17T18:24:11.322 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-16/ 2023-10-17T18:24:11.323 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NMOT7FoV2wRqjYAbndWwTtDcyUE= 2023-10-17T18:24:11.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NMOT7FoV2wRqjYAbndWwTtDcyUE=', 'Content-Length': '0'} 2023-10-17T18:24:11.349 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d909879e9d2b0013-00652ed14b-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.349 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:11.349 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:11.350 DEBUG:boto:path=/hbmuog-15/?versioning 2023-10-17T18:24:11.350 DEBUG:boto:auth_path=/hbmuog-15/?versioning 2023-10-17T18:24:11.351 DEBUG:boto:Method: PUT 2023-10-17T18:24:11.351 DEBUG:boto:Path: /hbmuog-15/?versioning 2023-10-17T18:24:11.351 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:24:11.352 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:11.352 DEBUG:boto:Port: 8000 2023-10-17T18:24:11.353 DEBUG:boto:Params: {} 2023-10-17T18:24:11.353 DEBUG:boto:Token: None 2023-10-17T18:24:11.353 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-15/?versioning 2023-10-17T18:24:11.354 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fellJfVA8vKJlSoBegQbqgxi95Q= 2023-10-17T18:24:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fellJfVA8vKJlSoBegQbqgxi95Q=', 'Content-Length': '228'} 2023-10-17T18:24:11.360 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5d10238029013df-00652ed14b-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.360 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:11.361 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:11.361 DEBUG:boto:path=/hbmuog-16/?versioning 2023-10-17T18:24:11.361 DEBUG:boto:auth_path=/hbmuog-16/?versioning 2023-10-17T18:24:11.362 DEBUG:boto:Method: PUT 2023-10-17T18:24:11.362 DEBUG:boto:Path: /hbmuog-16/?versioning 2023-10-17T18:24:11.363 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:24:11.363 DEBUG:boto:Headers: {} 2023-10-17T18:24:11.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:11.364 DEBUG:boto:Port: 8001 2023-10-17T18:24:11.364 DEBUG:boto:Params: {} 2023-10-17T18:24:11.365 DEBUG:boto:Token: None 2023-10-17T18:24:11.365 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:11 GMT /hbmuog-16/?versioning 2023-10-17T18:24:11.366 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tBfjBMEs0sqX08Y1ht9qzlsTgNA= 2023-10-17T18:24:11.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tBfjBMEs0sqX08Y1ht9qzlsTgNA=', 'Content-Length': '228'} 2023-10-17T18:24:11.374 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3545d9da24e00df-00652ed14b-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:11.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:11.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:11.376 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:11.433 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:11.433 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:11.468 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:24:11.480 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:24:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567051.371838_202.1", 2023-10-17T18:24:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:24:11.371838Z" 2023-10-17T18:24:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:24:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:24:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:24:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.484 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:24:11.485 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567051.371838_202.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:11.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:24:11.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:11.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:11.487 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:11.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:11.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:11.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:11.566+0000 7f88f5bfd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:11.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:11.567+0000 7f88f5bfd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:11.579 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:24:11.579 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:24:11.580 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:24:11.580 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:24:11.580 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:24:11.580 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:24:11.580 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:11.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.581 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:24:11.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.581 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:24:11.582 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:11.582 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:11.582 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567051.307942_191.1", 2023-10-17T18:24:11.582 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:11.582 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:24:11.583 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:11.583 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:24:11.307942Z", 2023-10-17T18:24:11.583 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:11.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.584 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:24:11.584 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:11.584 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:11.585 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:11.585 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:11.585 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:11.585 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:11.585 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:24:11.586 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:11.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:24:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:11.587 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:11.588 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:11.588 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:11.588 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:11.588 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:11.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:11.589 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:24:11.589 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:11.589 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:11.590 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:11.590 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:11.590 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:11.590 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:11.590 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:11.591 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:11.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:11.591 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:24:11.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.592 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:24:11.592 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:24:11.592 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:24:11.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:11.592 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:24:11Z" 2023-10-17T18:24:11.593 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:24:11.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567051.307942_191.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:24:11.307942Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:24:11Z" } 2023-10-17T18:24:11.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567051.307942_191.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:24:11.307942Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:24:11.595 DEBUG:rgw_multi.tests:log_status={0: '1_1697567051.371838_202.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:11.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567051.307942_191.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:11.595 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567051.371838_202.1 target=1_1697567051.307942_191.1 2023-10-17T18:24:16.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:16.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:16.598 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:16.656 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:16.656 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:16.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:16.688+0000 7f87d62b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:16.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:16.690+0000 7f87d62b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:16.702 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:24:16.703 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:24:16.703 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:24:16.703 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:24:16.705 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:24:16.705 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:24:16.706 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:16.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.706 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:24:16.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:16.707 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:24:16.707 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:16.707 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:16.707 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567051.371838_202.1", 2023-10-17T18:24:16.707 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:16.708 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:24:16.708 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:16.708 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:24:11.371838Z", 2023-10-17T18:24:16.708 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:16.708 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:16.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:16.709 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:24:16.709 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:16.709 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:16.710 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:16.710 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:16.710 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:16.710 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:16.710 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:24:16.711 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:16.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:16.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:16.711 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:24:16.712 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:16.712 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:16.712 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:16.712 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:16.712 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:16.714 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:24:16.714 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:16.714 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:16.714 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:16.714 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:16.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:16.716 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:24:16.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.716 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:24:16.716 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:24:16.716 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:24:16.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:16.717 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:24:16Z" 2023-10-17T18:24:16.717 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:24:16.718 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567051.371838_202.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:24:11.371838Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:24:16Z" } 2023-10-17T18:24:16.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567051.371838_202.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:24:11.371838Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:24:16.719 DEBUG:rgw_multi.tests:log_status={0: '1_1697567051.371838_202.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:16.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567051.371838_202.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:16.720 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:24:16.720 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:16.721 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:16.721 DEBUG:boto:Method: HEAD 2023-10-17T18:24:16.721 DEBUG:boto:Path: /hbmuog-15/ 2023-10-17T18:24:16.722 DEBUG:boto:Data: 2023-10-17T18:24:16.722 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:16.723 DEBUG:boto:Port: 8000 2023-10-17T18:24:16.723 DEBUG:boto:Params: {} 2023-10-17T18:24:16.724 DEBUG:boto:Token: None 2023-10-17T18:24:16.724 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-15/ 2023-10-17T18:24:16.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LfYaBgWx/1qeNsn3lQpkqrzBTh8= 2023-10-17T18:24:16.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LfYaBgWx/1qeNsn3lQpkqrzBTh8=', 'Content-Length': '0'} 2023-10-17T18:24:16.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023791018b43bd843-00652ed150-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.728 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:16.729 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:16.729 DEBUG:boto:Method: PUT 2023-10-17T18:24:16.729 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:16.730 DEBUG:boto:Data: 2023-10-17T18:24:16.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:24:16.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:16.731 DEBUG:boto:Port: 8000 2023-10-17T18:24:16.731 DEBUG:boto:Params: {} 2023-10-17T18:24:16.732 DEBUG:boto:Token: None 2023-10-17T18:24:16.732 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-15/obj 2023-10-17T18:24:16.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4IqhCFId7c/xLi6LwHzJrtysq44= 2023-10-17T18:24:16.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4IqhCFId7c/xLi6LwHzJrtysq44='} 2023-10-17T18:24:16.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-amz-request-id', 'tx0000065baa8a2ec41d8b5-00652ed150-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.779 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:16.779 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:16.780 DEBUG:boto:Method: HEAD 2023-10-17T18:24:16.780 DEBUG:boto:Path: /hbmuog-15/ 2023-10-17T18:24:16.781 DEBUG:boto:Data: 2023-10-17T18:24:16.781 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.781 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:16.782 DEBUG:boto:Port: 8000 2023-10-17T18:24:16.782 DEBUG:boto:Params: {} 2023-10-17T18:24:16.782 DEBUG:boto:Token: None 2023-10-17T18:24:16.783 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-15/ 2023-10-17T18:24:16.783 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LfYaBgWx/1qeNsn3lQpkqrzBTh8= 2023-10-17T18:24:16.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LfYaBgWx/1qeNsn3lQpkqrzBTh8=', 'Content-Length': '0'} 2023-10-17T18:24:16.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efdee9bae80e093c-00652ed150-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.786 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:16.787 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:16.787 DEBUG:boto:Method: DELETE 2023-10-17T18:24:16.787 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:16.788 DEBUG:boto:Data: 2023-10-17T18:24:16.788 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.789 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:16.789 DEBUG:boto:Port: 8000 2023-10-17T18:24:16.789 DEBUG:boto:Params: {} 2023-10-17T18:24:16.790 DEBUG:boto:Token: None 2023-10-17T18:24:16.790 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-15/obj 2023-10-17T18:24:16.791 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/HKRlNnQ695oRUhIowkpVX6ZrPc= 2023-10-17T18:24:16.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/HKRlNnQ695oRUhIowkpVX6ZrPc=', 'Content-Length': '0'} 2023-10-17T18:24:16.797 DEBUG:boto:Response headers: [('x-amz-version-id', 'EwtcRPiDNtS8dOWRqDJbgTOsveuEsX5'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000cf26178ddf13de04-00652ed150-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.798 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:16.798 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:16.799 DEBUG:boto:Method: HEAD 2023-10-17T18:24:16.800 DEBUG:boto:Path: /hbmuog-16/ 2023-10-17T18:24:16.800 DEBUG:boto:Data: 2023-10-17T18:24:16.801 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:16.801 DEBUG:boto:Port: 8001 2023-10-17T18:24:16.802 DEBUG:boto:Params: {} 2023-10-17T18:24:16.802 DEBUG:boto:Token: None 2023-10-17T18:24:16.802 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-16/ 2023-10-17T18:24:16.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K3h3pUzz0VdLVSdyC8USuMvjqsc= 2023-10-17T18:24:16.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K3h3pUzz0VdLVSdyC8USuMvjqsc=', 'Content-Length': '0'} 2023-10-17T18:24:16.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000601b60cd4a748d14-00652ed150-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.806 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:16.807 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:16.807 DEBUG:boto:Method: PUT 2023-10-17T18:24:16.808 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:16.808 DEBUG:boto:Data: 2023-10-17T18:24:16.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:24:16.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:16.809 DEBUG:boto:Port: 8001 2023-10-17T18:24:16.809 DEBUG:boto:Params: {} 2023-10-17T18:24:16.810 DEBUG:boto:Token: None 2023-10-17T18:24:16.810 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-16/obj 2023-10-17T18:24:16.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DdUYsv1h/0CpWLHNqNcuOj5w8pg= 2023-10-17T18:24:16.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DdUYsv1h/0CpWLHNqNcuOj5w8pg='} 2023-10-17T18:24:16.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-amz-request-id', 'tx00000f11dea3301819533-00652ed150-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.822 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:16.822 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:16.823 DEBUG:boto:Method: HEAD 2023-10-17T18:24:16.823 DEBUG:boto:Path: /hbmuog-16/ 2023-10-17T18:24:16.823 DEBUG:boto:Data: 2023-10-17T18:24:16.824 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.824 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:16.824 DEBUG:boto:Port: 8001 2023-10-17T18:24:16.825 DEBUG:boto:Params: {} 2023-10-17T18:24:16.825 DEBUG:boto:Token: None 2023-10-17T18:24:16.826 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-16/ 2023-10-17T18:24:16.826 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K3h3pUzz0VdLVSdyC8USuMvjqsc= 2023-10-17T18:24:16.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K3h3pUzz0VdLVSdyC8USuMvjqsc=', 'Content-Length': '0'} 2023-10-17T18:24:16.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000000d723ee5594344-00652ed150-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.829 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:16.830 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:16.830 DEBUG:boto:Method: DELETE 2023-10-17T18:24:16.830 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:16.831 DEBUG:boto:Data: 2023-10-17T18:24:16.831 DEBUG:boto:Headers: {} 2023-10-17T18:24:16.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:16.832 DEBUG:boto:Port: 8001 2023-10-17T18:24:16.832 DEBUG:boto:Params: {} 2023-10-17T18:24:16.833 DEBUG:boto:Token: None 2023-10-17T18:24:16.833 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:24:16 GMT /hbmuog-16/obj 2023-10-17T18:24:16.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Xr08FXCLU8klLhcXp7WVUnMuekc= 2023-10-17T18:24:16.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Xr08FXCLU8klLhcXp7WVUnMuekc=', 'Content-Length': '0'} 2023-10-17T18:24:16.840 DEBUG:boto:Response headers: [('x-amz-version-id', '3mgtw.QiYBy6RayHPLUuzguI2D0FgB6'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000006d961aebfd2c44de-00652ed150-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:16.841 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-15 2023-10-17T18:24:16.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:16.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:16.842 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:24:16.901 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:16.901 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:16.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:16.936+0000 7f375a41b780 1 waiting to reach incremental sync.. 2023-10-17T18:24:21.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:21.937+0000 7f375a41b780 1 waiting to reach incremental sync.. 2023-10-17T18:24:26.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:26.937+0000 7f375a41b780 1 waiting to reach incremental sync.. 2023-10-17T18:24:31.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:31.938+0000 7f375a41b780 1 bucket sync caught up with source: 2023-10-17T18:24:31.939 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2023-10-17T18:24:31.940 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2023-10-17T18:24:31.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:31.938+0000 7f375a41b780 0 bucket checkpoint complete 2023-10-17T18:24:31.956 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-15 2023-10-17T18:24:31.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:31.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:31.957 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:24:32.022 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:32.023 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:32.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:32.059+0000 7f0d2c277780 1 bucket sync caught up with source: 2023-10-17T18:24:32.061 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000002.33.6, , ] 2023-10-17T18:24:32.061 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000002.33.6, , ] 2023-10-17T18:24:32.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:32.059+0000 7f0d2c277780 0 bucket checkpoint complete 2023-10-17T18:24:32.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-15 zones={a2, a1} 2023-10-17T18:24:32.076 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.076 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.077 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.077 DEBUG:boto:Path: /hbmuog-15/ 2023-10-17T18:24:32.077 DEBUG:boto:Data: 2023-10-17T18:24:32.078 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.078 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.079 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.079 DEBUG:boto:Params: {} 2023-10-17T18:24:32.080 DEBUG:boto:Token: None 2023-10-17T18:24:32.080 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/ 2023-10-17T18:24:32.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ndw1X/WiqgYuN1y8jAxLzU6tffM= 2023-10-17T18:24:32.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ndw1X/WiqgYuN1y8jAxLzU6tffM=', 'Content-Length': '0'} 2023-10-17T18:24:32.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a1a6a210ff3fa26-00652ed160-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.084 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.085 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.085 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.085 DEBUG:boto:Path: /hbmuog-15/ 2023-10-17T18:24:32.086 DEBUG:boto:Data: 2023-10-17T18:24:32.086 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.087 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.087 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.087 DEBUG:boto:Params: {} 2023-10-17T18:24:32.088 DEBUG:boto:Token: None 2023-10-17T18:24:32.088 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/ 2023-10-17T18:24:32.089 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ndw1X/WiqgYuN1y8jAxLzU6tffM= 2023-10-17T18:24:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ndw1X/WiqgYuN1y8jAxLzU6tffM=', 'Content-Length': '0'} 2023-10-17T18:24:32.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007543d3e31b0a2924-00652ed160-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.092 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:24:32.092 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.093 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.093 DEBUG:boto:path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.094 DEBUG:boto:auth_path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.094 DEBUG:boto:Method: GET 2023-10-17T18:24:32.095 DEBUG:boto:Path: /hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.095 DEBUG:boto:Data: 2023-10-17T18:24:32.096 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.096 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.097 DEBUG:boto:Params: {} 2023-10-17T18:24:32.097 DEBUG:boto:Token: None 2023-10-17T18:24:32.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/?versions 2023-10-17T18:24:32.098 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc= 2023-10-17T18:24:32.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc=', 'Content-Length': '0'} 2023-10-17T18:24:32.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff399b7e41606cc-00652ed160-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.146 DEBUG:boto:b'hbmuog-15999falseobjEwtcRPiDNtS8dOWRqDJbgTOsveuEsX5true2023-10-17T18:24:16.793Zrgw-multisite-test-userMultisite Test UserNormalobjj8IQ-IYOw-iXHVn9mGopArAXC2No6jCfalse2023-10-17T18:24:16.735Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:32.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:32.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:32.149 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:24:32.149 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.150 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.151 DEBUG:boto:path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.151 DEBUG:boto:auth_path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.152 DEBUG:boto:Method: GET 2023-10-17T18:24:32.152 DEBUG:boto:Path: /hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.152 DEBUG:boto:Data: 2023-10-17T18:24:32.153 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.153 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.154 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.154 DEBUG:boto:Params: {} 2023-10-17T18:24:32.155 DEBUG:boto:Token: None 2023-10-17T18:24:32.155 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/?versions 2023-10-17T18:24:32.156 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc= 2023-10-17T18:24:32.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc=', 'Content-Length': '0'} 2023-10-17T18:24:32.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ddd43fb5bf344d-00652ed160-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.159 DEBUG:boto:b'hbmuog-15999falseobjEwtcRPiDNtS8dOWRqDJbgTOsveuEsX5true2023-10-17T18:24:16.793Zrgw-multisite-test-userMultisite Test UserNormalobjj8IQ-IYOw-iXHVn9mGopArAXC2No6jCfalse2023-10-17T18:24:16.735Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:32.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:32.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:32.161 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.161 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.162 DEBUG:boto:path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.162 DEBUG:boto:auth_path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.162 DEBUG:boto:Method: GET 2023-10-17T18:24:32.163 DEBUG:boto:Path: /hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.163 DEBUG:boto:Data: 2023-10-17T18:24:32.164 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.165 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.165 DEBUG:boto:Params: {} 2023-10-17T18:24:32.166 DEBUG:boto:Token: None 2023-10-17T18:24:32.166 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/?versions 2023-10-17T18:24:32.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc= 2023-10-17T18:24:32.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc=', 'Content-Length': '0'} 2023-10-17T18:24:32.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c04a898bba0ceae-00652ed160-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.170 DEBUG:boto:b'hbmuog-15999falseobjEwtcRPiDNtS8dOWRqDJbgTOsveuEsX5true2023-10-17T18:24:16.793Zrgw-multisite-test-userMultisite Test UserNormalobjj8IQ-IYOw-iXHVn9mGopArAXC2No6jCfalse2023-10-17T18:24:16.735Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:32.171 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.171 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.172 DEBUG:boto:path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.172 DEBUG:boto:auth_path=/hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.173 DEBUG:boto:Method: GET 2023-10-17T18:24:32.173 DEBUG:boto:Path: /hbmuog-15/?versions&max-keys=999 2023-10-17T18:24:32.174 DEBUG:boto:Data: 2023-10-17T18:24:32.174 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.174 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.175 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.175 DEBUG:boto:Params: {} 2023-10-17T18:24:32.176 DEBUG:boto:Token: None 2023-10-17T18:24:32.176 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/?versions 2023-10-17T18:24:32.177 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc= 2023-10-17T18:24:32.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFIcW+T15rh87X4BfzQTxazNmtc=', 'Content-Length': '0'} 2023-10-17T18:24:32.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085bb917623a9c2d3-00652ed160-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.180 DEBUG:boto:b'hbmuog-15999falseobjEwtcRPiDNtS8dOWRqDJbgTOsveuEsX5true2023-10-17T18:24:16.793Zrgw-multisite-test-userMultisite Test UserNormalobjj8IQ-IYOw-iXHVn9mGopArAXC2No6jCfalse2023-10-17T18:24:16.735Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:32.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:32.181 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.182 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.182 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.183 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:32.183 DEBUG:boto:Data: 2023-10-17T18:24:32.184 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.184 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.185 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.185 DEBUG:boto:Params: {} 2023-10-17T18:24:32.185 DEBUG:boto:Token: None 2023-10-17T18:24:32.186 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj 2023-10-17T18:24:32.186 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk= 2023-10-17T18:24:32.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk=', 'Content-Length': '0'} 2023-10-17T18:24:32.189 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000010526ccee581b33a-00652ed160-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.189 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.190 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.190 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.191 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:32.191 DEBUG:boto:Data: 2023-10-17T18:24:32.192 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.192 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.193 DEBUG:boto:Params: {} 2023-10-17T18:24:32.193 DEBUG:boto:Token: None 2023-10-17T18:24:32.194 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj 2023-10-17T18:24:32.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk= 2023-10-17T18:24:32.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk=', 'Content-Length': '0'} 2023-10-17T18:24:32.198 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d89927b17b98bc78-00652ed160-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:32.199 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.199 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.200 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.200 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.201 DEBUG:boto:Method: GET 2023-10-17T18:24:32.201 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.201 DEBUG:boto:Data: 2023-10-17T18:24:32.202 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.203 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.203 DEBUG:boto:Params: {} 2023-10-17T18:24:32.204 DEBUG:boto:Token: None 2023-10-17T18:24:32.204 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ= 2023-10-17T18:24:32.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ=', 'Content-Length': '0'} 2023-10-17T18:24:32.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c5d902873d2c0ba-00652ed160-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.208 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.208 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.209 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.209 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.209 DEBUG:boto:Method: GET 2023-10-17T18:24:32.210 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.210 DEBUG:boto:Data: 2023-10-17T18:24:32.211 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.211 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.211 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.212 DEBUG:boto:Params: {} 2023-10-17T18:24:32.212 DEBUG:boto:Token: None 2023-10-17T18:24:32.213 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.213 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ= 2023-10-17T18:24:32.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ=', 'Content-Length': '0'} 2023-10-17T18:24:32.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000426365fb8dec77e4-00652ed160-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.216 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.216 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.217 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.217 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.217 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.218 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.218 DEBUG:boto:Data: 2023-10-17T18:24:32.219 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.219 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.220 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.220 DEBUG:boto:Params: {} 2023-10-17T18:24:32.220 DEBUG:boto:Token: None 2023-10-17T18:24:32.221 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.222 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l0IQ0nBr4Z8/TxXiYBjiHMLdNIU= 2023-10-17T18:24:32.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l0IQ0nBr4Z8/TxXiYBjiHMLdNIU=', 'Content-Length': '0'} 2023-10-17T18:24:32.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d4c0049fc1ea526-00652ed160-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.225 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.225 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.226 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.226 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.227 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.228 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.228 DEBUG:boto:Data: 2023-10-17T18:24:32.229 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.230 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.230 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.231 DEBUG:boto:Params: {} 2023-10-17T18:24:32.232 DEBUG:boto:Token: None 2023-10-17T18:24:32.232 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.233 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l0IQ0nBr4Z8/TxXiYBjiHMLdNIU= 2023-10-17T18:24:32.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l0IQ0nBr4Z8/TxXiYBjiHMLdNIU=', 'Content-Length': '0'} 2023-10-17T18:24:32.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041c1aa9a9c49aa34-00652ed160-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:32.237 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.237 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.238 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.238 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.239 DEBUG:boto:Method: GET 2023-10-17T18:24:32.240 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.241 DEBUG:boto:Data: 2023-10-17T18:24:32.241 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.243 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.243 DEBUG:boto:Params: {} 2023-10-17T18:24:32.244 DEBUG:boto:Token: None 2023-10-17T18:24:32.245 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ= 2023-10-17T18:24:32.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ=', 'Content-Length': '0'} 2023-10-17T18:24:32.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066219394b43b576d-00652ed160-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.248 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.248 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.249 DEBUG:boto:path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.249 DEBUG:boto:auth_path=/hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.250 DEBUG:boto:Method: GET 2023-10-17T18:24:32.250 DEBUG:boto:Path: /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.251 DEBUG:boto:Data: 2023-10-17T18:24:32.251 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.252 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.252 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.253 DEBUG:boto:Params: {} 2023-10-17T18:24:32.253 DEBUG:boto:Token: None 2023-10-17T18:24:32.254 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj?versionId=j8IQ-IYOw-iXHVn9mGopArAXC2No6jC 2023-10-17T18:24:32.254 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ= 2023-10-17T18:24:32.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AKWen0zVEK+69OgZK8e/XejKQAQ=', 'Content-Length': '0'} 2023-10-17T18:24:32.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', 'j8IQ-IYOw-iXHVn9mGopArAXC2No6jC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aeef472f829aa2db-00652ed160-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.257 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.257 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.258 DEBUG:boto:path=/hbmuog-15/?prefix=obj 2023-10-17T18:24:32.258 DEBUG:boto:auth_path=/hbmuog-15/?prefix=obj 2023-10-17T18:24:32.259 DEBUG:boto:Method: GET 2023-10-17T18:24:32.259 DEBUG:boto:Path: /hbmuog-15/?prefix=obj 2023-10-17T18:24:32.259 DEBUG:boto:Data: 2023-10-17T18:24:32.260 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.261 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.261 DEBUG:boto:Params: {} 2023-10-17T18:24:32.261 DEBUG:boto:Token: None 2023-10-17T18:24:32.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/ 2023-10-17T18:24:32.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M4mEAWolamzrLwaSFGcknH0kGHM= 2023-10-17T18:24:32.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M4mEAWolamzrLwaSFGcknH0kGHM=', 'Content-Length': '0'} 2023-10-17T18:24:32.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764d8aa8e0605171-00652ed160-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.266 DEBUG:boto:b'hbmuog-15obj1000false' 2023-10-17T18:24:32.267 DEBUG:boto:path=/hbmuog-15/ 2023-10-17T18:24:32.267 DEBUG:boto:auth_path=/hbmuog-15/ 2023-10-17T18:24:32.267 DEBUG:boto:path=/hbmuog-15/?prefix=obj 2023-10-17T18:24:32.268 DEBUG:boto:auth_path=/hbmuog-15/?prefix=obj 2023-10-17T18:24:32.268 DEBUG:boto:Method: GET 2023-10-17T18:24:32.269 DEBUG:boto:Path: /hbmuog-15/?prefix=obj 2023-10-17T18:24:32.269 DEBUG:boto:Data: 2023-10-17T18:24:32.270 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.271 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.271 DEBUG:boto:Params: {} 2023-10-17T18:24:32.271 DEBUG:boto:Token: None 2023-10-17T18:24:32.272 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/ 2023-10-17T18:24:32.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M4mEAWolamzrLwaSFGcknH0kGHM= 2023-10-17T18:24:32.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M4mEAWolamzrLwaSFGcknH0kGHM=', 'Content-Length': '0'} 2023-10-17T18:24:32.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30483e2aece5f7f-00652ed160-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.318 DEBUG:boto:b'hbmuog-15obj1000false' 2023-10-17T18:24:32.319 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.319 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.320 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.321 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:32.321 DEBUG:boto:Data: 2023-10-17T18:24:32.321 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:32.323 DEBUG:boto:Port: 8001 2023-10-17T18:24:32.323 DEBUG:boto:Params: {} 2023-10-17T18:24:32.324 DEBUG:boto:Token: None 2023-10-17T18:24:32.325 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj 2023-10-17T18:24:32.326 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk= 2023-10-17T18:24:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk=', 'Content-Length': '0'} 2023-10-17T18:24:32.329 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d3552757a9c28f0d-00652ed160-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.329 DEBUG:boto:path=/hbmuog-15/obj 2023-10-17T18:24:32.330 DEBUG:boto:auth_path=/hbmuog-15/obj 2023-10-17T18:24:32.330 DEBUG:boto:Method: HEAD 2023-10-17T18:24:32.330 DEBUG:boto:Path: /hbmuog-15/obj 2023-10-17T18:24:32.331 DEBUG:boto:Data: 2023-10-17T18:24:32.331 DEBUG:boto:Headers: {} 2023-10-17T18:24:32.332 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:32.332 DEBUG:boto:Port: 8000 2023-10-17T18:24:32.333 DEBUG:boto:Params: {} 2023-10-17T18:24:32.333 DEBUG:boto:Token: None 2023-10-17T18:24:32.334 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:32 GMT /hbmuog-15/obj 2023-10-17T18:24:32.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk= 2023-10-17T18:24:32.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g+Vpo1yV0SAUbtKNVXzfpep6YZk=', 'Content-Length': '0'} 2023-10-17T18:24:32.337 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001b41ec891d1919db-00652ed160-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:32.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-15 zones={a2, a1} 2023-10-17T18:24:32.338 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-16 2023-10-17T18:24:32.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:32.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:32.339 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:24:32.397 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:32.398 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:32.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:32.429+0000 7fefc0862780 1 waiting to reach incremental sync.. 2023-10-17T18:24:37.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:37.429+0000 7fefc0862780 1 waiting to reach incremental sync.. 2023-10-17T18:24:42.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:42.430+0000 7fefc0862780 1 waiting to reach incremental sync.. 2023-10-17T18:24:47.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:47.431+0000 7fefc0862780 1 waiting to reach incremental sync.. 2023-10-17T18:24:52.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:52.433+0000 7fefc0862780 1 bucket sync caught up with source: 2023-10-17T18:24:52.434 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000002.13.6, , ] 2023-10-17T18:24:52.434 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000002.13.6, , ] 2023-10-17T18:24:52.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:52.433+0000 7fefc0862780 0 bucket checkpoint complete 2023-10-17T18:24:52.450 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-16 2023-10-17T18:24:52.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:52.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:52.451 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:24:52.515 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:52.516 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:52.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:52.553+0000 7f7a44d21780 1 bucket sync caught up with source: 2023-10-17T18:24:52.554 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.32.12, , ] 2023-10-17T18:24:52.554 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.32.12, , ] 2023-10-17T18:24:52.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:52.553+0000 7f7a44d21780 0 bucket checkpoint complete 2023-10-17T18:24:52.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-16 zones={a2, a1} 2023-10-17T18:24:52.567 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.567 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.568 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.568 DEBUG:boto:Path: /hbmuog-16/ 2023-10-17T18:24:52.568 DEBUG:boto:Data: 2023-10-17T18:24:52.569 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.570 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.570 DEBUG:boto:Params: {} 2023-10-17T18:24:52.571 DEBUG:boto:Token: None 2023-10-17T18:24:52.571 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/ 2023-10-17T18:24:52.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+HgMteEF294ZUmU7FlWqAL5s/Uw= 2023-10-17T18:24:52.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+HgMteEF294ZUmU7FlWqAL5s/Uw=', 'Content-Length': '0'} 2023-10-17T18:24:52.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a6b497cc99ef45a-00652ed174-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.575 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.576 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.576 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.576 DEBUG:boto:Path: /hbmuog-16/ 2023-10-17T18:24:52.577 DEBUG:boto:Data: 2023-10-17T18:24:52.577 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.578 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.578 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.578 DEBUG:boto:Params: {} 2023-10-17T18:24:52.579 DEBUG:boto:Token: None 2023-10-17T18:24:52.579 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/ 2023-10-17T18:24:52.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+HgMteEF294ZUmU7FlWqAL5s/Uw= 2023-10-17T18:24:52.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+HgMteEF294ZUmU7FlWqAL5s/Uw=', 'Content-Length': '0'} 2023-10-17T18:24:52.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9c9b341eb0c696e-00652ed174-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:24:52.583 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.584 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.584 DEBUG:boto:path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.585 DEBUG:boto:auth_path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.585 DEBUG:boto:Method: GET 2023-10-17T18:24:52.586 DEBUG:boto:Path: /hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.586 DEBUG:boto:Data: 2023-10-17T18:24:52.586 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.587 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.588 DEBUG:boto:Params: {} 2023-10-17T18:24:52.588 DEBUG:boto:Token: None 2023-10-17T18:24:52.589 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/?versions 2023-10-17T18:24:52.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI= 2023-10-17T18:24:52.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI=', 'Content-Length': '0'} 2023-10-17T18:24:52.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be96a9cbb07349a7-00652ed174-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.634 DEBUG:boto:b'hbmuog-16999falseobj3mgtw.QiYBy6RayHPLUuzguI2D0FgB6true2023-10-17T18:24:16.836Zrgw-multisite-test-userMultisite Test UserNormalobj6DOywNo66EGqKxQPq1FNNkjFyKoqjg7false2023-10-17T18:24:16.813Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:52.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:52.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:52.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:24:52.636 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.637 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.637 DEBUG:boto:path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.638 DEBUG:boto:auth_path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.638 DEBUG:boto:Method: GET 2023-10-17T18:24:52.638 DEBUG:boto:Path: /hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.639 DEBUG:boto:Data: 2023-10-17T18:24:52.639 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.640 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.640 DEBUG:boto:Params: {} 2023-10-17T18:24:52.641 DEBUG:boto:Token: None 2023-10-17T18:24:52.641 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/?versions 2023-10-17T18:24:52.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI= 2023-10-17T18:24:52.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI=', 'Content-Length': '0'} 2023-10-17T18:24:52.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d788077fe92c1891-00652ed174-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.645 DEBUG:boto:b'hbmuog-16999falseobj3mgtw.QiYBy6RayHPLUuzguI2D0FgB6true2023-10-17T18:24:16.836Zrgw-multisite-test-userMultisite Test UserNormalobj6DOywNo66EGqKxQPq1FNNkjFyKoqjg7false2023-10-17T18:24:16.813Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:52.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:52.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:24:52.647 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.647 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.647 DEBUG:boto:path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.648 DEBUG:boto:auth_path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.648 DEBUG:boto:Method: GET 2023-10-17T18:24:52.649 DEBUG:boto:Path: /hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.649 DEBUG:boto:Data: 2023-10-17T18:24:52.650 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.650 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.651 DEBUG:boto:Params: {} 2023-10-17T18:24:52.651 DEBUG:boto:Token: None 2023-10-17T18:24:52.652 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/?versions 2023-10-17T18:24:52.652 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI= 2023-10-17T18:24:52.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI=', 'Content-Length': '0'} 2023-10-17T18:24:52.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8fa60ea1800f5c-00652ed174-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.655 DEBUG:boto:b'hbmuog-16999falseobj3mgtw.QiYBy6RayHPLUuzguI2D0FgB6true2023-10-17T18:24:16.836Zrgw-multisite-test-userMultisite Test UserNormalobj6DOywNo66EGqKxQPq1FNNkjFyKoqjg7false2023-10-17T18:24:16.813Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:52.656 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.657 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.657 DEBUG:boto:path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.657 DEBUG:boto:auth_path=/hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.658 DEBUG:boto:Method: GET 2023-10-17T18:24:52.658 DEBUG:boto:Path: /hbmuog-16/?versions&max-keys=999 2023-10-17T18:24:52.659 DEBUG:boto:Data: 2023-10-17T18:24:52.659 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.660 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.660 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.660 DEBUG:boto:Params: {} 2023-10-17T18:24:52.661 DEBUG:boto:Token: None 2023-10-17T18:24:52.661 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/?versions 2023-10-17T18:24:52.662 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI= 2023-10-17T18:24:52.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:An7XtdVgmo6NCjiCfYuDNfwt7YI=', 'Content-Length': '0'} 2023-10-17T18:24:52.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3984914d793152-00652ed174-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.665 DEBUG:boto:b'hbmuog-16999falseobj3mgtw.QiYBy6RayHPLUuzguI2D0FgB6true2023-10-17T18:24:16.836Zrgw-multisite-test-userMultisite Test UserNormalobj6DOywNo66EGqKxQPq1FNNkjFyKoqjg7false2023-10-17T18:24:16.813Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:24:52.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:52.666 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.667 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.667 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.668 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:52.669 DEBUG:boto:Data: 2023-10-17T18:24:52.670 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.671 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.671 DEBUG:boto:Params: {} 2023-10-17T18:24:52.672 DEBUG:boto:Token: None 2023-10-17T18:24:52.672 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj 2023-10-17T18:24:52.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks= 2023-10-17T18:24:52.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks=', 'Content-Length': '0'} 2023-10-17T18:24:52.675 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e63390a8edf20a1d-00652ed174-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.675 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.676 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.676 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.677 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:52.677 DEBUG:boto:Data: 2023-10-17T18:24:52.678 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.678 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.678 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.679 DEBUG:boto:Params: {} 2023-10-17T18:24:52.679 DEBUG:boto:Token: None 2023-10-17T18:24:52.680 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj 2023-10-17T18:24:52.680 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks= 2023-10-17T18:24:52.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks=', 'Content-Length': '0'} 2023-10-17T18:24:52.683 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d566aa71b8e90656-00652ed174-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:52.684 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.684 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.684 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.685 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.685 DEBUG:boto:Method: GET 2023-10-17T18:24:52.686 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.686 DEBUG:boto:Data: 2023-10-17T18:24:52.687 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.688 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.688 DEBUG:boto:Params: {} 2023-10-17T18:24:52.688 DEBUG:boto:Token: None 2023-10-17T18:24:52.689 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.689 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc= 2023-10-17T18:24:52.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc=', 'Content-Length': '0'} 2023-10-17T18:24:52.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a004fd66cf2964b-00652ed174-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.693 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.693 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.693 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.694 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.694 DEBUG:boto:Method: GET 2023-10-17T18:24:52.695 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.695 DEBUG:boto:Data: 2023-10-17T18:24:52.696 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.697 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.697 DEBUG:boto:Params: {} 2023-10-17T18:24:52.697 DEBUG:boto:Token: None 2023-10-17T18:24:52.698 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc= 2023-10-17T18:24:52.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc=', 'Content-Length': '0'} 2023-10-17T18:24:52.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b995a5fa3933f576-00652ed174-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.701 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.702 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.702 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.703 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.703 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.703 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.704 DEBUG:boto:Data: 2023-10-17T18:24:52.704 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.705 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.705 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.705 DEBUG:boto:Params: {} 2023-10-17T18:24:52.706 DEBUG:boto:Token: None 2023-10-17T18:24:52.706 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A7lVPN8+rDbFNREgdpjAa0I3zpE= 2023-10-17T18:24:52.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A7lVPN8+rDbFNREgdpjAa0I3zpE=', 'Content-Length': '0'} 2023-10-17T18:24:52.709 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004525666220350bd9-00652ed174-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.709 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.710 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.710 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.711 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.711 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.712 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.712 DEBUG:boto:Data: 2023-10-17T18:24:52.712 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.713 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.713 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.714 DEBUG:boto:Params: {} 2023-10-17T18:24:52.714 DEBUG:boto:Token: None 2023-10-17T18:24:52.715 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A7lVPN8+rDbFNREgdpjAa0I3zpE= 2023-10-17T18:24:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A7lVPN8+rDbFNREgdpjAa0I3zpE=', 'Content-Length': '0'} 2023-10-17T18:24:52.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003932ef9d99585ef-00652ed174-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:24:52.719 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.719 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.719 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.720 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.720 DEBUG:boto:Method: GET 2023-10-17T18:24:52.721 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.721 DEBUG:boto:Data: 2023-10-17T18:24:52.721 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.722 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.723 DEBUG:boto:Params: {} 2023-10-17T18:24:52.723 DEBUG:boto:Token: None 2023-10-17T18:24:52.724 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc= 2023-10-17T18:24:52.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc=', 'Content-Length': '0'} 2023-10-17T18:24:52.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8b46b68f721361f-00652ed174-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.727 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.727 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.728 DEBUG:boto:path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.728 DEBUG:boto:auth_path=/hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.729 DEBUG:boto:Method: GET 2023-10-17T18:24:52.729 DEBUG:boto:Path: /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.730 DEBUG:boto:Data: 2023-10-17T18:24:52.730 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.731 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.731 DEBUG:boto:Params: {} 2023-10-17T18:24:52.732 DEBUG:boto:Token: None 2023-10-17T18:24:52.732 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj?versionId=6DOywNo66EGqKxQPq1FNNkjFyKoqjg7 2023-10-17T18:24:52.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc= 2023-10-17T18:24:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j3bj3jQOy08RXnqVu5xVJ735gcc=', 'Content-Length': '0'} 2023-10-17T18:24:52.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:24:16 GMT'), ('x-amz-version-id', '6DOywNo66EGqKxQPq1FNNkjFyKoqjg7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021d66def823dfbbc-00652ed174-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.736 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.736 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.737 DEBUG:boto:path=/hbmuog-16/?prefix=obj 2023-10-17T18:24:52.737 DEBUG:boto:auth_path=/hbmuog-16/?prefix=obj 2023-10-17T18:24:52.738 DEBUG:boto:Method: GET 2023-10-17T18:24:52.738 DEBUG:boto:Path: /hbmuog-16/?prefix=obj 2023-10-17T18:24:52.738 DEBUG:boto:Data: 2023-10-17T18:24:52.739 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.739 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.740 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.740 DEBUG:boto:Params: {} 2023-10-17T18:24:52.741 DEBUG:boto:Token: None 2023-10-17T18:24:52.741 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/ 2023-10-17T18:24:52.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aI8VNC9Lx1UhbAjAEzULTZdoxHA= 2023-10-17T18:24:52.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aI8VNC9Lx1UhbAjAEzULTZdoxHA=', 'Content-Length': '0'} 2023-10-17T18:24:52.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc328f764609e9fc-00652ed174-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.786 DEBUG:boto:b'hbmuog-16obj1000false' 2023-10-17T18:24:52.787 DEBUG:boto:path=/hbmuog-16/ 2023-10-17T18:24:52.788 DEBUG:boto:auth_path=/hbmuog-16/ 2023-10-17T18:24:52.788 DEBUG:boto:path=/hbmuog-16/?prefix=obj 2023-10-17T18:24:52.789 DEBUG:boto:auth_path=/hbmuog-16/?prefix=obj 2023-10-17T18:24:52.789 DEBUG:boto:Method: GET 2023-10-17T18:24:52.789 DEBUG:boto:Path: /hbmuog-16/?prefix=obj 2023-10-17T18:24:52.790 DEBUG:boto:Data: 2023-10-17T18:24:52.790 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.791 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.791 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.792 DEBUG:boto:Params: {} 2023-10-17T18:24:52.792 DEBUG:boto:Token: None 2023-10-17T18:24:52.793 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/ 2023-10-17T18:24:52.793 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aI8VNC9Lx1UhbAjAEzULTZdoxHA= 2023-10-17T18:24:52.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aI8VNC9Lx1UhbAjAEzULTZdoxHA=', 'Content-Length': '0'} 2023-10-17T18:24:52.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031f076b6fc42ac6b-00652ed174-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.797 DEBUG:boto:b'hbmuog-16obj1000false' 2023-10-17T18:24:52.797 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.798 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.799 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.800 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:52.800 DEBUG:boto:Data: 2023-10-17T18:24:52.801 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.802 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.802 DEBUG:boto:Params: {} 2023-10-17T18:24:52.803 DEBUG:boto:Token: None 2023-10-17T18:24:52.803 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj 2023-10-17T18:24:52.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks= 2023-10-17T18:24:52.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks=', 'Content-Length': '0'} 2023-10-17T18:24:52.806 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000dd16e3351ee5813c-00652ed174-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.807 DEBUG:boto:path=/hbmuog-16/obj 2023-10-17T18:24:52.807 DEBUG:boto:auth_path=/hbmuog-16/obj 2023-10-17T18:24:52.807 DEBUG:boto:Method: HEAD 2023-10-17T18:24:52.808 DEBUG:boto:Path: /hbmuog-16/obj 2023-10-17T18:24:52.808 DEBUG:boto:Data: 2023-10-17T18:24:52.809 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.809 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.810 DEBUG:boto:Params: {} 2023-10-17T18:24:52.810 DEBUG:boto:Token: None 2023-10-17T18:24:52.811 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-16/obj 2023-10-17T18:24:52.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks= 2023-10-17T18:24:52.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVa68iA7of5NbpMqVRAGqKaRRks=', 'Content-Length': '0'} 2023-10-17T18:24:52.814 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b5caaf8ac46da6d7-00652ed174-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-16 zones={a2, a1} 2023-10-17T18:24:52.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-10-17T18:24:52.816 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-17 2023-10-17T18:24:52.816 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:24:52.816 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:24:52.817 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.817 DEBUG:boto:Path: /hbmuog-17/ 2023-10-17T18:24:52.818 DEBUG:boto:Data: 2023-10-17T18:24:52.818 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.819 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.819 DEBUG:boto:Params: {} 2023-10-17T18:24:52.820 DEBUG:boto:Token: None 2023-10-17T18:24:52.820 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-17/ 2023-10-17T18:24:52.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+UImx4IuTPM0V9oQIpoMaAmVvEI= 2023-10-17T18:24:52.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+UImx4IuTPM0V9oQIpoMaAmVvEI=', 'Content-Length': '0'} 2023-10-17T18:24:52.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091eec4f8ddd2bc35-00652ed174-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.834 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-18 2023-10-17T18:24:52.835 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:24:52.835 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:24:52.835 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.836 DEBUG:boto:Path: /hbmuog-18/ 2023-10-17T18:24:52.836 DEBUG:boto:Data: 2023-10-17T18:24:52.837 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.837 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.838 DEBUG:boto:Params: {} 2023-10-17T18:24:52.838 DEBUG:boto:Token: None 2023-10-17T18:24:52.839 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-18/ 2023-10-17T18:24:52.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ilrRSfQVg+pcu1i9yTp/943ceg0= 2023-10-17T18:24:52.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ilrRSfQVg+pcu1i9yTp/943ceg0=', 'Content-Length': '0'} 2023-10-17T18:24:52.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8aa32f2f948e471-00652ed174-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.862 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:24:52.862 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:24:52.862 DEBUG:boto:path=/hbmuog-17/?versioning 2023-10-17T18:24:52.863 DEBUG:boto:auth_path=/hbmuog-17/?versioning 2023-10-17T18:24:52.863 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.864 DEBUG:boto:Path: /hbmuog-17/?versioning 2023-10-17T18:24:52.864 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:24:52.864 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.865 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.865 DEBUG:boto:Params: {} 2023-10-17T18:24:52.866 DEBUG:boto:Token: None 2023-10-17T18:24:52.866 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-17/?versioning 2023-10-17T18:24:52.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dL/cIP0XVpS66wpktLohgpiciFI= 2023-10-17T18:24:52.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dL/cIP0XVpS66wpktLohgpiciFI=', 'Content-Length': '228'} 2023-10-17T18:24:52.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f928161a88e9c4dd-00652ed174-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.873 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:24:52.873 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:24:52.873 DEBUG:boto:path=/hbmuog-17/?versioning 2023-10-17T18:24:52.875 DEBUG:boto:auth_path=/hbmuog-17/?versioning 2023-10-17T18:24:52.875 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.876 DEBUG:boto:Path: /hbmuog-17/?versioning 2023-10-17T18:24:52.876 DEBUG:boto:Data: Suspended Disabled 2023-10-17T18:24:52.876 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.877 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:52.877 DEBUG:boto:Port: 8000 2023-10-17T18:24:52.878 DEBUG:boto:Params: {} 2023-10-17T18:24:52.878 DEBUG:boto:Token: None 2023-10-17T18:24:52.878 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-17/?versioning 2023-10-17T18:24:52.879 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dL/cIP0XVpS66wpktLohgpiciFI= 2023-10-17T18:24:52.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dL/cIP0XVpS66wpktLohgpiciFI=', 'Content-Length': '230'} 2023-10-17T18:24:52.884 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e97a2196fc94f7e-00652ed174-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.885 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:24:52.885 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:24:52.885 DEBUG:boto:path=/hbmuog-18/?versioning 2023-10-17T18:24:52.886 DEBUG:boto:auth_path=/hbmuog-18/?versioning 2023-10-17T18:24:52.886 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.886 DEBUG:boto:Path: /hbmuog-18/?versioning 2023-10-17T18:24:52.887 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:24:52.887 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.888 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.888 DEBUG:boto:Params: {} 2023-10-17T18:24:52.889 DEBUG:boto:Token: None 2023-10-17T18:24:52.890 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-18/?versioning 2023-10-17T18:24:52.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pNlNVk2vs3z0X2xzw/hHFE+0vKM= 2023-10-17T18:24:52.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pNlNVk2vs3z0X2xzw/hHFE+0vKM=', 'Content-Length': '228'} 2023-10-17T18:24:52.898 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006d06004095decc37-00652ed174-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.899 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:24:52.899 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:24:52.900 DEBUG:boto:path=/hbmuog-18/?versioning 2023-10-17T18:24:52.900 DEBUG:boto:auth_path=/hbmuog-18/?versioning 2023-10-17T18:24:52.900 DEBUG:boto:Method: PUT 2023-10-17T18:24:52.901 DEBUG:boto:Path: /hbmuog-18/?versioning 2023-10-17T18:24:52.901 DEBUG:boto:Data: Suspended Disabled 2023-10-17T18:24:52.902 DEBUG:boto:Headers: {} 2023-10-17T18:24:52.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:52.902 DEBUG:boto:Port: 8001 2023-10-17T18:24:52.903 DEBUG:boto:Params: {} 2023-10-17T18:24:52.903 DEBUG:boto:Token: None 2023-10-17T18:24:52.904 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:24:52 GMT /hbmuog-18/?versioning 2023-10-17T18:24:52.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pNlNVk2vs3z0X2xzw/hHFE+0vKM= 2023-10-17T18:24:52.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pNlNVk2vs3z0X2xzw/hHFE+0vKM=', 'Content-Length': '230'} 2023-10-17T18:24:52.911 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a0fb4f2b2241aa1-00652ed174-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:52.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:52.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:52.913 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:52.963 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:52.963 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:52.994 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:24:53.006 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:24:53.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.006 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567092.909607_224.1", 2023-10-17T18:24:53.006 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:24:52.909607Z" 2023-10-17T18:24:53.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.007 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:53.007 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:24:53.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.008 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:53.008 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:24:53.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.009 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:53.009 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:24:53.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.010 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:24:53.011 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567092.909607_224.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:53.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:24:53.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:53.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:53.013 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:53.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:53.101+0000 7fee234b8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:53.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:53.103+0000 7fee234b8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:53.115 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:24:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:24:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:24:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:24:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:24:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:24:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:24:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:24:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567092.872126_218.1", 2023-10-17T18:24:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:24:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:24:52.872126Z", 2023-10-17T18:24:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:24:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:24:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:24:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:24:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:24:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:24:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:24:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:24:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:24:53Z" 2023-10-17T18:24:53.128 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:24:53.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567092.872126_218.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:24:52.872126Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:24:53Z" } 2023-10-17T18:24:53.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567092.872126_218.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:24:52.872126Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:24:53.130 DEBUG:rgw_multi.tests:log_status={0: '1_1697567092.909607_224.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:53.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567092.872126_218.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:53.131 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567092.909607_224.1 target=1_1697567092.872126_218.1 2023-10-17T18:24:58.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:58.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:24:58.133 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:24:58.193 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:58.193 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:58.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:58.227+0000 7fc7ed95b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:58.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:58.228+0000 7fc7ed95b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:24:58.241 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:24:58.242 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:24:58.242 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:24:58.242 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:24:58.242 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:58.243 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:24:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567092.909607_224.1", 2023-10-17T18:24:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:24:52.909607Z", 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:58.246 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:24:58.246 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:58.246 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:58.246 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:24:58.246 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:58.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:24:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:58.248 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:58.249 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:24:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:24:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:24:58.250 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:24:58.251 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:24:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:24:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:24:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.252 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:24:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:24:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:24:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:24:58.253 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:24:58Z" 2023-10-17T18:24:58.253 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:24:58.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567092.909607_224.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:24:52.909607Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:24:58Z" } 2023-10-17T18:24:58.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567092.909607_224.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:24:52.909607Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:24:58.255 DEBUG:rgw_multi.tests:log_status={0: '1_1697567092.909607_224.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:58.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567092.909607_224.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:24:58.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:24:58.256 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:24:58.257 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:24:58.257 DEBUG:boto:Method: HEAD 2023-10-17T18:24:58.257 DEBUG:boto:Path: /hbmuog-17/ 2023-10-17T18:24:58.258 DEBUG:boto:Data: 2023-10-17T18:24:58.258 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.259 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:58.259 DEBUG:boto:Port: 8000 2023-10-17T18:24:58.259 DEBUG:boto:Params: {} 2023-10-17T18:24:58.260 DEBUG:boto:Token: None 2023-10-17T18:24:58.260 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-17/ 2023-10-17T18:24:58.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:js+J13iyZlZuxfmPRnb43p7OPN4= 2023-10-17T18:24:58.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:js+J13iyZlZuxfmPRnb43p7OPN4=', 'Content-Length': '0'} 2023-10-17T18:24:58.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f141ff5d82c47321-00652ed17a-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.264 DEBUG:boto:path=/hbmuog-17/obj 2023-10-17T18:24:58.265 DEBUG:boto:auth_path=/hbmuog-17/obj 2023-10-17T18:24:58.265 DEBUG:boto:Method: PUT 2023-10-17T18:24:58.265 DEBUG:boto:Path: /hbmuog-17/obj 2023-10-17T18:24:58.266 DEBUG:boto:Data: 2023-10-17T18:24:58.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:24:58.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:58.268 DEBUG:boto:Port: 8000 2023-10-17T18:24:58.268 DEBUG:boto:Params: {} 2023-10-17T18:24:58.268 DEBUG:boto:Token: None 2023-10-17T18:24:58.269 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-17/obj 2023-10-17T18:24:58.269 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n7k4VJV3jm/J20dp3ZLfAQmJrBI= 2023-10-17T18:24:58.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n7k4VJV3jm/J20dp3ZLfAQmJrBI='} 2023-10-17T18:24:58.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1fbcf154dc5a710-00652ed17a-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.276 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:24:58.277 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:24:58.277 DEBUG:boto:Method: HEAD 2023-10-17T18:24:58.277 DEBUG:boto:Path: /hbmuog-17/ 2023-10-17T18:24:58.278 DEBUG:boto:Data: 2023-10-17T18:24:58.278 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.279 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:58.279 DEBUG:boto:Port: 8000 2023-10-17T18:24:58.279 DEBUG:boto:Params: {} 2023-10-17T18:24:58.280 DEBUG:boto:Token: None 2023-10-17T18:24:58.280 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-17/ 2023-10-17T18:24:58.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:js+J13iyZlZuxfmPRnb43p7OPN4= 2023-10-17T18:24:58.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:js+J13iyZlZuxfmPRnb43p7OPN4=', 'Content-Length': '0'} 2023-10-17T18:24:58.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d03576838e742c-00652ed17a-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.284 DEBUG:boto:path=/hbmuog-17/obj 2023-10-17T18:24:58.284 DEBUG:boto:auth_path=/hbmuog-17/obj 2023-10-17T18:24:58.284 DEBUG:boto:Method: DELETE 2023-10-17T18:24:58.285 DEBUG:boto:Path: /hbmuog-17/obj 2023-10-17T18:24:58.285 DEBUG:boto:Data: 2023-10-17T18:24:58.286 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:24:58.286 DEBUG:boto:Port: 8000 2023-10-17T18:24:58.287 DEBUG:boto:Params: {} 2023-10-17T18:24:58.287 DEBUG:boto:Token: None 2023-10-17T18:24:58.288 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-17/obj 2023-10-17T18:24:58.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PcEmtH3MqTUrW6FNSww3gLCjrzk= 2023-10-17T18:24:58.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PcEmtH3MqTUrW6FNSww3gLCjrzk=', 'Content-Length': '0'} 2023-10-17T18:24:58.297 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000004f03bcb340a57c62-00652ed17a-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.297 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:24:58.298 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:24:58.298 DEBUG:boto:Method: HEAD 2023-10-17T18:24:58.299 DEBUG:boto:Path: /hbmuog-18/ 2023-10-17T18:24:58.299 DEBUG:boto:Data: 2023-10-17T18:24:58.299 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:58.300 DEBUG:boto:Port: 8001 2023-10-17T18:24:58.301 DEBUG:boto:Params: {} 2023-10-17T18:24:58.301 DEBUG:boto:Token: None 2023-10-17T18:24:58.301 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-18/ 2023-10-17T18:24:58.302 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nUi/VyH7FCbtbdYczhFuvnF9YeM= 2023-10-17T18:24:58.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nUi/VyH7FCbtbdYczhFuvnF9YeM=', 'Content-Length': '0'} 2023-10-17T18:24:58.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c900d14f855e604-00652ed17a-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.305 DEBUG:boto:path=/hbmuog-18/obj 2023-10-17T18:24:58.306 DEBUG:boto:auth_path=/hbmuog-18/obj 2023-10-17T18:24:58.306 DEBUG:boto:Method: PUT 2023-10-17T18:24:58.307 DEBUG:boto:Path: /hbmuog-18/obj 2023-10-17T18:24:58.307 DEBUG:boto:Data: 2023-10-17T18:24:58.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-17T18:24:58.308 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:58.308 DEBUG:boto:Port: 8001 2023-10-17T18:24:58.309 DEBUG:boto:Params: {} 2023-10-17T18:24:58.309 DEBUG:boto:Token: None 2023-10-17T18:24:58.310 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-18/obj 2023-10-17T18:24:58.310 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kMw781KnxnF9rT/l7C1yGyat4/Y= 2023-10-17T18:24:58.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kMw781KnxnF9rT/l7C1yGyat4/Y='} 2023-10-17T18:24:58.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc9aae222bd5d2b7-00652ed17a-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.317 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:24:58.317 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:24:58.318 DEBUG:boto:Method: HEAD 2023-10-17T18:24:58.318 DEBUG:boto:Path: /hbmuog-18/ 2023-10-17T18:24:58.318 DEBUG:boto:Data: 2023-10-17T18:24:58.319 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:58.319 DEBUG:boto:Port: 8001 2023-10-17T18:24:58.320 DEBUG:boto:Params: {} 2023-10-17T18:24:58.320 DEBUG:boto:Token: None 2023-10-17T18:24:58.321 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-18/ 2023-10-17T18:24:58.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nUi/VyH7FCbtbdYczhFuvnF9YeM= 2023-10-17T18:24:58.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nUi/VyH7FCbtbdYczhFuvnF9YeM=', 'Content-Length': '0'} 2023-10-17T18:24:58.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000342d9d5a6cb5847a-00652ed17a-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.324 DEBUG:boto:path=/hbmuog-18/obj 2023-10-17T18:24:58.325 DEBUG:boto:auth_path=/hbmuog-18/obj 2023-10-17T18:24:58.325 DEBUG:boto:Method: DELETE 2023-10-17T18:24:58.325 DEBUG:boto:Path: /hbmuog-18/obj 2023-10-17T18:24:58.326 DEBUG:boto:Data: 2023-10-17T18:24:58.326 DEBUG:boto:Headers: {} 2023-10-17T18:24:58.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:24:58.327 DEBUG:boto:Port: 8001 2023-10-17T18:24:58.327 DEBUG:boto:Params: {} 2023-10-17T18:24:58.328 DEBUG:boto:Token: None 2023-10-17T18:24:58.328 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:24:58 GMT /hbmuog-18/obj 2023-10-17T18:24:58.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l6XevyCHxD5opcu1aWUHDuHLXG4= 2023-10-17T18:24:58.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:24:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l6XevyCHxD5opcu1aWUHDuHLXG4=', 'Content-Length': '0'} 2023-10-17T18:24:58.338 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e620bc8245f0c1c3-00652ed17a-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:24:58.339 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-17 2023-10-17T18:24:58.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:58.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:24:58.340 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:24:58.399 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:24:58.400 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:24:58.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:24:58.434+0000 7f79a7c4c780 1 waiting to reach incremental sync.. 2023-10-17T18:25:03.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:03.434+0000 7f79a7c4c780 1 waiting to reach incremental sync.. 2023-10-17T18:25:08.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:08.434+0000 7f79a7c4c780 1 waiting to reach incremental sync.. 2023-10-17T18:25:13.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:13.435+0000 7f79a7c4c780 1 bucket sync caught up with source: 2023-10-17T18:25:13.437 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.12.7, , ] 2023-10-17T18:25:13.438 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.12.7, , ] 2023-10-17T18:25:13.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:13.435+0000 7f79a7c4c780 0 bucket checkpoint complete 2023-10-17T18:25:13.453 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-17 2023-10-17T18:25:13.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:13.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:13.454 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:25:13.518 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:13.519 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:13.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:13.554+0000 7f04c4cd1780 1 waiting to reach incremental sync.. 2023-10-17T18:25:18.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:18.555+0000 7f04c4cd1780 1 waiting to reach incremental sync.. 2023-10-17T18:25:23.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:23.555+0000 7f04c4cd1780 1 waiting to reach incremental sync.. 2023-10-17T18:25:28.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:28.557+0000 7f04c4cd1780 1 waiting to reach incremental sync.. 2023-10-17T18:25:33.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.558+0000 7f04c4cd1780 1 bucket sync caught up with source: 2023-10-17T18:25:33.559 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.29.14, , ] 2023-10-17T18:25:33.559 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.29.14, , ] 2023-10-17T18:25:33.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.558+0000 7f04c4cd1780 0 bucket checkpoint complete 2023-10-17T18:25:33.575 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-17 zones={a2, a1} 2023-10-17T18:25:33.576 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.576 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.577 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.577 DEBUG:boto:Path: /hbmuog-17/ 2023-10-17T18:25:33.578 DEBUG:boto:Data: 2023-10-17T18:25:33.578 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.578 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.579 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.579 DEBUG:boto:Params: {} 2023-10-17T18:25:33.580 DEBUG:boto:Token: None 2023-10-17T18:25:33.580 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/ 2023-10-17T18:25:33.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zze5RJEfDKHcvsgI+JGllrAkSfQ= 2023-10-17T18:25:33.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zze5RJEfDKHcvsgI+JGllrAkSfQ=', 'Content-Length': '0'} 2023-10-17T18:25:33.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018a824a22b5a196a-00652ed19d-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.585 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.585 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.586 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.586 DEBUG:boto:Path: /hbmuog-17/ 2023-10-17T18:25:33.587 DEBUG:boto:Data: 2023-10-17T18:25:33.587 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.588 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.588 DEBUG:boto:Params: {} 2023-10-17T18:25:33.589 DEBUG:boto:Token: None 2023-10-17T18:25:33.589 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/ 2023-10-17T18:25:33.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zze5RJEfDKHcvsgI+JGllrAkSfQ= 2023-10-17T18:25:33.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zze5RJEfDKHcvsgI+JGllrAkSfQ=', 'Content-Length': '0'} 2023-10-17T18:25:33.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000092c628c5d42ad2c-00652ed19d-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:25:33.593 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.594 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.594 DEBUG:boto:path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.595 DEBUG:boto:auth_path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.595 DEBUG:boto:Method: GET 2023-10-17T18:25:33.596 DEBUG:boto:Path: /hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.596 DEBUG:boto:Data: 2023-10-17T18:25:33.596 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.597 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.598 DEBUG:boto:Params: {} 2023-10-17T18:25:33.598 DEBUG:boto:Token: None 2023-10-17T18:25:33.599 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/?versions 2023-10-17T18:25:33.599 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU= 2023-10-17T18:25:33.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU=', 'Content-Length': '0'} 2023-10-17T18:25:33.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b700f2a1e271d1bb-00652ed19d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.605 DEBUG:boto:b'hbmuog-17999falseobjnulltrue2023-10-17T18:24:58.290Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:25:33.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:25:33.607 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.608 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.608 DEBUG:boto:path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.608 DEBUG:boto:auth_path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.609 DEBUG:boto:Method: GET 2023-10-17T18:25:33.609 DEBUG:boto:Path: /hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.610 DEBUG:boto:Data: 2023-10-17T18:25:33.610 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.611 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.611 DEBUG:boto:Params: {} 2023-10-17T18:25:33.612 DEBUG:boto:Token: None 2023-10-17T18:25:33.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/?versions 2023-10-17T18:25:33.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU= 2023-10-17T18:25:33.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU=', 'Content-Length': '0'} 2023-10-17T18:25:33.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000171ff61c8a2e131b-00652ed19d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.616 DEBUG:boto:b'hbmuog-17999falseobjnulltrue2023-10-17T18:24:58.290Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:25:33.617 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.618 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.618 DEBUG:boto:path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.619 DEBUG:boto:auth_path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.619 DEBUG:boto:Method: GET 2023-10-17T18:25:33.620 DEBUG:boto:Path: /hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.620 DEBUG:boto:Data: 2023-10-17T18:25:33.620 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.621 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.622 DEBUG:boto:Params: {} 2023-10-17T18:25:33.622 DEBUG:boto:Token: None 2023-10-17T18:25:33.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/?versions 2023-10-17T18:25:33.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU= 2023-10-17T18:25:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU=', 'Content-Length': '0'} 2023-10-17T18:25:33.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea746ef0da71274a-00652ed19d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.666 DEBUG:boto:b'hbmuog-17999falseobjnulltrue2023-10-17T18:24:58.290Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.667 DEBUG:boto:path=/hbmuog-17/ 2023-10-17T18:25:33.668 DEBUG:boto:auth_path=/hbmuog-17/ 2023-10-17T18:25:33.668 DEBUG:boto:path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.669 DEBUG:boto:auth_path=/hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.669 DEBUG:boto:Method: GET 2023-10-17T18:25:33.669 DEBUG:boto:Path: /hbmuog-17/?versions&max-keys=999 2023-10-17T18:25:33.670 DEBUG:boto:Data: 2023-10-17T18:25:33.670 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.671 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.671 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.672 DEBUG:boto:Params: {} 2023-10-17T18:25:33.672 DEBUG:boto:Token: None 2023-10-17T18:25:33.673 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/?versions 2023-10-17T18:25:33.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU= 2023-10-17T18:25:33.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+Wy+xM+C6ul3S68l+sKBU+XFkaU=', 'Content-Length': '0'} 2023-10-17T18:25:33.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a074ecb038d675-00652ed19d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.677 DEBUG:boto:b'hbmuog-17999falseobjnulltrue2023-10-17T18:24:58.290Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:25:33.678 DEBUG:boto:path=/hbmuog-17/obj 2023-10-17T18:25:33.678 DEBUG:boto:auth_path=/hbmuog-17/obj 2023-10-17T18:25:33.679 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.679 DEBUG:boto:Path: /hbmuog-17/obj 2023-10-17T18:25:33.680 DEBUG:boto:Data: 2023-10-17T18:25:33.680 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.681 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.681 DEBUG:boto:Params: {} 2023-10-17T18:25:33.682 DEBUG:boto:Token: None 2023-10-17T18:25:33.682 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/obj 2023-10-17T18:25:33.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MFhVbc9lunzNmCfIn803S2vcHP8= 2023-10-17T18:25:33.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MFhVbc9lunzNmCfIn803S2vcHP8=', 'Content-Length': '0'} 2023-10-17T18:25:33.685 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000017d3393af9581624-00652ed19d-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.685 DEBUG:boto:path=/hbmuog-17/obj 2023-10-17T18:25:33.686 DEBUG:boto:auth_path=/hbmuog-17/obj 2023-10-17T18:25:33.686 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.687 DEBUG:boto:Path: /hbmuog-17/obj 2023-10-17T18:25:33.687 DEBUG:boto:Data: 2023-10-17T18:25:33.687 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.688 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.689 DEBUG:boto:Params: {} 2023-10-17T18:25:33.689 DEBUG:boto:Token: None 2023-10-17T18:25:33.689 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-17/obj 2023-10-17T18:25:33.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MFhVbc9lunzNmCfIn803S2vcHP8= 2023-10-17T18:25:33.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MFhVbc9lunzNmCfIn803S2vcHP8=', 'Content-Length': '0'} 2023-10-17T18:25:33.692 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007a43cfb20ae3e51b-00652ed19d-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-17 zones={a2, a1} 2023-10-17T18:25:33.694 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-18 2023-10-17T18:25:33.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:33.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:33.696 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:25:33.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:33.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:33.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.803+0000 7f0e4b530780 1 bucket sync caught up with source: 2023-10-17T18:25:33.803 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.7.14, , ] 2023-10-17T18:25:33.804 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.7.14, , ] 2023-10-17T18:25:33.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.803+0000 7f0e4b530780 0 bucket checkpoint complete 2023-10-17T18:25:33.817 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-18 2023-10-17T18:25:33.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:33.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:25:33.819 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:25:33.878 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:33.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.913+0000 7fe129719780 1 bucket sync caught up with source: 2023-10-17T18:25:33.914 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000003.27.7, , ] 2023-10-17T18:25:33.915 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000003.27.7, , ] 2023-10-17T18:25:33.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:33.913+0000 7fe129719780 0 bucket checkpoint complete 2023-10-17T18:25:33.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-18 zones={a2, a1} 2023-10-17T18:25:33.927 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.927 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.928 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.928 DEBUG:boto:Path: /hbmuog-18/ 2023-10-17T18:25:33.929 DEBUG:boto:Data: 2023-10-17T18:25:33.929 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.930 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.930 DEBUG:boto:Params: {} 2023-10-17T18:25:33.931 DEBUG:boto:Token: None 2023-10-17T18:25:33.931 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/ 2023-10-17T18:25:33.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4fo27Jwa85xPlxv6LYFaaUt/OxI= 2023-10-17T18:25:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4fo27Jwa85xPlxv6LYFaaUt/OxI=', 'Content-Length': '0'} 2023-10-17T18:25:33.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079e01e9aea73ef0f-00652ed19d-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.935 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.936 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.936 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.937 DEBUG:boto:Path: /hbmuog-18/ 2023-10-17T18:25:33.937 DEBUG:boto:Data: 2023-10-17T18:25:33.937 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.938 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.939 DEBUG:boto:Params: {} 2023-10-17T18:25:33.939 DEBUG:boto:Token: None 2023-10-17T18:25:33.939 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/ 2023-10-17T18:25:33.940 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4fo27Jwa85xPlxv6LYFaaUt/OxI= 2023-10-17T18:25:33.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4fo27Jwa85xPlxv6LYFaaUt/OxI=', 'Content-Length': '0'} 2023-10-17T18:25:33.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002efbd97bf5180555-00652ed19d-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:25:33.944 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.944 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.945 DEBUG:boto:path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.945 DEBUG:boto:auth_path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.945 DEBUG:boto:Method: GET 2023-10-17T18:25:33.946 DEBUG:boto:Path: /hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.946 DEBUG:boto:Data: 2023-10-17T18:25:33.947 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.947 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.947 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.948 DEBUG:boto:Params: {} 2023-10-17T18:25:33.948 DEBUG:boto:Token: None 2023-10-17T18:25:33.949 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/?versions 2023-10-17T18:25:33.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc= 2023-10-17T18:25:33.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc=', 'Content-Length': '0'} 2023-10-17T18:25:33.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b31c362135ea1fd-00652ed19d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.952 DEBUG:boto:b'hbmuog-18999falseobjnulltrue2023-10-17T18:24:58.331Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:25:33.953 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:25:33.954 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.954 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.955 DEBUG:boto:path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.955 DEBUG:boto:auth_path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.955 DEBUG:boto:Method: GET 2023-10-17T18:25:33.956 DEBUG:boto:Path: /hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.956 DEBUG:boto:Data: 2023-10-17T18:25:33.957 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.957 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.957 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.958 DEBUG:boto:Params: {} 2023-10-17T18:25:33.958 DEBUG:boto:Token: None 2023-10-17T18:25:33.959 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/?versions 2023-10-17T18:25:33.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc= 2023-10-17T18:25:33.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc=', 'Content-Length': '0'} 2023-10-17T18:25:33.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54b13f0f2eb7186-00652ed19d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.962 DEBUG:boto:b'hbmuog-18999falseobjnulltrue2023-10-17T18:24:58.331Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-17T18:25:33.963 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.964 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.964 DEBUG:boto:path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.964 DEBUG:boto:auth_path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.965 DEBUG:boto:Method: GET 2023-10-17T18:25:33.965 DEBUG:boto:Path: /hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.966 DEBUG:boto:Data: 2023-10-17T18:25:33.966 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.967 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.967 DEBUG:boto:Params: {} 2023-10-17T18:25:33.968 DEBUG:boto:Token: None 2023-10-17T18:25:33.968 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/?versions 2023-10-17T18:25:33.969 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc= 2023-10-17T18:25:33.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc=', 'Content-Length': '0'} 2023-10-17T18:25:33.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d9d1cbd3526f64-00652ed19d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.972 DEBUG:boto:b'hbmuog-18999falseobjnulltrue2023-10-17T18:24:58.331Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.972 DEBUG:boto:path=/hbmuog-18/ 2023-10-17T18:25:33.973 DEBUG:boto:auth_path=/hbmuog-18/ 2023-10-17T18:25:33.973 DEBUG:boto:path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.974 DEBUG:boto:auth_path=/hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.974 DEBUG:boto:Method: GET 2023-10-17T18:25:33.974 DEBUG:boto:Path: /hbmuog-18/?versions&max-keys=999 2023-10-17T18:25:33.975 DEBUG:boto:Data: 2023-10-17T18:25:33.975 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.975 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.976 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.976 DEBUG:boto:Params: {} 2023-10-17T18:25:33.977 DEBUG:boto:Token: None 2023-10-17T18:25:33.977 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/?versions 2023-10-17T18:25:33.978 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc= 2023-10-17T18:25:33.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yu9H29xVluznUBs0/mBjUzq9yOc=', 'Content-Length': '0'} 2023-10-17T18:25:33.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc23748506e39235-00652ed19d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.981 DEBUG:boto:b'hbmuog-18999falseobjnulltrue2023-10-17T18:24:58.331Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:25:33.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-17T18:25:33.982 DEBUG:boto:path=/hbmuog-18/obj 2023-10-17T18:25:33.982 DEBUG:boto:auth_path=/hbmuog-18/obj 2023-10-17T18:25:33.982 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.983 DEBUG:boto:Path: /hbmuog-18/obj 2023-10-17T18:25:33.983 DEBUG:boto:Data: 2023-10-17T18:25:33.984 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.984 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:33.984 DEBUG:boto:Port: 8001 2023-10-17T18:25:33.985 DEBUG:boto:Params: {} 2023-10-17T18:25:33.985 DEBUG:boto:Token: None 2023-10-17T18:25:33.985 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/obj 2023-10-17T18:25:33.986 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hSZ0CObnYoeurc1GlyAB5DHj5qc= 2023-10-17T18:25:33.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hSZ0CObnYoeurc1GlyAB5DHj5qc=', 'Content-Length': '0'} 2023-10-17T18:25:33.988 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009ac2f055fd57e525-00652ed19d-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.988 DEBUG:boto:path=/hbmuog-18/obj 2023-10-17T18:25:33.989 DEBUG:boto:auth_path=/hbmuog-18/obj 2023-10-17T18:25:33.989 DEBUG:boto:Method: HEAD 2023-10-17T18:25:33.990 DEBUG:boto:Path: /hbmuog-18/obj 2023-10-17T18:25:33.990 DEBUG:boto:Data: 2023-10-17T18:25:33.991 DEBUG:boto:Headers: {} 2023-10-17T18:25:33.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:33.991 DEBUG:boto:Port: 8000 2023-10-17T18:25:33.992 DEBUG:boto:Params: {} 2023-10-17T18:25:33.992 DEBUG:boto:Token: None 2023-10-17T18:25:33.993 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:33 GMT /hbmuog-18/obj 2023-10-17T18:25:33.993 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hSZ0CObnYoeurc1GlyAB5DHj5qc= 2023-10-17T18:25:33.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hSZ0CObnYoeurc1GlyAB5DHj5qc=', 'Content-Length': '0'} 2023-10-17T18:25:33.995 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004f6b207ac697ac9d-00652ed19d-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:33.996 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-18 zones={a2, a1} 2023-10-17T18:25:33.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-10-17T18:25:33.997 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-19 2023-10-17T18:25:33.998 DEBUG:boto:path=/hbmuog-19/ 2023-10-17T18:25:33.998 DEBUG:boto:auth_path=/hbmuog-19/ 2023-10-17T18:25:33.998 DEBUG:boto:Method: PUT 2023-10-17T18:25:33.999 DEBUG:boto:Path: /hbmuog-19/ 2023-10-17T18:25:33.999 DEBUG:boto:Data: 2023-10-17T18:25:34.000 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.000 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.000 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.001 DEBUG:boto:Params: {} 2023-10-17T18:25:34.001 DEBUG:boto:Token: None 2023-10-17T18:25:34.002 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-19/ 2023-10-17T18:25:34.002 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SlfCWjhH0DiQDfxcfct56uD8/Nc= 2023-10-17T18:25:34.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SlfCWjhH0DiQDfxcfct56uD8/Nc=', 'Content-Length': '0'} 2023-10-17T18:25:34.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e6d6244c4ea24fe-00652ed19e-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.016 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-20 2023-10-17T18:25:34.016 DEBUG:boto:path=/hbmuog-20/ 2023-10-17T18:25:34.016 DEBUG:boto:auth_path=/hbmuog-20/ 2023-10-17T18:25:34.017 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.017 DEBUG:boto:Path: /hbmuog-20/ 2023-10-17T18:25:34.017 DEBUG:boto:Data: 2023-10-17T18:25:34.018 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.019 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.019 DEBUG:boto:Params: {} 2023-10-17T18:25:34.020 DEBUG:boto:Token: None 2023-10-17T18:25:34.020 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-20/ 2023-10-17T18:25:34.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BM+Pc8e9Kc43tlg8fhB0R5fNV9g= 2023-10-17T18:25:34.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BM+Pc8e9Kc43tlg8fhB0R5fNV9g=', 'Content-Length': '0'} 2023-10-17T18:25:34.042 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f78238bd2eb12f4b-00652ed19e-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.042 DEBUG:boto:Using access key provided by client. 2023-10-17T18:25:34.043 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:25:34.044 DEBUG:boto:Using access key provided by client. 2023-10-17T18:25:34.044 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:25:34.045 DEBUG:boto:Using access key provided by client. 2023-10-17T18:25:34.045 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:25:34.046 DEBUG:boto:Using access key provided by client. 2023-10-17T18:25:34.047 DEBUG:boto:Using secret key provided by client. 2023-10-17T18:25:34.047 INFO:rgw_multi.tests:create bucket zone=b1 name=hbmuog-21 2023-10-17T18:25:34.048 DEBUG:boto:path=/hbmuog-21/ 2023-10-17T18:25:34.048 DEBUG:boto:auth_path=/hbmuog-21/ 2023-10-17T18:25:34.048 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.049 DEBUG:boto:Path: /hbmuog-21/ 2023-10-17T18:25:34.049 DEBUG:boto:Data: 2023-10-17T18:25:34.049 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.050 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.050 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.051 DEBUG:boto:Params: {} 2023-10-17T18:25:34.051 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:25:34.052 DEBUG:boto:Token: None 2023-10-17T18:25:34.052 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-21/ 2023-10-17T18:25:34.052 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MPnP7XWgD9dBq7h6F1Q9AmQ45H4= 2023-10-17T18:25:34.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MPnP7XWgD9dBq7h6F1Q9AmQ45H4=', 'Content-Length': '0'} 2023-10-17T18:25:34.085 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cfa675063416453-00652ed19e-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.085 INFO:rgw_multi.tests:create bucket zone=b2 name=hbmuog-22 2023-10-17T18:25:34.086 DEBUG:boto:path=/hbmuog-22/ 2023-10-17T18:25:34.086 DEBUG:boto:auth_path=/hbmuog-22/ 2023-10-17T18:25:34.086 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.087 DEBUG:boto:Path: /hbmuog-22/ 2023-10-17T18:25:34.087 DEBUG:boto:Data: 2023-10-17T18:25:34.088 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.088 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.088 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.089 DEBUG:boto:Params: {} 2023-10-17T18:25:34.089 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:25:34.090 DEBUG:boto:Token: None 2023-10-17T18:25:34.090 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-22/ 2023-10-17T18:25:34.091 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wwurNrx7i3s9oGvToKD4uCI1Ras= 2023-10-17T18:25:34.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wwurNrx7i3s9oGvToKD4uCI1Ras=', 'Content-Length': '0'} 2023-10-17T18:25:34.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000535be6d69fb1d99a-00652ed19e-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.121 DEBUG:boto:path=/hbmuog-19/ 2023-10-17T18:25:34.122 DEBUG:boto:auth_path=/hbmuog-19/ 2023-10-17T18:25:34.122 DEBUG:boto:path=/hbmuog-19/?versioning 2023-10-17T18:25:34.123 DEBUG:boto:auth_path=/hbmuog-19/?versioning 2023-10-17T18:25:34.123 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.123 DEBUG:boto:Path: /hbmuog-19/?versioning 2023-10-17T18:25:34.124 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:25:34.124 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.124 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.125 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.125 DEBUG:boto:Params: {} 2023-10-17T18:25:34.126 DEBUG:boto:Token: None 2023-10-17T18:25:34.126 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-19/?versioning 2023-10-17T18:25:34.126 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rQ4elTA9k2Zs2DWg1GkAxhjstGc= 2023-10-17T18:25:34.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rQ4elTA9k2Zs2DWg1GkAxhjstGc=', 'Content-Length': '228'} 2023-10-17T18:25:34.132 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f53c6040c8bfcff-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.132 DEBUG:boto:path=/hbmuog-19/ 2023-10-17T18:25:34.132 DEBUG:boto:auth_path=/hbmuog-19/ 2023-10-17T18:25:34.133 DEBUG:boto:path=/hbmuog-19/?versioning 2023-10-17T18:25:34.133 DEBUG:boto:auth_path=/hbmuog-19/?versioning 2023-10-17T18:25:34.133 DEBUG:boto:Method: GET 2023-10-17T18:25:34.134 DEBUG:boto:Path: /hbmuog-19/?versioning 2023-10-17T18:25:34.134 DEBUG:boto:Data: 2023-10-17T18:25:34.135 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.135 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.136 DEBUG:boto:Params: {} 2023-10-17T18:25:34.136 DEBUG:boto:Token: None 2023-10-17T18:25:34.136 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-19/?versioning 2023-10-17T18:25:34.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kjRf3RnSOzxn0HXDRyf4IdR9jic= 2023-10-17T18:25:34.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kjRf3RnSOzxn0HXDRyf4IdR9jic=', 'Content-Length': '0'} 2023-10-17T18:25:34.139 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000948bbed4730e4ad6-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.139 DEBUG:boto:EnabledDisabled 2023-10-17T18:25:34.140 DEBUG:boto:path=/hbmuog-20/ 2023-10-17T18:25:34.140 DEBUG:boto:auth_path=/hbmuog-20/ 2023-10-17T18:25:34.141 DEBUG:boto:path=/hbmuog-20/?versioning 2023-10-17T18:25:34.141 DEBUG:boto:auth_path=/hbmuog-20/?versioning 2023-10-17T18:25:34.141 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.142 DEBUG:boto:Path: /hbmuog-20/?versioning 2023-10-17T18:25:34.142 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:25:34.143 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.143 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.144 DEBUG:boto:Params: {} 2023-10-17T18:25:34.144 DEBUG:boto:Token: None 2023-10-17T18:25:34.145 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-20/?versioning 2023-10-17T18:25:34.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hz3zDSb8qGS4AbuT/K/EjEnErUw= 2023-10-17T18:25:34.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hz3zDSb8qGS4AbuT/K/EjEnErUw=', 'Content-Length': '228'} 2023-10-17T18:25:34.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c37df0c9f4c3a7b-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.153 DEBUG:boto:path=/hbmuog-20/ 2023-10-17T18:25:34.153 DEBUG:boto:auth_path=/hbmuog-20/ 2023-10-17T18:25:34.153 DEBUG:boto:path=/hbmuog-20/?versioning 2023-10-17T18:25:34.154 DEBUG:boto:auth_path=/hbmuog-20/?versioning 2023-10-17T18:25:34.154 DEBUG:boto:Method: GET 2023-10-17T18:25:34.155 DEBUG:boto:Path: /hbmuog-20/?versioning 2023-10-17T18:25:34.155 DEBUG:boto:Data: 2023-10-17T18:25:34.155 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.156 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.156 DEBUG:boto:Params: {} 2023-10-17T18:25:34.157 DEBUG:boto:Token: None 2023-10-17T18:25:34.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-20/?versioning 2023-10-17T18:25:34.158 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:behxhOsCdsVf8si82DULFQ+XlDs= 2023-10-17T18:25:34.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:behxhOsCdsVf8si82DULFQ+XlDs=', 'Content-Length': '0'} 2023-10-17T18:25:34.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000385357deee0f4088-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.160 DEBUG:boto:EnabledDisabled 2023-10-17T18:25:34.160 DEBUG:boto:path=/hbmuog-21/ 2023-10-17T18:25:34.161 DEBUG:boto:auth_path=/hbmuog-21/ 2023-10-17T18:25:34.161 DEBUG:boto:path=/hbmuog-21/?versioning 2023-10-17T18:25:34.161 DEBUG:boto:auth_path=/hbmuog-21/?versioning 2023-10-17T18:25:34.162 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.162 DEBUG:boto:Path: /hbmuog-21/?versioning 2023-10-17T18:25:34.162 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:25:34.163 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.163 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.164 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.164 DEBUG:boto:Params: {} 2023-10-17T18:25:34.164 DEBUG:boto:Token: None 2023-10-17T18:25:34.165 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-21/?versioning 2023-10-17T18:25:34.165 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lxz9tiKbpOZe1l9mGIB6vsrYRmI= 2023-10-17T18:25:34.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lxz9tiKbpOZe1l9mGIB6vsrYRmI=', 'Content-Length': '228'} 2023-10-17T18:25:34.173 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b3e5940c7b26d62-00652ed19e-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.174 DEBUG:boto:path=/hbmuog-21/ 2023-10-17T18:25:34.174 DEBUG:boto:auth_path=/hbmuog-21/ 2023-10-17T18:25:34.175 DEBUG:boto:path=/hbmuog-21/?versioning 2023-10-17T18:25:34.175 DEBUG:boto:auth_path=/hbmuog-21/?versioning 2023-10-17T18:25:34.175 DEBUG:boto:Method: GET 2023-10-17T18:25:34.176 DEBUG:boto:Path: /hbmuog-21/?versioning 2023-10-17T18:25:34.176 DEBUG:boto:Data: 2023-10-17T18:25:34.177 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.177 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.177 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.178 DEBUG:boto:Params: {} 2023-10-17T18:25:34.178 DEBUG:boto:Token: None 2023-10-17T18:25:34.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-21/?versioning 2023-10-17T18:25:34.179 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8wqTc+v6RCOCEq6JD7DyaP01Gn0= 2023-10-17T18:25:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8wqTc+v6RCOCEq6JD7DyaP01Gn0=', 'Content-Length': '0'} 2023-10-17T18:25:34.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008fbdf120bf4db3e0-00652ed19e-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.181 DEBUG:boto:EnabledDisabled 2023-10-17T18:25:34.182 DEBUG:boto:path=/hbmuog-22/ 2023-10-17T18:25:34.182 DEBUG:boto:auth_path=/hbmuog-22/ 2023-10-17T18:25:34.183 DEBUG:boto:path=/hbmuog-22/?versioning 2023-10-17T18:25:34.183 DEBUG:boto:auth_path=/hbmuog-22/?versioning 2023-10-17T18:25:34.183 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.184 DEBUG:boto:Path: /hbmuog-22/?versioning 2023-10-17T18:25:34.184 DEBUG:boto:Data: Enabled Disabled 2023-10-17T18:25:34.184 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.185 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.185 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.185 DEBUG:boto:Params: {} 2023-10-17T18:25:34.186 DEBUG:boto:Token: None 2023-10-17T18:25:34.186 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-22/?versioning 2023-10-17T18:25:34.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qMyZjjlexdElbq5dIglIdqitPSE= 2023-10-17T18:25:34.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qMyZjjlexdElbq5dIglIdqitPSE=', 'Content-Length': '228'} 2023-10-17T18:25:34.195 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f6fb3c79f4a663e-00652ed19e-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.195 DEBUG:boto:path=/hbmuog-22/ 2023-10-17T18:25:34.196 DEBUG:boto:auth_path=/hbmuog-22/ 2023-10-17T18:25:34.196 DEBUG:boto:path=/hbmuog-22/?versioning 2023-10-17T18:25:34.197 DEBUG:boto:auth_path=/hbmuog-22/?versioning 2023-10-17T18:25:34.197 DEBUG:boto:Method: GET 2023-10-17T18:25:34.197 DEBUG:boto:Path: /hbmuog-22/?versioning 2023-10-17T18:25:34.198 DEBUG:boto:Data: 2023-10-17T18:25:34.198 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.198 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.199 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.199 DEBUG:boto:Params: {} 2023-10-17T18:25:34.200 DEBUG:boto:Token: None 2023-10-17T18:25:34.200 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-22/?versioning 2023-10-17T18:25:34.200 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rIuj9/q/kF/dxK3I7WMiowq1gP0= 2023-10-17T18:25:34.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rIuj9/q/kF/dxK3I7WMiowq1gP0=', 'Content-Length': '0'} 2023-10-17T18:25:34.202 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000514090f886da6b45-00652ed19e-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.203 DEBUG:boto:EnabledDisabled 2023-10-17T18:25:34.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-10-17T18:25:34.204 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-23 2023-10-17T18:25:34.204 DEBUG:boto:path=/hbmuog-23/ 2023-10-17T18:25:34.204 DEBUG:boto:auth_path=/hbmuog-23/ 2023-10-17T18:25:34.205 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.205 DEBUG:boto:Path: /hbmuog-23/ 2023-10-17T18:25:34.206 DEBUG:boto:Data: 2023-10-17T18:25:34.206 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.207 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.207 DEBUG:boto:Params: {} 2023-10-17T18:25:34.208 DEBUG:boto:Token: None 2023-10-17T18:25:34.208 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-23/ 2023-10-17T18:25:34.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IbdwyTpbofyiDZ5g5y8DGTw2faI= 2023-10-17T18:25:34.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IbdwyTpbofyiDZ5g5y8DGTw2faI=', 'Content-Length': '0'} 2023-10-17T18:25:34.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c9052ed9c2821bf-00652ed19e-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.222 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-24 2023-10-17T18:25:34.223 DEBUG:boto:path=/hbmuog-24/ 2023-10-17T18:25:34.223 DEBUG:boto:auth_path=/hbmuog-24/ 2023-10-17T18:25:34.223 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.224 DEBUG:boto:Path: /hbmuog-24/ 2023-10-17T18:25:34.224 DEBUG:boto:Data: 2023-10-17T18:25:34.224 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.225 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.226 DEBUG:boto:Params: {} 2023-10-17T18:25:34.226 DEBUG:boto:Token: None 2023-10-17T18:25:34.226 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-24/ 2023-10-17T18:25:34.227 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q7c8UXvqf7FDl5UDMBQ3DwXRR44= 2023-10-17T18:25:34.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q7c8UXvqf7FDl5UDMBQ3DwXRR44=', 'Content-Length': '0'} 2023-10-17T18:25:34.249 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009cef0c4d3067a36b-00652ed19e-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.250 INFO:rgw_multi.tests:create bucket zone=b1 name=hbmuog-25 2023-10-17T18:25:34.250 DEBUG:boto:path=/hbmuog-25/ 2023-10-17T18:25:34.251 DEBUG:boto:auth_path=/hbmuog-25/ 2023-10-17T18:25:34.251 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.251 DEBUG:boto:Path: /hbmuog-25/ 2023-10-17T18:25:34.252 DEBUG:boto:Data: 2023-10-17T18:25:34.252 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.252 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.253 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.253 DEBUG:boto:Params: {} 2023-10-17T18:25:34.254 DEBUG:boto:Token: None 2023-10-17T18:25:34.254 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-25/ 2023-10-17T18:25:34.254 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MXNTGH2NKOR5OJRH/uUJEKRWiXQ= 2023-10-17T18:25:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MXNTGH2NKOR5OJRH/uUJEKRWiXQ=', 'Content-Length': '0'} 2023-10-17T18:25:34.284 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a9ad9a7ae24a1f9-00652ed19e-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.284 INFO:rgw_multi.tests:create bucket zone=b2 name=hbmuog-26 2023-10-17T18:25:34.285 DEBUG:boto:path=/hbmuog-26/ 2023-10-17T18:25:34.285 DEBUG:boto:auth_path=/hbmuog-26/ 2023-10-17T18:25:34.286 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.286 DEBUG:boto:Path: /hbmuog-26/ 2023-10-17T18:25:34.286 DEBUG:boto:Data: 2023-10-17T18:25:34.287 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.287 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.288 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.288 DEBUG:boto:Params: {} 2023-10-17T18:25:34.288 DEBUG:boto:Token: None 2023-10-17T18:25:34.289 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-26/ 2023-10-17T18:25:34.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9Nsi6+Rn+e5thjALmt2KnG7t7lQ= 2023-10-17T18:25:34.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9Nsi6+Rn+e5thjALmt2KnG7t7lQ=', 'Content-Length': '0'} 2023-10-17T18:25:34.313 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003e193e265e1a6d9f-00652ed19e-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.314 DEBUG:boto:path=/hbmuog-23/ 2023-10-17T18:25:34.314 DEBUG:boto:auth_path=/hbmuog-23/ 2023-10-17T18:25:34.315 DEBUG:boto:path=/hbmuog-23/?acl 2023-10-17T18:25:34.315 DEBUG:boto:auth_path=/hbmuog-23/?acl 2023-10-17T18:25:34.315 DEBUG:boto:Method: GET 2023-10-17T18:25:34.316 DEBUG:boto:Path: /hbmuog-23/?acl 2023-10-17T18:25:34.316 DEBUG:boto:Data: 2023-10-17T18:25:34.316 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.317 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.318 DEBUG:boto:Params: {} 2023-10-17T18:25:34.318 DEBUG:boto:Token: None 2023-10-17T18:25:34.318 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-23/?acl 2023-10-17T18:25:34.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B6/qOfH/FtyjfoIZtd3WkbcOwU8= 2023-10-17T18:25:34.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B6/qOfH/FtyjfoIZtd3WkbcOwU8=', 'Content-Length': '0'} 2023-10-17T18:25:34.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000855677844a296c77-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.321 DEBUG:boto:path=/hbmuog-23/ 2023-10-17T18:25:34.322 DEBUG:boto:auth_path=/hbmuog-23/ 2023-10-17T18:25:34.322 DEBUG:boto:path=/hbmuog-23/?acl 2023-10-17T18:25:34.323 DEBUG:boto:auth_path=/hbmuog-23/?acl 2023-10-17T18:25:34.323 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.323 DEBUG:boto:Path: /hbmuog-23/?acl 2023-10-17T18:25:34.324 DEBUG:boto:Data: 2023-10-17T18:25:34.324 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:25:34.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.325 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.325 DEBUG:boto:Params: {} 2023-10-17T18:25:34.326 DEBUG:boto:Token: None 2023-10-17T18:25:34.326 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT x-amz-acl:public-read /hbmuog-23/?acl 2023-10-17T18:25:34.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NBtSYSnSMBrDwvfOgV4SDPEX9yE= 2023-10-17T18:25:34.327 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NBtSYSnSMBrDwvfOgV4SDPEX9yE=', 'Content-Length': '0'} 2023-10-17T18:25:34.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000850d04f39f4e5ce7-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.332 DEBUG:boto:path=/hbmuog-23/ 2023-10-17T18:25:34.332 DEBUG:boto:auth_path=/hbmuog-23/ 2023-10-17T18:25:34.333 DEBUG:boto:path=/hbmuog-23/?acl 2023-10-17T18:25:34.333 DEBUG:boto:auth_path=/hbmuog-23/?acl 2023-10-17T18:25:34.334 DEBUG:boto:Method: GET 2023-10-17T18:25:34.334 DEBUG:boto:Path: /hbmuog-23/?acl 2023-10-17T18:25:34.334 DEBUG:boto:Data: 2023-10-17T18:25:34.335 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.335 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.336 DEBUG:boto:Params: {} 2023-10-17T18:25:34.336 DEBUG:boto:Token: None 2023-10-17T18:25:34.337 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-23/?acl 2023-10-17T18:25:34.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B6/qOfH/FtyjfoIZtd3WkbcOwU8= 2023-10-17T18:25:34.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B6/qOfH/FtyjfoIZtd3WkbcOwU8=', 'Content-Length': '0'} 2023-10-17T18:25:34.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000941374ad4256e615-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.386 DEBUG:boto:path=/hbmuog-24/ 2023-10-17T18:25:34.387 DEBUG:boto:auth_path=/hbmuog-24/ 2023-10-17T18:25:34.387 DEBUG:boto:path=/hbmuog-24/?acl 2023-10-17T18:25:34.388 DEBUG:boto:auth_path=/hbmuog-24/?acl 2023-10-17T18:25:34.388 DEBUG:boto:Method: GET 2023-10-17T18:25:34.388 DEBUG:boto:Path: /hbmuog-24/?acl 2023-10-17T18:25:34.389 DEBUG:boto:Data: 2023-10-17T18:25:34.389 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.390 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.390 DEBUG:boto:Params: {} 2023-10-17T18:25:34.391 DEBUG:boto:Token: None 2023-10-17T18:25:34.391 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-24/?acl 2023-10-17T18:25:34.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:axkkBpjJe0Hj41EHBaoyd4TDe0I= 2023-10-17T18:25:34.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:axkkBpjJe0Hj41EHBaoyd4TDe0I=', 'Content-Length': '0'} 2023-10-17T18:25:34.393 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086b3581635ed0e11-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.394 DEBUG:boto:path=/hbmuog-24/ 2023-10-17T18:25:34.394 DEBUG:boto:auth_path=/hbmuog-24/ 2023-10-17T18:25:34.395 DEBUG:boto:path=/hbmuog-24/?acl 2023-10-17T18:25:34.395 DEBUG:boto:auth_path=/hbmuog-24/?acl 2023-10-17T18:25:34.396 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.396 DEBUG:boto:Path: /hbmuog-24/?acl 2023-10-17T18:25:34.396 DEBUG:boto:Data: 2023-10-17T18:25:34.397 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:25:34.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.398 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.398 DEBUG:boto:Params: {} 2023-10-17T18:25:34.398 DEBUG:boto:Token: None 2023-10-17T18:25:34.399 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT x-amz-acl:public-read /hbmuog-24/?acl 2023-10-17T18:25:34.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eYF9eEtqwlfHoT6vIsBy3DnwLTM= 2023-10-17T18:25:34.400 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eYF9eEtqwlfHoT6vIsBy3DnwLTM=', 'Content-Length': '0'} 2023-10-17T18:25:34.407 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000067f38964abb5d76b-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.407 DEBUG:boto:path=/hbmuog-24/ 2023-10-17T18:25:34.408 DEBUG:boto:auth_path=/hbmuog-24/ 2023-10-17T18:25:34.408 DEBUG:boto:path=/hbmuog-24/?acl 2023-10-17T18:25:34.409 DEBUG:boto:auth_path=/hbmuog-24/?acl 2023-10-17T18:25:34.409 DEBUG:boto:Method: GET 2023-10-17T18:25:34.409 DEBUG:boto:Path: /hbmuog-24/?acl 2023-10-17T18:25:34.410 DEBUG:boto:Data: 2023-10-17T18:25:34.410 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.411 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.411 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.411 DEBUG:boto:Params: {} 2023-10-17T18:25:34.412 DEBUG:boto:Token: None 2023-10-17T18:25:34.412 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-24/?acl 2023-10-17T18:25:34.413 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:axkkBpjJe0Hj41EHBaoyd4TDe0I= 2023-10-17T18:25:34.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:axkkBpjJe0Hj41EHBaoyd4TDe0I=', 'Content-Length': '0'} 2023-10-17T18:25:34.415 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e19cf51c88f4101-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.415 DEBUG:boto:path=/hbmuog-25/ 2023-10-17T18:25:34.416 DEBUG:boto:auth_path=/hbmuog-25/ 2023-10-17T18:25:34.416 DEBUG:boto:path=/hbmuog-25/?acl 2023-10-17T18:25:34.417 DEBUG:boto:auth_path=/hbmuog-25/?acl 2023-10-17T18:25:34.417 DEBUG:boto:Method: GET 2023-10-17T18:25:34.417 DEBUG:boto:Path: /hbmuog-25/?acl 2023-10-17T18:25:34.418 DEBUG:boto:Data: 2023-10-17T18:25:34.418 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.419 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.419 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.420 DEBUG:boto:Params: {} 2023-10-17T18:25:34.420 DEBUG:boto:Token: None 2023-10-17T18:25:34.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-25/?acl 2023-10-17T18:25:34.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/dpH/xN3N0Xrc8U3Q8qnw+pUeL4= 2023-10-17T18:25:34.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/dpH/xN3N0Xrc8U3Q8qnw+pUeL4=', 'Content-Length': '0'} 2023-10-17T18:25:34.423 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c0f00a469aa86245-00652ed19e-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.424 DEBUG:boto:path=/hbmuog-25/ 2023-10-17T18:25:34.424 DEBUG:boto:auth_path=/hbmuog-25/ 2023-10-17T18:25:34.424 DEBUG:boto:path=/hbmuog-25/?acl 2023-10-17T18:25:34.425 DEBUG:boto:auth_path=/hbmuog-25/?acl 2023-10-17T18:25:34.425 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.426 DEBUG:boto:Path: /hbmuog-25/?acl 2023-10-17T18:25:34.426 DEBUG:boto:Data: 2023-10-17T18:25:34.429 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:25:34.429 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.430 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.430 DEBUG:boto:Params: {} 2023-10-17T18:25:34.431 DEBUG:boto:Token: None 2023-10-17T18:25:34.431 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT x-amz-acl:public-read /hbmuog-25/?acl 2023-10-17T18:25:34.432 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pO6jrtqIAT3Zlg9rYUhtERdczIQ= 2023-10-17T18:25:34.432 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pO6jrtqIAT3Zlg9rYUhtERdczIQ=', 'Content-Length': '0'} 2023-10-17T18:25:34.441 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7d6b013e02ca013-00652ed19e-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.441 DEBUG:boto:path=/hbmuog-25/ 2023-10-17T18:25:34.442 DEBUG:boto:auth_path=/hbmuog-25/ 2023-10-17T18:25:34.442 DEBUG:boto:path=/hbmuog-25/?acl 2023-10-17T18:25:34.442 DEBUG:boto:auth_path=/hbmuog-25/?acl 2023-10-17T18:25:34.443 DEBUG:boto:Method: GET 2023-10-17T18:25:34.443 DEBUG:boto:Path: /hbmuog-25/?acl 2023-10-17T18:25:34.444 DEBUG:boto:Data: 2023-10-17T18:25:34.444 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.444 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.445 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.445 DEBUG:boto:Params: {} 2023-10-17T18:25:34.446 DEBUG:boto:Token: None 2023-10-17T18:25:34.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-25/?acl 2023-10-17T18:25:34.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/dpH/xN3N0Xrc8U3Q8qnw+pUeL4= 2023-10-17T18:25:34.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/dpH/xN3N0Xrc8U3Q8qnw+pUeL4=', 'Content-Length': '0'} 2023-10-17T18:25:34.448 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075a15f51146dc150-00652ed19e-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.490 DEBUG:boto:path=/hbmuog-26/ 2023-10-17T18:25:34.491 DEBUG:boto:auth_path=/hbmuog-26/ 2023-10-17T18:25:34.491 DEBUG:boto:path=/hbmuog-26/?acl 2023-10-17T18:25:34.492 DEBUG:boto:auth_path=/hbmuog-26/?acl 2023-10-17T18:25:34.492 DEBUG:boto:Method: GET 2023-10-17T18:25:34.492 DEBUG:boto:Path: /hbmuog-26/?acl 2023-10-17T18:25:34.493 DEBUG:boto:Data: 2023-10-17T18:25:34.493 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.494 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.494 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.494 DEBUG:boto:Params: {} 2023-10-17T18:25:34.495 DEBUG:boto:Token: None 2023-10-17T18:25:34.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-26/?acl 2023-10-17T18:25:34.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ryRudQIF65K2D0d23BPe6KCnYE8= 2023-10-17T18:25:34.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ryRudQIF65K2D0d23BPe6KCnYE8=', 'Content-Length': '0'} 2023-10-17T18:25:34.498 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e8a5f4e22c61fc3-00652ed19e-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.498 DEBUG:boto:path=/hbmuog-26/ 2023-10-17T18:25:34.499 DEBUG:boto:auth_path=/hbmuog-26/ 2023-10-17T18:25:34.499 DEBUG:boto:path=/hbmuog-26/?acl 2023-10-17T18:25:34.500 DEBUG:boto:auth_path=/hbmuog-26/?acl 2023-10-17T18:25:34.500 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.500 DEBUG:boto:Path: /hbmuog-26/?acl 2023-10-17T18:25:34.501 DEBUG:boto:Data: 2023-10-17T18:25:34.501 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-17T18:25:34.502 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.502 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.503 DEBUG:boto:Params: {} 2023-10-17T18:25:34.503 DEBUG:boto:Token: None 2023-10-17T18:25:34.503 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT x-amz-acl:public-read /hbmuog-26/?acl 2023-10-17T18:25:34.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:swvhnxHzXdFD9aWm5hQ0NmCAh+k= 2023-10-17T18:25:34.504 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:swvhnxHzXdFD9aWm5hQ0NmCAh+k=', 'Content-Length': '0'} 2023-10-17T18:25:34.513 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d7d8c4a851675ff-00652ed19e-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.513 DEBUG:boto:path=/hbmuog-26/ 2023-10-17T18:25:34.514 DEBUG:boto:auth_path=/hbmuog-26/ 2023-10-17T18:25:34.514 DEBUG:boto:path=/hbmuog-26/?acl 2023-10-17T18:25:34.514 DEBUG:boto:auth_path=/hbmuog-26/?acl 2023-10-17T18:25:34.515 DEBUG:boto:Method: GET 2023-10-17T18:25:34.515 DEBUG:boto:Path: /hbmuog-26/?acl 2023-10-17T18:25:34.516 DEBUG:boto:Data: 2023-10-17T18:25:34.516 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.516 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.517 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.517 DEBUG:boto:Params: {} 2023-10-17T18:25:34.518 DEBUG:boto:Token: None 2023-10-17T18:25:34.518 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-26/?acl 2023-10-17T18:25:34.518 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ryRudQIF65K2D0d23BPe6KCnYE8= 2023-10-17T18:25:34.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ryRudQIF65K2D0d23BPe6KCnYE8=', 'Content-Length': '0'} 2023-10-17T18:25:34.520 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000907f4a1b67b9de35-00652ed19e-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-10-17T18:25:34.522 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-27 2023-10-17T18:25:34.522 DEBUG:boto:path=/hbmuog-27/ 2023-10-17T18:25:34.522 DEBUG:boto:auth_path=/hbmuog-27/ 2023-10-17T18:25:34.523 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.523 DEBUG:boto:Path: /hbmuog-27/ 2023-10-17T18:25:34.523 DEBUG:boto:Data: 2023-10-17T18:25:34.524 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.524 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.524 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.525 DEBUG:boto:Params: {} 2023-10-17T18:25:34.525 DEBUG:boto:Token: None 2023-10-17T18:25:34.526 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-27/ 2023-10-17T18:25:34.526 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hBvEpjc0EQs2Ple9zLNhmYnWaqE= 2023-10-17T18:25:34.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hBvEpjc0EQs2Ple9zLNhmYnWaqE=', 'Content-Length': '0'} 2023-10-17T18:25:34.539 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006598739e613cf61f-00652ed19e-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.540 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-28 2023-10-17T18:25:34.540 DEBUG:boto:path=/hbmuog-28/ 2023-10-17T18:25:34.540 DEBUG:boto:auth_path=/hbmuog-28/ 2023-10-17T18:25:34.541 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.541 DEBUG:boto:Path: /hbmuog-28/ 2023-10-17T18:25:34.542 DEBUG:boto:Data: 2023-10-17T18:25:34.542 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.542 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.543 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.543 DEBUG:boto:Params: {} 2023-10-17T18:25:34.544 DEBUG:boto:Token: None 2023-10-17T18:25:34.544 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-28/ 2023-10-17T18:25:34.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYZqbu//6m1kLt5F3Jjuqj1avUA= 2023-10-17T18:25:34.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYZqbu//6m1kLt5F3Jjuqj1avUA=', 'Content-Length': '0'} 2023-10-17T18:25:34.566 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a320e3d6fc35bdc-00652ed19e-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.567 INFO:rgw_multi.tests:create bucket zone=b1 name=hbmuog-29 2023-10-17T18:25:34.567 DEBUG:boto:path=/hbmuog-29/ 2023-10-17T18:25:34.567 DEBUG:boto:auth_path=/hbmuog-29/ 2023-10-17T18:25:34.568 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.568 DEBUG:boto:Path: /hbmuog-29/ 2023-10-17T18:25:34.569 DEBUG:boto:Data: 2023-10-17T18:25:34.569 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.570 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:34.570 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.571 DEBUG:boto:Params: {} 2023-10-17T18:25:34.571 DEBUG:boto:Token: None 2023-10-17T18:25:34.571 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-29/ 2023-10-17T18:25:34.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GNdmK2pPFYlqJWHlK7CQoAoNqrc= 2023-10-17T18:25:34.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GNdmK2pPFYlqJWHlK7CQoAoNqrc=', 'Content-Length': '0'} 2023-10-17T18:25:34.597 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000751357fe3da06b66-00652ed19e-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.597 INFO:rgw_multi.tests:create bucket zone=b2 name=hbmuog-30 2023-10-17T18:25:34.598 DEBUG:boto:path=/hbmuog-30/ 2023-10-17T18:25:34.598 DEBUG:boto:auth_path=/hbmuog-30/ 2023-10-17T18:25:34.599 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.599 DEBUG:boto:Path: /hbmuog-30/ 2023-10-17T18:25:34.600 DEBUG:boto:Data: 2023-10-17T18:25:34.600 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.600 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:34.601 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.601 DEBUG:boto:Params: {} 2023-10-17T18:25:34.602 DEBUG:boto:Token: None 2023-10-17T18:25:34.602 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-30/ 2023-10-17T18:25:34.603 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q3IdLhtIEWywaQBkShel2wPK5uM= 2023-10-17T18:25:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q3IdLhtIEWywaQBkShel2wPK5uM=', 'Content-Length': '0'} 2023-10-17T18:25:34.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe4864328c36e6b5-00652ed19e-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.627 DEBUG:boto:path=/hbmuog-27/ 2023-10-17T18:25:34.628 DEBUG:boto:auth_path=/hbmuog-27/ 2023-10-17T18:25:34.628 DEBUG:boto:path=/hbmuog-27/?cors 2023-10-17T18:25:34.629 DEBUG:boto:auth_path=/hbmuog-27/?cors 2023-10-17T18:25:34.629 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.629 DEBUG:boto:Path: /hbmuog-27/?cors 2023-10-17T18:25:34.630 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-17T18:25:34.630 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-17T18:25:34.631 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.631 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.631 DEBUG:boto:Params: {} 2023-10-17T18:25:34.632 DEBUG:boto:Token: None 2023-10-17T18:25:34.632 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-27/?cors 2023-10-17T18:25:34.633 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JNdPjEKqFdoJEJuAcIKqU+l2/zU= 2023-10-17T18:25:34.633 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JNdPjEKqFdoJEJuAcIKqU+l2/zU=', 'Content-Length': '218'} 2023-10-17T18:25:34.642 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000093db4f93f72061a-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.643 DEBUG:boto:path=/hbmuog-27/ 2023-10-17T18:25:34.643 DEBUG:boto:auth_path=/hbmuog-27/ 2023-10-17T18:25:34.644 DEBUG:boto:path=/hbmuog-27/?cors 2023-10-17T18:25:34.644 DEBUG:boto:auth_path=/hbmuog-27/?cors 2023-10-17T18:25:34.645 DEBUG:boto:Method: GET 2023-10-17T18:25:34.645 DEBUG:boto:Path: /hbmuog-27/?cors 2023-10-17T18:25:34.645 DEBUG:boto:Data: 2023-10-17T18:25:34.646 DEBUG:boto:Headers: {} 2023-10-17T18:25:34.646 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:34.647 DEBUG:boto:Port: 8000 2023-10-17T18:25:34.647 DEBUG:boto:Params: {} 2023-10-17T18:25:34.647 DEBUG:boto:Token: None 2023-10-17T18:25:34.648 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-27/?cors 2023-10-17T18:25:34.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m1AGPqg9+QqHyDwDq75dvvVfuUQ= 2023-10-17T18:25:34.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m1AGPqg9+QqHyDwDq75dvvVfuUQ=', 'Content-Length': '0'} 2023-10-17T18:25:34.650 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7b8ff8fc6435944-00652ed19e-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 17 Oct 2023 18:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:34.650 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-17T18:25:34.651 DEBUG:boto:path=/hbmuog-28/ 2023-10-17T18:25:34.651 DEBUG:boto:auth_path=/hbmuog-28/ 2023-10-17T18:25:34.652 DEBUG:boto:path=/hbmuog-28/?cors 2023-10-17T18:25:34.652 DEBUG:boto:auth_path=/hbmuog-28/?cors 2023-10-17T18:25:34.652 DEBUG:boto:Method: PUT 2023-10-17T18:25:34.653 DEBUG:boto:Path: /hbmuog-28/?cors 2023-10-17T18:25:34.653 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-17T18:25:34.654 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-17T18:25:34.654 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:34.654 DEBUG:boto:Port: 8001 2023-10-17T18:25:34.655 DEBUG:boto:Params: {} 2023-10-17T18:25:34.655 DEBUG:boto:Token: None 2023-10-17T18:25:34.656 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 17 Oct 2023 18:25:34 GMT /hbmuog-28/?cors 2023-10-17T18:25:34.656 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mGWQtIVUNqbFcf8Xj7LpXGjSdlw= 2023-10-17T18:25:34.657 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mGWQtIVUNqbFcf8Xj7LpXGjSdlw=', 'Content-Length': '218'} 2023-10-17T18:25:35.666 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004368e9b82bee02ed-00652ed19e-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:35.666 DEBUG:boto:path=/hbmuog-28/ 2023-10-17T18:25:35.667 DEBUG:boto:auth_path=/hbmuog-28/ 2023-10-17T18:25:35.667 DEBUG:boto:path=/hbmuog-28/?cors 2023-10-17T18:25:35.668 DEBUG:boto:auth_path=/hbmuog-28/?cors 2023-10-17T18:25:35.668 DEBUG:boto:Method: GET 2023-10-17T18:25:35.668 DEBUG:boto:Path: /hbmuog-28/?cors 2023-10-17T18:25:35.669 DEBUG:boto:Data: 2023-10-17T18:25:35.669 DEBUG:boto:Headers: {} 2023-10-17T18:25:35.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:35.670 DEBUG:boto:Port: 8001 2023-10-17T18:25:35.670 DEBUG:boto:Params: {} 2023-10-17T18:25:35.671 DEBUG:boto:Token: None 2023-10-17T18:25:35.671 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:35 GMT /hbmuog-28/?cors 2023-10-17T18:25:35.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h1w9CcajQQ4ziF7c8bdalln+EP0= 2023-10-17T18:25:35.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h1w9CcajQQ4ziF7c8bdalln+EP0=', 'Content-Length': '0'} 2023-10-17T18:25:35.674 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008fcf8657c3be3c7e-00652ed19f-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 17 Oct 2023 18:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:35.718 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-17T18:25:35.719 DEBUG:boto:path=/hbmuog-29/ 2023-10-17T18:25:35.719 DEBUG:boto:auth_path=/hbmuog-29/ 2023-10-17T18:25:35.720 DEBUG:boto:path=/hbmuog-29/?cors 2023-10-17T18:25:35.720 DEBUG:boto:auth_path=/hbmuog-29/?cors 2023-10-17T18:25:35.721 DEBUG:boto:Method: PUT 2023-10-17T18:25:35.721 DEBUG:boto:Path: /hbmuog-29/?cors 2023-10-17T18:25:35.721 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-17T18:25:35.722 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-17T18:25:35.722 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:35.722 DEBUG:boto:Port: 8000 2023-10-17T18:25:35.723 DEBUG:boto:Params: {} 2023-10-17T18:25:35.723 DEBUG:boto:Token: None 2023-10-17T18:25:35.724 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 17 Oct 2023 18:25:35 GMT /hbmuog-29/?cors 2023-10-17T18:25:35.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pc3jR1D+7GVDaOm/JZyiGSfmnCE= 2023-10-17T18:25:35.725 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pc3jR1D+7GVDaOm/JZyiGSfmnCE=', 'Content-Length': '218'} 2023-10-17T18:25:36.735 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b15ed0fd4a5ce4e7-00652ed19f-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:36.736 DEBUG:boto:path=/hbmuog-29/ 2023-10-17T18:25:36.737 DEBUG:boto:auth_path=/hbmuog-29/ 2023-10-17T18:25:36.737 DEBUG:boto:path=/hbmuog-29/?cors 2023-10-17T18:25:36.738 DEBUG:boto:auth_path=/hbmuog-29/?cors 2023-10-17T18:25:36.738 DEBUG:boto:Method: GET 2023-10-17T18:25:36.738 DEBUG:boto:Path: /hbmuog-29/?cors 2023-10-17T18:25:36.739 DEBUG:boto:Data: 2023-10-17T18:25:36.739 DEBUG:boto:Headers: {} 2023-10-17T18:25:36.740 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:25:36.740 DEBUG:boto:Port: 8000 2023-10-17T18:25:36.740 DEBUG:boto:Params: {} 2023-10-17T18:25:36.741 DEBUG:boto:Token: None 2023-10-17T18:25:36.741 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:36 GMT /hbmuog-29/?cors 2023-10-17T18:25:36.742 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gpMF98dCY1S0dXDDnsuDH1McGc8= 2023-10-17T18:25:36.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gpMF98dCY1S0dXDDnsuDH1McGc8=', 'Content-Length': '0'} 2023-10-17T18:25:36.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009046016186f3aa36-00652ed1a0-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 17 Oct 2023 18:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:36.744 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-17T18:25:36.745 DEBUG:boto:path=/hbmuog-30/ 2023-10-17T18:25:36.745 DEBUG:boto:auth_path=/hbmuog-30/ 2023-10-17T18:25:36.746 DEBUG:boto:path=/hbmuog-30/?cors 2023-10-17T18:25:36.746 DEBUG:boto:auth_path=/hbmuog-30/?cors 2023-10-17T18:25:36.747 DEBUG:boto:Method: PUT 2023-10-17T18:25:36.747 DEBUG:boto:Path: /hbmuog-30/?cors 2023-10-17T18:25:36.747 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-17T18:25:36.748 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-17T18:25:36.748 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:36.748 DEBUG:boto:Port: 8001 2023-10-17T18:25:36.749 DEBUG:boto:Params: {} 2023-10-17T18:25:36.749 DEBUG:boto:Token: None 2023-10-17T18:25:36.750 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 17 Oct 2023 18:25:36 GMT /hbmuog-30/?cors 2023-10-17T18:25:36.750 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:un22HEJJlH4SssYuMEsqZjaIph8= 2023-10-17T18:25:36.751 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:un22HEJJlH4SssYuMEsqZjaIph8=', 'Content-Length': '218'} 2023-10-17T18:25:37.762 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c288238de58f076d-00652ed1a0-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:37.762 DEBUG:boto:path=/hbmuog-30/ 2023-10-17T18:25:37.763 DEBUG:boto:auth_path=/hbmuog-30/ 2023-10-17T18:25:37.763 DEBUG:boto:path=/hbmuog-30/?cors 2023-10-17T18:25:37.764 DEBUG:boto:auth_path=/hbmuog-30/?cors 2023-10-17T18:25:37.764 DEBUG:boto:Method: GET 2023-10-17T18:25:37.765 DEBUG:boto:Path: /hbmuog-30/?cors 2023-10-17T18:25:37.765 DEBUG:boto:Data: 2023-10-17T18:25:37.765 DEBUG:boto:Headers: {} 2023-10-17T18:25:37.766 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:25:37.766 DEBUG:boto:Port: 8001 2023-10-17T18:25:37.767 DEBUG:boto:Params: {} 2023-10-17T18:25:37.767 DEBUG:boto:Token: None 2023-10-17T18:25:37.768 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:25:37 GMT /hbmuog-30/?cors 2023-10-17T18:25:37.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lxbZ294aFla05MKgfgs9HBpuSLs= 2023-10-17T18:25:37.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lxbZ294aFla05MKgfgs9HBpuSLs=', 'Content-Length': '0'} 2023-10-17T18:25:37.770 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038ddb36962e5969f-00652ed1a1-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 17 Oct 2023 18:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:37.818 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-17T18:25:37.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-10-17T18:25:37.822 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-31 2023-10-17T18:25:37.823 DEBUG:boto:path=/hbmuog-31/ 2023-10-17T18:25:37.823 DEBUG:boto:auth_path=/hbmuog-31/ 2023-10-17T18:25:37.823 DEBUG:boto:Method: PUT 2023-10-17T18:25:37.824 DEBUG:boto:Path: /hbmuog-31/ 2023-10-17T18:25:37.824 DEBUG:boto:Data: 2023-10-17T18:25:37.825 DEBUG:boto:Headers: {} 2023-10-17T18:25:37.825 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:37.825 DEBUG:boto:Port: 8000 2023-10-17T18:25:37.826 DEBUG:boto:Params: {} 2023-10-17T18:25:37.826 DEBUG:boto:Token: None 2023-10-17T18:25:37.827 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:37 GMT /hbmuog-31/ 2023-10-17T18:25:37.827 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P7MBW/QO1YyqrVxGtt8MSFPIytM= 2023-10-17T18:25:37.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P7MBW/QO1YyqrVxGtt8MSFPIytM=', 'Content-Length': '0'} 2023-10-17T18:25:37.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000201483c2c55b41ec-00652ed1a1-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:37.845 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-32 2023-10-17T18:25:37.845 DEBUG:boto:path=/hbmuog-32/ 2023-10-17T18:25:37.845 DEBUG:boto:auth_path=/hbmuog-32/ 2023-10-17T18:25:37.846 DEBUG:boto:Method: PUT 2023-10-17T18:25:37.846 DEBUG:boto:Path: /hbmuog-32/ 2023-10-17T18:25:37.847 DEBUG:boto:Data: 2023-10-17T18:25:37.847 DEBUG:boto:Headers: {} 2023-10-17T18:25:37.848 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:37.848 DEBUG:boto:Port: 8001 2023-10-17T18:25:37.849 DEBUG:boto:Params: {} 2023-10-17T18:25:37.849 DEBUG:boto:Token: None 2023-10-17T18:25:37.850 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:37 GMT /hbmuog-32/ 2023-10-17T18:25:37.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZjV76p6ZsoQ+hDyNtqQOcTxvKhE= 2023-10-17T18:25:37.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZjV76p6ZsoQ+hDyNtqQOcTxvKhE=', 'Content-Length': '0'} 2023-10-17T18:25:37.885 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5985a715454aeb5-00652ed1a1-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:37.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:37.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:37.887 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:37.943 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:37.943 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:37.977 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:25:37.989 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:25:37.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:37.990 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567137.874580_310.1", 2023-10-17T18:25:37.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:37.874580Z" 2023-10-17T18:25:37.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:37.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:37.991 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:37.991 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:25:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:25:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:37.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:37.993 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:37.993 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:25:37.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:37.994 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:25:37.995 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567137.874580_310.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:37.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:25:37.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:37.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:37.996 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:38.047 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:38.047 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:38.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:38.077+0000 7f4e6ae0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:38.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:38.079+0000 7f4e6ae0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:38.092 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:38.092 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:38.092 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:38.093 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:38.093 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:38.093 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:25:38.093 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:38.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.094 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:38.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:38.094 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:38.094 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567137.861771_307.1", 2023-10-17T18:25:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:25:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:37.861771Z", 2023-10-17T18:25:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:25:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:25:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:25:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:38.107 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:38Z" 2023-10-17T18:25:38.107 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:38.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567137.861771_307.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:37.861771Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:25:38Z" } 2023-10-17T18:25:38.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567137.861771_307.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:37.861771Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:25:38.109 DEBUG:rgw_multi.tests:log_status={0: '1_1697567137.874580_310.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:38.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567137.861771_307.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:38.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567137.874580_310.1 target=1_1697567137.861771_307.1 2023-10-17T18:25:43.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.112 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:43.171 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:43.172 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:43.203+0000 7f7a5583d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:43.204+0000 7f7a5583d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:43.219 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:25:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567137.874580_310.1", 2023-10-17T18:25:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:25:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:37.874580Z", 2023-10-17T18:25:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:25:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:25:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:25:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:43Z" 2023-10-17T18:25:43.233 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:43.234 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567137.874580_310.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:37.874580Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:25:43Z" } 2023-10-17T18:25:43.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567137.874580_310.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:37.874580Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:25:43.235 DEBUG:rgw_multi.tests:log_status={0: '1_1697567137.874580_310.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:43.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567137.874580_310.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:43.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:25:43.236 DEBUG:boto:path=/hbmuog-31/ 2023-10-17T18:25:43.237 DEBUG:boto:auth_path=/hbmuog-31/ 2023-10-17T18:25:43.237 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.238 DEBUG:boto:Path: /hbmuog-31/ 2023-10-17T18:25:43.238 DEBUG:boto:Data: 2023-10-17T18:25:43.238 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.239 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.240 DEBUG:boto:Params: {} 2023-10-17T18:25:43.240 DEBUG:boto:Token: None 2023-10-17T18:25:43.240 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-31/ 2023-10-17T18:25:43.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aGzBTJmNzYwva4yKVIE91Rp7u/I= 2023-10-17T18:25:43.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aGzBTJmNzYwva4yKVIE91Rp7u/I=', 'Content-Length': '0'} 2023-10-17T18:25:43.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9be714e250bbe80-00652ed1a7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.244 DEBUG:boto:path=/hbmuog-31/foo 2023-10-17T18:25:43.245 DEBUG:boto:auth_path=/hbmuog-31/foo 2023-10-17T18:25:43.245 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.245 DEBUG:boto:Path: /hbmuog-31/foo 2023-10-17T18:25:43.246 DEBUG:boto:Data: 2023-10-17T18:25:43.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:25:43.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.247 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.248 DEBUG:boto:Params: {} 2023-10-17T18:25:43.248 DEBUG:boto:Token: None 2023-10-17T18:25:43.249 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-31/foo 2023-10-17T18:25:43.249 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GzpB93OuyxuAjoqEJviLoXdu2XE= 2023-10-17T18:25:43.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GzpB93OuyxuAjoqEJviLoXdu2XE='} 2023-10-17T18:25:43.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003db7019afcb98a67-00652ed1a7-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.256 DEBUG:boto:path=/hbmuog-31/ 2023-10-17T18:25:43.256 DEBUG:boto:auth_path=/hbmuog-31/ 2023-10-17T18:25:43.257 DEBUG:boto:Method: DELETE 2023-10-17T18:25:43.257 DEBUG:boto:Path: /hbmuog-31/ 2023-10-17T18:25:43.258 DEBUG:boto:Data: 2023-10-17T18:25:43.258 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.259 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.259 DEBUG:boto:Params: {} 2023-10-17T18:25:43.260 DEBUG:boto:Token: None 2023-10-17T18:25:43.260 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-31/ 2023-10-17T18:25:43.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XM8bq8TuTYXLeAkA2+8+HgFATeY= 2023-10-17T18:25:43.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XM8bq8TuTYXLeAkA2+8+HgFATeY=', 'Content-Length': '0'} 2023-10-17T18:25:43.268 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx0000085f7ae0677a7f656-00652ed1a7-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.311 DEBUG:boto:path=/hbmuog-32/ 2023-10-17T18:25:43.311 DEBUG:boto:auth_path=/hbmuog-32/ 2023-10-17T18:25:43.312 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.312 DEBUG:boto:Path: /hbmuog-32/ 2023-10-17T18:25:43.313 DEBUG:boto:Data: 2023-10-17T18:25:43.313 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.314 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.314 DEBUG:boto:Params: {} 2023-10-17T18:25:43.315 DEBUG:boto:Token: None 2023-10-17T18:25:43.315 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-32/ 2023-10-17T18:25:43.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LQ0AU3YxRHS741LdP4X0UYD9lKw= 2023-10-17T18:25:43.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LQ0AU3YxRHS741LdP4X0UYD9lKw=', 'Content-Length': '0'} 2023-10-17T18:25:43.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e248c3f9a5550cdd-00652ed1a7-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.319 DEBUG:boto:path=/hbmuog-32/foo 2023-10-17T18:25:43.320 DEBUG:boto:auth_path=/hbmuog-32/foo 2023-10-17T18:25:43.320 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.320 DEBUG:boto:Path: /hbmuog-32/foo 2023-10-17T18:25:43.321 DEBUG:boto:Data: 2023-10-17T18:25:43.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:25:43.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.322 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.323 DEBUG:boto:Params: {} 2023-10-17T18:25:43.323 DEBUG:boto:Token: None 2023-10-17T18:25:43.324 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-32/foo 2023-10-17T18:25:43.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8YyyWnBP+1uU6yV6McmGyzP31hk= 2023-10-17T18:25:43.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8YyyWnBP+1uU6yV6McmGyzP31hk='} 2023-10-17T18:25:43.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000909ea109fa450ac6-00652ed1a7-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.331 DEBUG:boto:path=/hbmuog-32/ 2023-10-17T18:25:43.331 DEBUG:boto:auth_path=/hbmuog-32/ 2023-10-17T18:25:43.332 DEBUG:boto:Method: DELETE 2023-10-17T18:25:43.332 DEBUG:boto:Path: /hbmuog-32/ 2023-10-17T18:25:43.333 DEBUG:boto:Data: 2023-10-17T18:25:43.333 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.333 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.334 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.334 DEBUG:boto:Params: {} 2023-10-17T18:25:43.335 DEBUG:boto:Token: None 2023-10-17T18:25:43.335 DEBUG:boto:StringToSign: DELETE Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-32/ 2023-10-17T18:25:43.336 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+20aIUq8sYQSlSHW4p2KXwy+7JI= 2023-10-17T18:25:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+20aIUq8sYQSlSHW4p2KXwy+7JI=', 'Content-Length': '0'} 2023-10-17T18:25:43.343 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000005adab8eb2fdb5a5d-00652ed1a7-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.343 DEBUG:boto:path=/hbmuog-31/ 2023-10-17T18:25:43.344 DEBUG:boto:auth_path=/hbmuog-31/ 2023-10-17T18:25:43.344 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.345 DEBUG:boto:Path: /hbmuog-31/ 2023-10-17T18:25:43.345 DEBUG:boto:Data: 2023-10-17T18:25:43.345 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.346 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.347 DEBUG:boto:Params: {} 2023-10-17T18:25:43.347 DEBUG:boto:Token: None 2023-10-17T18:25:43.348 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-31/ 2023-10-17T18:25:43.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aGzBTJmNzYwva4yKVIE91Rp7u/I= 2023-10-17T18:25:43.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aGzBTJmNzYwva4yKVIE91Rp7u/I=', 'Content-Length': '0'} 2023-10-17T18:25:43.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b7cbd8ec7105cc6-00652ed1a7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.351 DEBUG:boto:path=/hbmuog-32/ 2023-10-17T18:25:43.352 DEBUG:boto:auth_path=/hbmuog-32/ 2023-10-17T18:25:43.352 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.353 DEBUG:boto:Path: /hbmuog-32/ 2023-10-17T18:25:43.353 DEBUG:boto:Data: 2023-10-17T18:25:43.354 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.355 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.355 DEBUG:boto:Params: {} 2023-10-17T18:25:43.355 DEBUG:boto:Token: None 2023-10-17T18:25:43.356 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-32/ 2023-10-17T18:25:43.356 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LQ0AU3YxRHS741LdP4X0UYD9lKw= 2023-10-17T18:25:43.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LQ0AU3YxRHS741LdP4X0UYD9lKw=', 'Content-Length': '0'} 2023-10-17T18:25:43.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f641c24cd57738b-00652ed1a7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-10-17T18:25:43.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-10-17T18:25:43.367 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-33 2023-10-17T18:25:43.367 DEBUG:boto:path=/hbmuog-33/ 2023-10-17T18:25:43.368 DEBUG:boto:auth_path=/hbmuog-33/ 2023-10-17T18:25:43.368 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.369 DEBUG:boto:Path: /hbmuog-33/ 2023-10-17T18:25:43.369 DEBUG:boto:Data: 2023-10-17T18:25:43.369 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.370 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.371 DEBUG:boto:Params: {} 2023-10-17T18:25:43.371 DEBUG:boto:Token: None 2023-10-17T18:25:43.371 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-33/ 2023-10-17T18:25:43.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UXXOw8Y13xzPIsrRm6mmSdFNm6M= 2023-10-17T18:25:43.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UXXOw8Y13xzPIsrRm6mmSdFNm6M=', 'Content-Length': '0'} 2023-10-17T18:25:43.386 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad4a702f2eef3986-00652ed1a7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.387 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-34 2023-10-17T18:25:43.387 DEBUG:boto:path=/hbmuog-34/ 2023-10-17T18:25:43.388 DEBUG:boto:auth_path=/hbmuog-34/ 2023-10-17T18:25:43.388 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.388 DEBUG:boto:Path: /hbmuog-34/ 2023-10-17T18:25:43.389 DEBUG:boto:Data: 2023-10-17T18:25:43.389 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.390 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.390 DEBUG:boto:Params: {} 2023-10-17T18:25:43.391 DEBUG:boto:Token: None 2023-10-17T18:25:43.391 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-34/ 2023-10-17T18:25:43.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ehuNmT/Vp6wihkZEJxFbT5Yg+x0= 2023-10-17T18:25:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ehuNmT/Vp6wihkZEJxFbT5Yg+x0=', 'Content-Length': '0'} 2023-10-17T18:25:43.416 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b8c32bd147eb5b8-00652ed1a7-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.416 DEBUG:boto:path=/hbmuog-33/ 2023-10-17T18:25:43.417 DEBUG:boto:auth_path=/hbmuog-33/ 2023-10-17T18:25:43.417 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.418 DEBUG:boto:Path: /hbmuog-33/ 2023-10-17T18:25:43.418 DEBUG:boto:Data: 2023-10-17T18:25:43.418 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.419 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.420 DEBUG:boto:Params: {} 2023-10-17T18:25:43.420 DEBUG:boto:Token: None 2023-10-17T18:25:43.421 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-33/ 2023-10-17T18:25:43.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pszdq0ItaMJWGaF5iV0hL+Aowvc= 2023-10-17T18:25:43.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pszdq0ItaMJWGaF5iV0hL+Aowvc=', 'Content-Length': '0'} 2023-10-17T18:25:43.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004231853af97efe8c-00652ed1a7-1113-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.424 DEBUG:boto:path=/hbmuog-33/key 2023-10-17T18:25:43.425 DEBUG:boto:auth_path=/hbmuog-33/key 2023-10-17T18:25:43.425 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.425 DEBUG:boto:Path: /hbmuog-33/key 2023-10-17T18:25:43.426 DEBUG:boto:Data: 2023-10-17T18:25:43.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-17T18:25:43.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:25:43.427 DEBUG:boto:Port: 8000 2023-10-17T18:25:43.427 DEBUG:boto:Params: {} 2023-10-17T18:25:43.428 DEBUG:boto:Token: None 2023-10-17T18:25:43.428 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-33/key 2023-10-17T18:25:43.429 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jyXcT1ARmhxElos3LaYF/n7It4w= 2023-10-17T18:25:43.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jyXcT1ARmhxElos3LaYF/n7It4w='} 2023-10-17T18:25:43.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ed2384e7b94d374-00652ed1a7-1113-a1'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.475 DEBUG:boto:path=/hbmuog-34/ 2023-10-17T18:25:43.475 DEBUG:boto:auth_path=/hbmuog-34/ 2023-10-17T18:25:43.476 DEBUG:boto:Method: HEAD 2023-10-17T18:25:43.476 DEBUG:boto:Path: /hbmuog-34/ 2023-10-17T18:25:43.477 DEBUG:boto:Data: 2023-10-17T18:25:43.477 DEBUG:boto:Headers: {} 2023-10-17T18:25:43.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.478 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.478 DEBUG:boto:Params: {} 2023-10-17T18:25:43.479 DEBUG:boto:Token: None 2023-10-17T18:25:43.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-34/ 2023-10-17T18:25:43.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zf0wFuz35WEgQdCcjsmzCBTEX2Y= 2023-10-17T18:25:43.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zf0wFuz35WEgQdCcjsmzCBTEX2Y=', 'Content-Length': '0'} 2023-10-17T18:25:43.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5120ab672b6395b-00652ed1a7-110f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.483 DEBUG:boto:path=/hbmuog-34/key 2023-10-17T18:25:43.484 DEBUG:boto:auth_path=/hbmuog-34/key 2023-10-17T18:25:43.484 DEBUG:boto:Method: PUT 2023-10-17T18:25:43.484 DEBUG:boto:Path: /hbmuog-34/key 2023-10-17T18:25:43.485 DEBUG:boto:Data: 2023-10-17T18:25:43.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-17T18:25:43.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:25:43.486 DEBUG:boto:Port: 8001 2023-10-17T18:25:43.487 DEBUG:boto:Params: {} 2023-10-17T18:25:43.487 DEBUG:boto:Token: None 2023-10-17T18:25:43.487 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 17 Oct 2023 18:25:43 GMT /hbmuog-34/key 2023-10-17T18:25:43.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ktLH4V7OaUb5UbgDbDF+BaRLgtE= 2023-10-17T18:25:43.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:25:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ktLH4V7OaUb5UbgDbDF+BaRLgtE='} 2023-10-17T18:25:43.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000788368191442a615-00652ed1a7-110f-a2'), ('Date', 'Tue, 17 Oct 2023 18:25:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:25:43.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.536 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:43.588 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:43.588 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:43.618 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:25:43.630 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:25:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:25:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:43.407101Z" 2023-10-17T18:25:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:25:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:25:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:25:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.634 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:25:43.635 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:43.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:25:43.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:43.636 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:43.687 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:43.687 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:43.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:43.714+0000 7f6ebd5b9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:43.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:43.715+0000 7f6ebd5b9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:43.728 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:43.728 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:43.728 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:43.729 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:43.729 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:43.729 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:25:43.729 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.730 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:43.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.730 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:43.731 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.731 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.731 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.379123_311.1", 2023-10-17T18:25:43.731 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.731 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:25:43.732 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.732 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.379123Z", 2023-10-17T18:25:43.732 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.733 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:43.733 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.733 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.734 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:43.734 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.734 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.734 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.735 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:25:43.735 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.735 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.736 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:43.736 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.736 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.736 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.737 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.737 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.737 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.737 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:43.737 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.738 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:43.738 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:43.739 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:43.739 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:43.739 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:43.739 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:43.739 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:43.740 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:43.740 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:43.740 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:43.740 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.740 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:43.741 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:43.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.741 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:43.741 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:25:43.742 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:25:43.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:43.742 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:43Z" 2023-10-17T18:25:43.742 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:43.743 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.379123_311.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.379123Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:25:43Z" } 2023-10-17T18:25:43.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.379123_311.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.379123Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:25:43.744 DEBUG:rgw_multi.tests:log_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:43.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.379123_311.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:43.745 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567143.407101_318.1 target=1_1697567143.379123_311.1 2023-10-17T18:25:48.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.748 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:48.806 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:48.807 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:48.837+0000 7f0ca0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:48.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:25:48.838+0000 7f0ca0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:25:48.853 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:48.853 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:48.853 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:48.853 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:48.853 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:48.854 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:25:48.854 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:48.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.854 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:25:48.855 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:48.856 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:25:48.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:48.856 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:25:48.856 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:48.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.857 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:48.857 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:48.858 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:48.858 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:25:48.858 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:48.858 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:48.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:48.859 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:25:48.859 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:48.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.860 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:48.860 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:48.860 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:48.860 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:48.861 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:48.861 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:48.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:48.861 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:48.861 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:48.862 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.862 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:48.863 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:48.863 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:25:48.863 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:25:48.863 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:48.863 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:48.864 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:48.864 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:25:48.864 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:25:48.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.865 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:48.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.865 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:48.865 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:25:48.866 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:25:48.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.866 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:48Z" 2023-10-17T18:25:48.866 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:48.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:25:48Z" } 2023-10-17T18:25:48.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:25:48.868 DEBUG:rgw_multi.tests:log_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:48.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:25:48.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:25:48.869 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T18:25:48.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.871 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:48.922 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:48.969 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:25:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:25:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:25:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-17T18:25:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:43.255543Z" 2023-10-17T18:25:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:25:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-17T18:25:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:43.435151Z" 2023-10-17T18:25:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:48.973 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:25:48.974 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-17T18:25:48.975 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:25:48.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:48.976 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:49.028 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:49.029 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:49.071 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:25:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:25:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:25:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-17T18:25:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.380239Z" 2023-10-17T18:25:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:25:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003042", 2023-10-17T18:25:49.083 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:49.083 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:49.083 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:49.083 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:20.177318Z" 2023-10-17T18:25:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:25:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:25:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:49Z" 2023-10-17T18:25:49.086 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:49.087 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:21:09.376621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.380239Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003042", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:20.177318Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:25:49Z" } 2023-10-17T18:25:49.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:21:09.376621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.380239Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:20.177318Z'}}] 2023-10-17T18:25:49.088 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-17T18:25:49.088 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003042'} 2023-10-17T18:25:49.089 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000003183 target=00000000000000000000:00000000000000003042 2023-10-17T18:25:54.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.091 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:54.152 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:54.152 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:54.199 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:54.199 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:54.199 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:54.199 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:54.200 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:54.200 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:25:54.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.200 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:54.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.201 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:54.201 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.201 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.201 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:25:54.202 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.202 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.202 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:25:54.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.203 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:54.203 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.203 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.204 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-17T18:25:54.204 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.204 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.204 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.204 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.255543Z" 2023-10-17T18:25:54.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.205 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:54.205 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.206 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.206 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:54.206 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.206 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.206 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:54.207 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.208 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.208 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-17T18:25:54.208 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.208 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.208 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.209 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.435151Z" 2023-10-17T18:25:54.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.209 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:54.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.210 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:54.210 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:25:54.210 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:25:54.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.210 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:54Z" 2023-10-17T18:25:54.211 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:54.211 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:21:09.376621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.255543Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.435151Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:25:54Z" } 2023-10-17T18:25:54.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:21:09.376621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.255543Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.435151Z'}}] 2023-10-17T18:25:54.212 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-17T18:25:54.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-17T18:25:54.213 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:25:54.213 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T18:25:54.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.215 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:54.265 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:54.265 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:54.311 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:25:54.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.312 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:25:54.312 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:25:54.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:25:54.313 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:49.471975Z" 2023-10-17T18:25:54.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:54.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:25:54.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.314 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-17T18:25:54.315 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:50.629973Z" 2023-10-17T18:25:54.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.315 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:25:54.316 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:25:54.317 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:25:54.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:54.318 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:54.371 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:54.372 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:54.414 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:54.414 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:54.414 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:54.415 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:54.415 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:54.415 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:25:54.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.415 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:25:54.416 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.417 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.417 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.417 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:25:54.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.418 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:54.418 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.418 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.419 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-17T18:25:54.419 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.419 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.419 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.419 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.330496Z" 2023-10-17T18:25:54.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.420 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:54.420 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.421 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.421 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:54.421 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.421 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.422 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.422 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:25:54.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:54.423 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:54.423 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-17T18:25:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:54.424 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:54.425 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.494105Z" 2023-10-17T18:25:54.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:54.425 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:54.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.426 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:54.426 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:25:54.426 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:25:54.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:54.427 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:54Z" 2023-10-17T18:25:54.427 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:54.428 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.330496Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.494105Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:25:54Z" } 2023-10-17T18:25:54.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.330496Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.494105Z'}}] 2023-10-17T18:25:54.429 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:25:54.430 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004029'} 2023-10-17T18:25:54.430 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-10-17T18:25:59.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:59.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:25:59.433 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:25:59.494 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:25:59.495 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:25:59.542 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:25:59.543 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:25:59.543 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:25:59.543 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:25:59.543 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:25:59.543 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:25:59.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.544 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:25:59.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:59.544 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:25:59.545 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:59.545 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:59.545 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:25:59.545 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:25:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:59.547 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:25:59.547 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:59.547 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:59.548 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-17T18:25:59.548 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:59.548 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:59.548 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:59.548 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.330496Z" 2023-10-17T18:25:59.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:59.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:59.549 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:25:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:25:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:59.550 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:59.551 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:59.551 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:25:59.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:59.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:25:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:25:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:25:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:25:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-17T18:25:59.552 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:25:59.553 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:25:59.553 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:25:59.553 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.494105Z" 2023-10-17T18:25:59.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:59.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:25:59.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:25:59.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.554 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:25:59.554 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:25:59.555 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:25:59.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:25:59.555 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:25:59Z" 2023-10-17T18:25:59.555 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:25:59.556 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.330496Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.494105Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:25:59Z" } 2023-10-17T18:25:59.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.330496Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.494105Z'}}] 2023-10-17T18:25:59.557 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:25:59.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004029'} 2023-10-17T18:25:59.558 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-10-17T18:26:04.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:04.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:04.561 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:04.622 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:04.622 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:26:04.670 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:26:04.670 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:26:04.671 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:26:04.671 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:26:04.671 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:26:04.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.671 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:26:04.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:04.672 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:26:04.672 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:04.672 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:04.673 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:26:04.673 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:04.673 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:04.673 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:04.673 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:26:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:26:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:04.675 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:04.675 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-17T18:26:04.675 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:04.675 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:04.675 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:04.676 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.330496Z" 2023-10-17T18:26:04.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:04.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:04.677 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:26:04.677 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:04.677 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:04.677 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:26:04.677 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:04.678 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:04.678 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:04.678 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:26:04.678 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:04.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:04.679 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:26:04.679 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:04.679 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:04.680 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-17T18:26:04.680 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:04.680 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:04.680 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:04.680 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.494105Z" 2023-10-17T18:26:04.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:04.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:04.681 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:04.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.681 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:26:04.682 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:26:04.682 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:26:04.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:04.682 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:26:04Z" 2023-10-17T18:26:04.683 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:26:04.684 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.330496Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.494105Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:26:04Z" } 2023-10-17T18:26:04.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.330496Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.494105Z'}}] 2023-10-17T18:26:04.685 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:26:04.685 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004029'} 2023-10-17T18:26:04.685 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-10-17T18:26:09.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:09.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:09.688 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:09.750 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:09.797 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:26:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:26:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:26:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:26:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:26:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:26:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:26:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:26:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:26:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:26:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:26:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:26:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.471975Z" 2023-10-17T18:26:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:26:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:26:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:26:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:26:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-10-17T18:26:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.494105Z" 2023-10-17T18:26:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:26:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:26:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:26:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:26:09Z" 2023-10-17T18:26:09.811 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:26:09.812 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.471975Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:43.494105Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:26:09Z" } 2023-10-17T18:26:09.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.471975Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.494105Z'}}] 2023-10-17T18:26:09.813 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:26:09.813 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004029'} 2023-10-17T18:26:09.814 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-10-17T18:26:14.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:14.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:14.816 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:14.877 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:14.877 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:14.924 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:26:14.924 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:26:14.924 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:26:14.925 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:26:14.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:26:14.925 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:26:14.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.925 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:26:14.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:14.926 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:26:14.926 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:14.926 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:14.927 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001499", 2023-10-17T18:26:14.927 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:14.927 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:14.927 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:14.927 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:20:49.427456Z" 2023-10-17T18:26:14.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:14.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:14.928 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:26:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:26:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:14.929 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.471975Z" 2023-10-17T18:26:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:14.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.629973Z" 2023-10-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:26:14Z" 2023-10-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:26:14.940 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:20:49.427456Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.471975Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.629973Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:26:14Z" } 2023-10-17T18:26:14.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:20:49.427456Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.471975Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.629973Z'}}] 2023-10-17T18:26:14.941 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:26:14.942 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001499', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:26:14.942 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:26:14.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:14.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:14.943 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:14.993 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:14.993 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:26:15.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:15.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:15.047 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.144 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.196 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.197 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.238 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.241 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.333 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.335 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.386 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.387 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.427 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.430 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.480 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.480 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.521 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.524 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.572 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.572 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.618 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:26:15.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.619 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-17T18:26:15.619 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:21:09.376621Z" 2023-10-17T18:26:15.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.620 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:26:15.620 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:26:15.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.621 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:26:15.621 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:26:15.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:15.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:15.622 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:26:15.622 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:26:15.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:15.622 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:26:15.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:15.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:15.624 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:26:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.723 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.814 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.817 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.868 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.868 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:15.909 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:15.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:15.912 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:15.962 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:15.963 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:16.004 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:16.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:16.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:26:16.007 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:26:16.059 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:16.059 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:16.103 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:26:16.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-10-17T18:26:16.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:16.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:16.107 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:26:16.157 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:16.158 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:16.178 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:26:16.179 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:26:16.179 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:26:16.179 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:26:16.180 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:26:16.180 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:26:16.180 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:26:16.181 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:26:16.181 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:26:16.181 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:26:16.181 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:26:16.182 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:26:16.182 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:26:16.184 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:26:16.184 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:26:16.184 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:26:16.185 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:26:16.185 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:26:16.185 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:26:16.186 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:26:16.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.186 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:26:16.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.187 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:26:16.187 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:26:16.188 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:26:16.188 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:26:16.188 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:26:16.188 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-17T18:26:16.189 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:26:16.189 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.190 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:26:16.190 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:26:16.191 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:26:16.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.192 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.192 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.192 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:26:16.192 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:26:16.193 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:26:16.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:16.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:26:16.195 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:26:16.249 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:26:16.249 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:26:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:26:16.316 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:26:16.316 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:26:16.317 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5, 2023-10-17T18:26:16.317 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:26:16.317 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:26:16.317 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:26:16.318 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:26:16.318 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:26:16.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.318 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:26:16.319 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:26:16.319 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:26:16.319 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:26:16.319 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.319 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:26:16.320 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:26:16.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:26:16.320 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:26:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:26:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:26:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:26:16.321 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.322 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:26:16.322 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.322 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:26:16.322 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:26:16.323 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:26:16.323 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:26:16.323 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:26:16.323 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:26:16.324 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:26:16.324 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:26:16.324 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:26:16.324 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:26:16.325 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.325 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.325 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:26:16.326 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:26:16.326 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.326 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:26:16.327 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.327 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:26:16.327 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:26:16.327 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:26:16.328 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:26:16.328 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:26:16.328 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": false, 2023-10-17T18:26:16.328 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:26:16.329 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:26:16.329 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:26:16.329 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:26:16.329 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.330 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.330 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.330 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:26:16.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.331 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:26:16.331 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:26:16.331 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:26:16.332 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:26:16.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.332 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.333 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:26:16.333 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:26:16.333 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:26:16.333 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:26:16.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.334 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:26:16.334 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.334 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.335 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:26:16.335 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:26:16.335 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:26:16.336 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:26:16.336 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.336 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:26:16.336 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:26:16.337 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:26:16.337 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:26:16.337 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:26:16.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.338 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:26:16.338 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:26:16.338 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.338 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:26:16.338 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.339 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:26:16.339 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:26:16.339 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:26:16.339 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:26:16.340 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:26:16.340 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:26:16.340 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:26:16.340 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:26:16.340 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:26:16.341 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:26:16.342 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.343 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:26:16.343 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:26:16.344 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:26:16.344 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:26:16.344 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.344 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:26:16.345 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:26:16.345 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:26:16.345 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:26:16.345 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:26:16.346 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:26:16.346 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:26:16.346 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:26:16.346 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:26:16.347 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:26:16.347 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.347 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.348 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:26:16.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.348 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:26:16.348 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:26:16.349 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:26:16.349 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:26:16.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.349 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.350 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:26:16.350 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:26:16.350 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:26:16.350 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:26:16.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.351 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:26:16.351 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:26:16.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.352 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:26:16.352 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:26:16.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.352 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:26:16.353 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:26:16.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.353 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:26:16.354 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:26:16.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.354 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:26:16.355 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:26:16.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:26:16.355 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:26:16.355 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:26:16.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:26:16.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.356 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:26:16.357 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:26:16.357 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:26:16.357 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:26:16.357 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:26:16.357 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:26:16.358 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:26:16.358 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:26:16.358 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:26:16.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.359 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:26:16.359 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:26:16.359 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:26:16.359 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:26:16.360 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:26:16.360 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:26:16.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.360 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:26:16.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:26:16.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:26:16.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:26:16.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:26:16.362 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:26:16.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.362 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:26:16.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:26:16.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:26:16.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:26:16.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:26:16.363 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:26:16.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.364 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:26:16.364 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:26:16.364 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:26:16.364 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:26:16.365 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:26:16.365 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:26:16.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:26:16.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:26:16.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:26:16.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:26:16.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:26:16.366 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:26:16.367 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-10-17T18:27:16.368 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-35 2023-10-17T18:27:16.369 DEBUG:boto:path=/hbmuog-35/ 2023-10-17T18:27:16.370 DEBUG:boto:auth_path=/hbmuog-35/ 2023-10-17T18:27:16.370 DEBUG:boto:Method: PUT 2023-10-17T18:27:16.370 DEBUG:boto:Path: /hbmuog-35/ 2023-10-17T18:27:16.371 DEBUG:boto:Data: 2023-10-17T18:27:16.371 DEBUG:boto:Headers: {} 2023-10-17T18:27:16.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:27:16.372 DEBUG:boto:Port: 8000 2023-10-17T18:27:16.372 DEBUG:boto:Params: {} 2023-10-17T18:27:16.373 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:27:16.374 DEBUG:boto:Token: None 2023-10-17T18:27:16.374 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:27:16 GMT /hbmuog-35/ 2023-10-17T18:27:16.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m6TJg3yf4KqDvTI0MBn4Wl52/oA= 2023-10-17T18:27:16.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:27:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m6TJg3yf4KqDvTI0MBn4Wl52/oA=', 'Content-Length': '0'} 2023-10-17T18:27:16.396 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bc7c5b481ec4ae2-00652ed204-1513-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:27:16.397 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:27:16.398 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:27:16.398 DEBUG:boto:Method: PUT 2023-10-17T18:27:16.399 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:27:16.399 DEBUG:boto:Data: 2023-10-17T18:27:16.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:27:16.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:27:16.400 DEBUG:boto:Port: 8000 2023-10-17T18:27:16.401 DEBUG:boto:Params: {} 2023-10-17T18:27:16.401 DEBUG:boto:Token: None 2023-10-17T18:27:16.401 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 17 Oct 2023 18:27:16 GMT /hbmuog-35/testredirect 2023-10-17T18:27:16.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:54pkIcKo96nHtjtJWcoIr9xEM8w= 2023-10-17T18:27:16.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:27:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:54pkIcKo96nHtjtJWcoIr9xEM8w='} 2023-10-17T18:27:16.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c30ae856f631e6e-00652ed204-1513-a1'), ('Date', 'Tue, 17 Oct 2023 18:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:27:16.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:16.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:16.413 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:16.469 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:16.470 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:16.503 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:27:16.516 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:27:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:27:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:27:16.394899Z" 2023-10-17T18:27:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:27:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:27:16.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.519 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:16.519 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:27:16.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.520 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:27:16.521 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:16.521 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:27:16.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:16.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:16.522 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:16.576 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:16.577 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:16.606+0000 7fa6544e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:16.607+0000 7fa6544e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:16.619 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:16.619 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:16.619 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:16.620 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:16.620 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:16.620 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:16.620 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:16.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.621 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:16.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.621 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:16.621 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:16.621 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:16.622 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:16.622 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:16.622 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:16.622 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:16.622 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:16.623 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:16.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.623 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:16.624 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:16.624 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:16.624 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:16.624 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:16.624 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.626 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:16.626 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:16.626 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:16.626 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:16.626 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:16.627 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:16.627 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:16.627 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:16.627 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:16.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:16.628 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:16.628 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:16.628 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:16.629 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:16.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:16.630 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:16.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:16Z" 2023-10-17T18:27:16.631 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:16.632 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:16Z" } 2023-10-17T18:27:16.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:16.633 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:16.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:16.634 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:21.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:21.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:21.636 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:21.694 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:21.695 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:21.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:21.727+0000 7f94deb3d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:21.728+0000 7f94deb3d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:21.741 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:21.741 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:21.741 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:21.741 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:21.741 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:21.742 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:21.742 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:21.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.742 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:21.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:21.743 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:21.743 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:21.743 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:21.743 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:21.744 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:21.744 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:21.744 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:21.744 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:21.744 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:21.745 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:21.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:21.745 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:21.745 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:21.746 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:21.746 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:21.746 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:21.746 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:21.746 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:21.748 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:21.748 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:21.748 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:21.748 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:21.748 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:21.749 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:21.749 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:21.749 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:21.749 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:21.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:21.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:21.750 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:21.750 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:21.751 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:21.751 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:21.751 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:21.751 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:21.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:21.752 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:21.752 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:21.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:21.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:21.752 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:21.753 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:21Z" 2023-10-17T18:27:21.754 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:21.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:21Z" } 2023-10-17T18:27:21.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:21.756 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:21.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:21.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:26.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:26.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:26.759 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:26.818 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:26.819 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:26.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:26.850+0000 7fc31b65d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:26.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:26.851+0000 7fc31b65d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:26.864 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:26.864 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:26.865 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:26.865 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:26.865 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:26.865 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:26.866 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:26.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.866 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:26.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:26.867 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:26.867 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:26.867 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:26.867 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:26.867 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:26.868 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:26.868 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:26.868 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:26.868 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:26.868 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:26.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:26.869 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:26.869 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:26.869 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:26.870 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:26.870 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:26.870 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:26.870 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:26.871 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:26.871 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:26.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:26.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:26.872 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:26.872 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:26.872 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:26.872 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:26.872 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:26.873 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:26.873 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:26.873 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:26.873 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:26.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:26.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:26.874 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:26.874 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:26.874 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:26.876 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:26.876 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:26.876 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:26.876 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:26.877 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:26Z" 2023-10-17T18:27:26.878 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:26.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:26Z" } 2023-10-17T18:27:26.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:26.880 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:26.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:26.880 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:31.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:31.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:31.882 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:31.941 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:31.942 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:31.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:31.973+0000 7f6d03658780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:31.974+0000 7f6d03658780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:31.988 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:31.989 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:31.989 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:31.989 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:31.989 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:31.990 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:31.990 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:31.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:31.990 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:31.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:31.991 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:31.991 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:31.991 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:31.991 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:31.992 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:31.992 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:31.992 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:31.992 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:31.993 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:31.994 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:31.994 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:31.994 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:31.994 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:31.994 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:31.995 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:31.995 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:31.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:31.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:31.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:31.996 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:31.996 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:31.996 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:31.996 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:31.996 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:31.997 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:31.997 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:31.997 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:31.997 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:31.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:31.998 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:31.999 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:31.999 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:31.999 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:31.999 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:31.999 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:32.000 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:32.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:32.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:32.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:32.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:32.001 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:32.001 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:32.001 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:32.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:32.002 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:31Z" 2023-10-17T18:27:32.002 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:32.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:31Z" } 2023-10-17T18:27:32.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:32.004 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:32.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:32.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:37.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:37.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:37.007 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:37.065 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:37.065 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:37.096+0000 7f8be7636780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:37.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:37.097+0000 7f8be7636780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:37Z" 2023-10-17T18:27:37.126 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:37.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:37Z" } 2023-10-17T18:27:37.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:37.128 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:37.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:37.128 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:42.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:42.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:42.131 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:42.191 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:42.191 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:42.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:42.223+0000 7ffa29e48780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:42.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:42.224+0000 7ffa29e48780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:42.239 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:42.239 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:42.239 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:42.239 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:42.240 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:42.240 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:42.240 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:42.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.241 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:42.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:42.241 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:42.241 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:42.241 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:42.242 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567143.407101_318.1", 2023-10-17T18:27:42.242 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:42.242 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:42.242 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:42.242 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:43.407101Z", 2023-10-17T18:27:42.243 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:42.243 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:42.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:42.243 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:42.244 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:42.244 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:42.244 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:42.244 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:42.244 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:42.245 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:42.245 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:42.245 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:42.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:42.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:42.246 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:42.246 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:42.246 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:42.247 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:42.247 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:42.247 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:42.247 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:42.247 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:42.248 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:42.248 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:42.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:42.248 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:42.249 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:42.249 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:42.249 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:42.249 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:42.249 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:42.250 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:42.250 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:42.250 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:42.250 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:42.250 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:42.251 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:42.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.251 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:42.251 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:42.251 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:42.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:42.252 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:42Z" 2023-10-17T18:27:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:42.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567143.407101_318.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:25:43.407101Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:42Z" } 2023-10-17T18:27:42.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567143.407101_318.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:25:43.407101Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:42.254 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:42.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567143.407101_318.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:42.255 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567236.394899_344.1 target=1_1697567143.407101_318.1 2023-10-17T18:27:47.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:47.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:27:47.259 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:27:47.320 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:47.320 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:47.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:47.351+0000 7f018c69c780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:47.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:27:47.353+0000 7f018c69c780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:27:47.367 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:47.368 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:27:47.368 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:27:47.368 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:27:47.368 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:27:47.369 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:47.369 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.369 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:27:47.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.370 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:27:47.370 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:47.370 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:47.370 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:27:47.371 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:47.371 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:27:47.371 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:47.371 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:27:47.371 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.372 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:27:47.373 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:47.373 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:47.373 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:27:47.373 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:47.373 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:47.374 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:47.374 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:27:47.374 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.375 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:27:47.375 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:47.375 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:47.375 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:47.376 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:47.376 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:47.376 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:47.376 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:47.376 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.377 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:27:47.378 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:47.378 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:27:47.378 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:27:47.378 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:27:47.378 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:27:47.379 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:27:47.379 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:27:47.379 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.380 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:27:47.380 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:27:47.380 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:27:47.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.381 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:27:47Z" 2023-10-17T18:27:47.381 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:47.382 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:27:47Z" } 2023-10-17T18:27:47.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:27:47.383 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:47.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:27:47.384 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:27:47.384 DEBUG:boto:path=/hbmuog-35/ 2023-10-17T18:27:47.385 DEBUG:boto:auth_path=/hbmuog-35/ 2023-10-17T18:27:47.385 DEBUG:boto:Method: HEAD 2023-10-17T18:27:47.385 DEBUG:boto:Path: /hbmuog-35/ 2023-10-17T18:27:47.386 DEBUG:boto:Data: 2023-10-17T18:27:47.386 DEBUG:boto:Headers: {} 2023-10-17T18:27:47.387 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:27:47.387 DEBUG:boto:Port: 8001 2023-10-17T18:27:47.387 DEBUG:boto:Params: {} 2023-10-17T18:27:47.388 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:27:47.389 DEBUG:boto:Token: None 2023-10-17T18:27:47.389 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:27:47 GMT /hbmuog-35/ 2023-10-17T18:27:47.390 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x4snt+1glTPLIiT0fq36JcWtKqg= 2023-10-17T18:27:47.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:27:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x4snt+1glTPLIiT0fq36JcWtKqg=', 'Content-Length': '0'} 2023-10-17T18:27:47.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe0c23f7a5ee2469-00652ed223-150f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:27:47.396 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:27:47.397 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:27:47.397 DEBUG:boto:Method: HEAD 2023-10-17T18:27:47.398 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:27:47.398 DEBUG:boto:Data: 2023-10-17T18:27:47.399 DEBUG:boto:Headers: {} 2023-10-17T18:27:47.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:27:47.399 DEBUG:boto:Port: 8001 2023-10-17T18:27:47.400 DEBUG:boto:Params: {} 2023-10-17T18:27:47.400 DEBUG:boto:Token: None 2023-10-17T18:27:47.401 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:27:47 GMT /hbmuog-35/testredirect 2023-10-17T18:27:47.401 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbv8jcFKCId3JSfZnTL0S0KQrbY= 2023-10-17T18:27:47.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:27:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbv8jcFKCId3JSfZnTL0S0KQrbY=', 'Content-Length': '0'} 2023-10-17T18:27:47.403 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000037b9feb04918acfa-00652ed223-150f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:27:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:27:47.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:27:47.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:27:47.405 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:27:47.457 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:47.457 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:47.477 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:47.478 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:27:47.478 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:27:47.478 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:27:47.478 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:27:47.479 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:27:47.479 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:27:47.479 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:27:47.479 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:27:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:27:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:27:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:27:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:27:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:27:47.481 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:27:47.481 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:27:47.481 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:27:47.481 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:27:47.481 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:27:47.482 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:27:47.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.482 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:27:47.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.483 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:27:47.483 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:27:47.483 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:27:47.483 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:27:47.484 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:27:47.484 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-17T18:27:47.484 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:27:47.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.485 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:27:47.485 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:27:47.485 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:27:47.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.486 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.486 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.486 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:27:47.486 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:27:47.486 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:47.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:27:47.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:27:47.489 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:27:47.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:27:47.539 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:27:47.580 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:27:47.604 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:27:47.605 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:47.605 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6, 2023-10-17T18:27:47.605 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:27:47.605 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:27:47.606 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:27:47.606 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:27:47.606 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:27:47.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.607 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:27:47.607 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:27:47.607 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:27:47.607 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:27:47.608 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.608 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:27:47.608 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.608 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:27:47.609 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:27:47.609 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:27:47.609 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:27:47.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.609 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:27:47.610 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:27:47.610 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.610 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:27:47.610 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.611 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:27:47.611 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:27:47.611 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:27:47.611 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:27:47.611 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:27:47.612 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:27:47.612 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:27:47.612 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:27:47.613 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:27:47.613 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:27:47.613 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:27:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:27:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:27:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:27:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:27:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:27:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:27:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:27:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": false, 2023-10-17T18:27:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:27:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:27:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:27:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:27:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:27:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:27:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:27:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:27:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:27:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:27:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:27:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:27:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:27:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:27:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:27:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:27:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:27:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:27:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:27:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:27:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:27:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:27:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:27:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:27:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:27:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:27:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:27:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:27:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:27:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:27:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:27:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:27:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:27:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:27:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:27:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:27:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.633 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:27:47.633 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:27:47.634 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:27:47.634 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:27:47.634 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.634 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:27:47.635 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:27:47.635 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:27:47.635 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:27:47.635 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:27:47.636 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:27:47.636 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:27:47.636 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:27:47.636 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:27:47.636 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:27:47.637 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.637 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.637 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.637 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.637 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:27:47.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.638 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:27:47.638 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:27:47.638 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:27:47.639 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:27:47.639 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.639 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.639 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.639 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:27:47.640 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:27:47.640 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:27:47.640 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:27:47.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.641 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:27:47.641 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:27:47.641 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.642 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:27:47.642 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:27:47.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.643 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:27:47.643 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:27:47.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.644 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:27:47.644 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:27:47.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.645 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:27:47.645 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:27:47.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:27:47.646 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:27:47.646 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:27:47.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.647 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:27:47.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.647 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:27:47.647 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:27:47.648 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:27:47.648 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:27:47.648 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:27:47.648 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:27:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:27:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:27:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:27:47.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:27:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:27:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:27:47.650 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:27:47.651 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:27:47.651 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:27:47.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.651 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:27:47.652 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:27:47.652 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:27:47.652 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:27:47.652 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:27:47.652 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:27:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:27:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:27:47.653 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:27:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:27:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:27:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:27:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.654 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:27:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:27:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:27:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:27:47.655 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:27:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:27:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:27:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:27:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:27:47.656 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:27:47.657 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:27:47.657 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:27:47.657 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T18:28:47.659 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.660 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.660 DEBUG:boto:Method: HEAD 2023-10-17T18:28:47.661 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.661 DEBUG:boto:Data: 2023-10-17T18:28:47.662 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.663 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.663 DEBUG:boto:Params: {} 2023-10-17T18:28:47.664 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.665 DEBUG:boto:Token: None 2023-10-17T18:28:47.665 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tws81IjQqVwon8QeXxMkmiry6bc= 2023-10-17T18:28:47.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tws81IjQqVwon8QeXxMkmiry6bc=', 'Content-Length': '0'} 2023-10-17T18:28:47.672 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx00000a85fb01b22c972b3-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.672 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.673 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-10-17T18:28:47.673 DEBUG:boto:Token: None 2023-10-17T18:28:47.674 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.674 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tws81IjQqVwon8QeXxMkmiry6bc= 2023-10-17T18:28:47.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tws81IjQqVwon8QeXxMkmiry6bc=', 'Content-Length': '0'} 2023-10-17T18:28:47.679 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000dfc6de1ef32987fc-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.680 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.680 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.681 DEBUG:boto:Method: GET 2023-10-17T18:28:47.681 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.682 DEBUG:boto:Data: 2023-10-17T18:28:47.682 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.683 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.683 DEBUG:boto:Params: {} 2023-10-17T18:28:47.684 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.684 DEBUG:boto:Token: None 2023-10-17T18:28:47.685 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.689 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx000007b05bbbafebabb7c-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.689 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.690 DEBUG:boto:Token: None 2023-10-17T18:28:47.690 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.691 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.694 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000641fe03a40f62db5-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.695 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.695 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.695 DEBUG:boto:Method: PUT 2023-10-17T18:28:47.696 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.696 DEBUG:boto:Data: 2023-10-17T18:28:47.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:28:47.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:28:47.697 DEBUG:boto:Port: 8000 2023-10-17T18:28:47.698 DEBUG:boto:Params: {} 2023-10-17T18:28:47.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:28:47.699 DEBUG:boto:Token: None 2023-10-17T18:28:47.699 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:25qSFRzJ/b8QdVk9heMa3HZQRD0= 2023-10-17T18:28:47.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:25qSFRzJ/b8QdVk9heMa3HZQRD0='} 2023-10-17T18:28:47.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8d1681c14d23d73-00652ed25f-1567-a1'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.713 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.713 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.714 DEBUG:boto:Method: GET 2023-10-17T18:28:47.714 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.715 DEBUG:boto:Data: 2023-10-17T18:28:47.715 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.716 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.716 DEBUG:boto:Params: {} 2023-10-17T18:28:47.717 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.717 DEBUG:boto:Token: None 2023-10-17T18:28:47.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.718 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.722 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx000008e8d065c5f078a00-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.722 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.723 DEBUG:boto:Token: None 2023-10-17T18:28:47.723 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.726 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx0000001449f147b3dbc60-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.727 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.728 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.728 DEBUG:boto:Method: PUT 2023-10-17T18:28:47.729 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.729 DEBUG:boto:Data: 2023-10-17T18:28:47.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:28:47.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:28:47.730 DEBUG:boto:Port: 8000 2023-10-17T18:28:47.731 DEBUG:boto:Params: {} 2023-10-17T18:28:47.731 DEBUG:boto:Token: None 2023-10-17T18:28:47.732 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v17Kp48C6wF6pKctRdr8yBZWilI= 2023-10-17T18:28:47.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v17Kp48C6wF6pKctRdr8yBZWilI='} 2023-10-17T18:28:47.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d6b30ae3ae778e5-00652ed25f-1567-a1'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.739 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.741 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.741 DEBUG:boto:Method: GET 2023-10-17T18:28:47.742 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.742 DEBUG:boto:Data: 2023-10-17T18:28:47.743 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.744 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.744 DEBUG:boto:Params: {} 2023-10-17T18:28:47.745 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.745 DEBUG:boto:Token: None 2023-10-17T18:28:47.746 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.746 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.749 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx0000083af3b86bc1f5afe-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.750 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.750 DEBUG:boto:Token: None 2023-10-17T18:28:47.751 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.751 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.754 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000002291dc64a1f1c94e-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.755 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.755 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.756 DEBUG:boto:Method: PUT 2023-10-17T18:28:47.757 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.758 DEBUG:boto:Data: 2023-10-17T18:28:47.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:28:47.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:28:47.760 DEBUG:boto:Port: 8000 2023-10-17T18:28:47.760 DEBUG:boto:Params: {} 2023-10-17T18:28:47.762 DEBUG:boto:Token: None 2023-10-17T18:28:47.763 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.763 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WU3fI4aZlvBs2PRVW9uzT43vDjw= 2023-10-17T18:28:47.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WU3fI4aZlvBs2PRVW9uzT43vDjw='} 2023-10-17T18:28:47.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ecff35d19a53884a-00652ed25f-1567-a1'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.770 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.771 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.771 DEBUG:boto:Method: GET 2023-10-17T18:28:47.772 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.772 DEBUG:boto:Data: 2023-10-17T18:28:47.772 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.773 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.773 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.774 DEBUG:boto:Params: {} 2023-10-17T18:28:47.774 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.775 DEBUG:boto:Token: None 2023-10-17T18:28:47.775 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.779 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx00000dd9657a17c506cf6-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.780 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.780 DEBUG:boto:Token: None 2023-10-17T18:28:47.781 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.783 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000f84c091d2ea88bda-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.826 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.827 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.827 DEBUG:boto:Method: PUT 2023-10-17T18:28:47.828 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.828 DEBUG:boto:Data: 2023-10-17T18:28:47.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:28:47.829 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:28:47.830 DEBUG:boto:Port: 8000 2023-10-17T18:28:47.830 DEBUG:boto:Params: {} 2023-10-17T18:28:47.830 DEBUG:boto:Token: None 2023-10-17T18:28:47.831 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.831 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RXwp/geu6ou9+Pu1PMijH0ToeJ8= 2023-10-17T18:28:47.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RXwp/geu6ou9+Pu1PMijH0ToeJ8='} 2023-10-17T18:28:47.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042a355b0d8116177-00652ed25f-1567-a1'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.838 DEBUG:boto:path=/hbmuog-35/testredirect 2023-10-17T18:28:47.839 DEBUG:boto:auth_path=/hbmuog-35/testredirect 2023-10-17T18:28:47.839 DEBUG:boto:Method: GET 2023-10-17T18:28:47.840 DEBUG:boto:Path: /hbmuog-35/testredirect 2023-10-17T18:28:47.840 DEBUG:boto:Data: 2023-10-17T18:28:47.840 DEBUG:boto:Headers: {} 2023-10-17T18:28:47.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:28:47.841 DEBUG:boto:Port: 8001 2023-10-17T18:28:47.842 DEBUG:boto:Params: {} 2023-10-17T18:28:47.842 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:28:47.843 DEBUG:boto:Token: None 2023-10-17T18:28:47.843 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.847 DEBUG:boto:Response headers: [('Location', 'http://smithi049.front.sepia.ceph.com:8000/hbmuog-35/testredirect?'), ('x-amz-request-id', 'tx000005533d7d7a87108c8-00652ed25f-156b-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.848 DEBUG:boto:Redirecting: http://smithi049.front.sepia.ceph.com/hbmuog-35/testredirect 2023-10-17T18:28:47.848 DEBUG:boto:Token: None 2023-10-17T18:28:47.849 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:28:47 GMT /hbmuog-35/testredirect 2023-10-17T18:28:47.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw= 2023-10-17T18:28:47.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:28:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HrIImGiYyPKQ+mQoqO0za1uSLrw=', 'Content-Length': '0'} 2023-10-17T18:28:47.852 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000420d6eeef7a766c9-00652ed25f-1567-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:28:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:28:47.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:28:47.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:28:47.853 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:28:47.909 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:28:47.910 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:28:47.932 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:28:47.932 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:28:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:28:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:28:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:28:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:28:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:28:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:28:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:28:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:28:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:28:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:28:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:28:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:28:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:28:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:28:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:28:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:28:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:28:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:28:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:28:47.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:47.937 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:28:47.937 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:28:47.937 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:28:47.937 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:28:47.938 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:28:47.938 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-17T18:28:47.938 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:28:47.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:47.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:47.939 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:28:47.939 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:28:47.939 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:28:47.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:47.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:47.940 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:47.940 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:28:47.940 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:28:47.940 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:28:47.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:28:47.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:28:47.942 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:28:47.994 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:28:47.994 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:28:48.040 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:28:48.065 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:28:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:28:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7, 2023-10-17T18:28:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:28:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:28:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:28:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:28:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:28:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:28:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:28:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:28:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:28:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:28:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:28:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:28:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:28:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:28:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:28:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:28:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:28:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:28:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:28:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:28:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:28:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:28:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:28:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:28:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:28:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:28:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:28:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:28:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:28:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:28:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:28:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:28:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:28:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:28:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:28:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:28:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:28:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:28:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:28:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:28:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:28:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:28:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:28:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:28:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:28:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:28:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:28:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:28:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:28:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:28:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:28:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:28:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:28:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:28:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:28:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:28:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:28:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:28:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:28:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:28:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:28:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:28:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:28:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:28:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:28:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:28:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:28:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:28:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:28:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:28:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:28:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:28:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:28:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:28:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:28:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:28:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:28:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:28:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:28:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:28:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:28:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:28:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:28:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:28:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:28:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:28:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:28:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:28:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:28:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:28:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:28:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:28:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:28:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:28:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:28:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:28:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:28:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:28:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:28:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:28:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:28:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:28:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:28:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:28:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:28:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:28:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:28:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:28:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:28:48.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.102 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:28:48.102 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:28:48.102 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:28:48.102 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:28:48.103 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:28:48.103 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:28:48.103 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:28:48.103 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:28:48.103 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:28:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:28:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:28:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:28:48.104 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:28:48.105 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:28:48.105 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:28:48.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.105 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:28:48.105 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:28:48.106 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:28:48.106 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:28:48.106 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:28:48.106 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:28:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:28:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:28:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:28:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:28:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:28:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:28:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:28:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:28:48.109 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:28:48.109 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:28:48.109 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:28:48.109 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:28:48.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:28:48.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:28:48.110 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:28:48.110 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:28:48.110 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:28:48.110 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:28:48.111 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-10-17T18:29:48.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:29:48.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:29:48.114 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:29:48.173 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:29:48.173 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:29:48.196 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:29:48.197 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:29:48.197 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:29:48.197 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-17T18:29:48.197 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "a2.rgw.control", 2023-10-17T18:29:48.197 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-17T18:29:48.198 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-17T18:29:48.198 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "a2.rgw.log", 2023-10-17T18:29:48.198 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-17T18:29:48.198 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-17T18:29:48.199 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-17T18:29:48.199 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-17T18:29:48.199 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-17T18:29:48.199 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-17T18:29:48.199 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-17T18:29:48.200 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-17T18:29:48.200 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "a2.rgw.otp", 2023-10-17T18:29:48.200 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:29:48.200 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:29:48.200 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:29:48.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.201 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:29:48.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.201 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:29:48.202 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:29:48.202 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-17T18:29:48.202 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:29:48.203 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:29:48.203 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-17T18:29:48.203 INFO:teuthology.orchestra.run.smithi049.stdout: "compression_type": "random" 2023-10-17T18:29:48.203 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.204 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-17T18:29:48.204 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:29:48.204 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:29:48.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.205 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.205 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:29:48.205 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-17T18:29:48.205 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:29:48.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:29:48.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:29:48.208 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:29:48.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:29:48.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:29:48.308 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:29:48.332 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:29:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:29:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8, 2023-10-17T18:29:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:29:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:29:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:29:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:29:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:29:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:29:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:29:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:29:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:29:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:29:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:29:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:29:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:29:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:29:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:29:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:29:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:29:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:29:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:29:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:29:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:29:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:29:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:29:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:29:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:29:48.341 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:29:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:29:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:29:48.342 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:29:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:29:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:29:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:29:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:29:48.343 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:29:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:29:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.344 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:29:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:29:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:29:48.345 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:29:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:29:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.346 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:29:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:29:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:29:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:29:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.347 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:29:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.348 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:29:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:29:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:29:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:29:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.349 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:29:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:29:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:29:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:29:48.350 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:29:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:29:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:29:48.351 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:29:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:29:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:29:48.352 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:29:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:29:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:29:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:29:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:29:48.353 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:29:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:29:48.354 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:29:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:29:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:29:48.356 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:29:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:29:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:29:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:29:48.357 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:29:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:29:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:29:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:29:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:29:48.358 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:29:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:29:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:29:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.359 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:29:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.360 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:29:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:29:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:29:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:29:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:29:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:29:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:29:48.362 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:29:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:29:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:29:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:29:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:29:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:29:48.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:29:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:29:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:29:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:29:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:29:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:29:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:29:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:29:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:29:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:29:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:29:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:29:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:29:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:29:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:29:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:29:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:29:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:29:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:29:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:29:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:29:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:29:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:29:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:29:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:29:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:29:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:29:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:29:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:29:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:29:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:29:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:29:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:29:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:29:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:29:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:29:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:29:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:29:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:29:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:29:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:29:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:29:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:29:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:29:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:29:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:29:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:29:48.377 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:29:48.377 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-10-17T18:30:48.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-10-17T18:30:48.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:30:48.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:30:48.382 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:30:48.441 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:48.442 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:48.476 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:30:48.490 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:30:48.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.490 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:30:48.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:27:16.394899Z" 2023-10-17T18:30:48.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.491 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:30:48.491 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:30:48.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.492 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:30:48.492 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:30:48.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:30:48.493 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:30:48.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.493 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:30:48.494 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:30:48.495 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:30:48.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:30:48.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:30:48.497 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:30:48.551 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:48.551 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:48.579+0000 7f9cd339e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:30:48.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:48.580+0000 7f9cd339e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:30:48.593 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:48.594 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:30:48.594 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:30:48.594 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:30:48.595 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:30:48.595 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:48.595 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:48.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.596 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:30:48.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.596 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:30:48.596 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:48.596 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:30:48.597 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:30:48.597 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:30:48.597 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:30:48.597 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:30:48.597 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:30:48.598 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:48.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:30:48.599 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:48.599 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:30:48.599 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:30:48.599 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:30:48.600 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:30:48.600 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:30:48.600 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:30:48.600 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:48.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.601 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:30:48.601 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:48.601 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:30:48.602 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:30:48.602 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:30:48.602 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:30:48.602 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:30:48.603 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:30:48.603 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:48.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.604 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:30:48.604 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:48.604 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:30:48.604 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:30:48.604 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:30:48.605 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:30:48.605 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:30:48.605 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:30:48.605 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:48.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:30:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:30:48.607 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:30:48.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.607 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:30:48Z" 2023-10-17T18:30:48.607 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:48.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:30:48Z" } 2023-10-17T18:30:48.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:30:48.609 DEBUG:rgw_multi.tests:log_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:30:48.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:30:48.610 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:30:48.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:48.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'DMAYVT4VX644XZ44', '--secret', '60Msyu4HAmpNQZQddMibqX6UaYh6P6FT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:48.612 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key DMAYVT4VX644XZ44 --secret 60Msyu4HAmpNQZQddMibqX6UaYh6P6FT --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:48.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:48.665 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:48.680 INFO:teuthology.orchestra.run.smithi049.stderr:NOTICE: set zone's realm_id=19b3e86c-cc39-4be9-99c4-971ddfc4a8c5 2023-10-17T18:30:48.694 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:48.694 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "620b31b6-4774-472a-9371-dd7206658861", 2023-10-17T18:30:48.695 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove", 2023-10-17T18:30:48.695 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-17T18:30:48.695 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "remove.rgw.control", 2023-10-17T18:30:48.695 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-17T18:30:48.695 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-17T18:30:48.696 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "remove.rgw.log", 2023-10-17T18:30:48.696 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-17T18:30:48.696 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-17T18:30:48.696 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-17T18:30:48.697 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-17T18:30:48.697 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-17T18:30:48.697 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-17T18:30:48.697 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-17T18:30:48.697 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-17T18:30:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "remove.rgw.otp", 2023-10-17T18:30:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:30:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "DMAYVT4VX644XZ44", 2023-10-17T18:30:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "60Msyu4HAmpNQZQddMibqX6UaYh6P6FT" 2023-10-17T18:30:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.699 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:30:48.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:48.699 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:30:48.699 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:48.699 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-17T18:30:48.700 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:30:48.700 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:30:48.700 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-17T18:30:48.700 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-17T18:30:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:30:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:30:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-17T18:30:48.702 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:48.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:48.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:48.705 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:48.754 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:48.754 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:48.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:48.765+0000 7ff2e6fe1780 0 period (cfc9d53d-2765-4da9-b7a8-37407f517c55 does not have zone 620b31b6-4774-472a-9371-dd7206658861 configured 2023-10-17T18:30:55.510 INFO:teuthology.orchestra.run.smithi049.stderr:Sending period to new master zone 4c84373e-d451-49bb-a8c9-c37d5e5e2160 2023-10-17T18:30:55.536 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:55.536 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:55.536 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9, 2023-10-17T18:30:55.537 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:30:55.537 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:30:55.537 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:30:55.537 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:55.538 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:30:55.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.538 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:30:55.541 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:30:55.541 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:55.542 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:30:55.542 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.542 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.542 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.543 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:55.543 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:55.543 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.543 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:55.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.544 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.544 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:30:55.544 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.544 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.545 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.545 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.545 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.545 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.546 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.546 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.546 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.546 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.546 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.547 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.547 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.547 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.548 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "620b31b6-4774-472a-9371-dd7206658861", 2023-10-17T18:30:55.548 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove", 2023-10-17T18:30:55.549 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [], 2023-10-17T18:30:55.549 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.549 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.549 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.549 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.550 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.550 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.550 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.550 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.551 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.551 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.551 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.552 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:30:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:30:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.554 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.554 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.554 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.554 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.555 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.555 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.555 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.555 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.555 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:55.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.557 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:55.557 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:55.557 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:55.557 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:55.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.558 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.558 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:55.558 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:55.559 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:55.559 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:55.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.559 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:55.559 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.560 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:30:55.560 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:30:55.561 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:55.561 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:55.561 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.561 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.562 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.562 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:55.562 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:55.562 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:55.562 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:55.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.563 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:55.563 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:30:55.563 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.563 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:30:55.564 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.564 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.564 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.564 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.565 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.565 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.565 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.565 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.565 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.566 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.566 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.566 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.566 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.567 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:55.567 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:30:55.567 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.568 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.568 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.568 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.568 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.568 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.569 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.569 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.569 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.569 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.570 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.570 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.570 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.570 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.570 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.571 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.571 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:55.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.571 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:55.572 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:55.572 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:55.572 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:55.572 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.573 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.573 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:55.573 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:55.573 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:55.574 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:55.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.574 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:55.574 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:30:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:55.576 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:30:55.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.576 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.577 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:30:55.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.577 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "620b31b6-4774-472a-9371-dd7206658861", 2023-10-17T18:30:55.577 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 2621868951 2023-10-17T18:30:55.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.578 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:55.578 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:30:55.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.579 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:55.579 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:30:55.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.580 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.580 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:30:55.580 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.580 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:30:55.581 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:30:55.581 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:55.581 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:55.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:55.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:55.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:55.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.583 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:30:55.583 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:55.584 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:55.584 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:55.584 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:55.584 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:55.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.585 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:30:55.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:55.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:55.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:55.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:55.586 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:55.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.586 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:30:55.586 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:55.586 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:30:55.587 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:55.588 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:55.588 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:55.588 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:55.588 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:55.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.589 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:55.590 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:30:55.591 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:55.591 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:55.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.593 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:55.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:55.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:55.671 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:55.671 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:30:55.671 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.672 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:55.673 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:55.673 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.673 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:55.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.674 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:55.674 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:30:55.674 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.674 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:55.674 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.675 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.675 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.675 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.675 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.675 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.676 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.676 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.676 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.676 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.676 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.677 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.677 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.677 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:55.678 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:30:55.678 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:55.678 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:30:55.678 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.679 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:55.679 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:55.679 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:55.679 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:55.679 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:55.680 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:55.680 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:55.680 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:55.680 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:55.680 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:55.681 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.681 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.681 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.681 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:55.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:55.682 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:55.682 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:55.682 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:55.682 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:55.683 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.683 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:55.683 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:55.683 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:55.683 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:55.684 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:55.684 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:55.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:55.684 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:55.684 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:55.685 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:55.685 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:55.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.687 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:55.750 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:55.750 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:55.778 INFO:teuthology.orchestra.run.smithi049.stderr:zone name remove not found in zonegroup a 2023-10-17T18:30:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-17T18:30:55.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.785 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:55.847 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:55.848 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:55.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:55.909 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:55.981 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:55.981 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:56.001+0000 7ff74c91e780 0 WARNING: can't generate connection for zone 620b31b6-4774-472a-9371-dd7206658861 id remove: no endpoints defined 2023-10-17T18:30:56.054 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:56.054 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10, 2023-10-17T18:30:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:30:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:30:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:30:56.055 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:30:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:30:56.056 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:30:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:30:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.057 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.058 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:30:56.059 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.060 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:30:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.063 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:30:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.064 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.065 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.066 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.067 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:56.068 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.069 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.070 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.071 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:30:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:30:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:56.072 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.073 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.074 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:30:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:30:56.075 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.076 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.077 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.078 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.079 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.080 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:30:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.081 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.083 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.084 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.085 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:56.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.087 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.088 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.089 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:30:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:30:56.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:30:56.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:30:56.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:30:56.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.094 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.094 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:30:56.094 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.095 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:30:56.095 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:30:56.095 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:56.095 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:56.096 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:56.096 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:56.096 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:56.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.096 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:30:56.097 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:56.097 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:56.097 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:56.097 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:56.098 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:56.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.098 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:30:56.098 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.098 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:30:56.100 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.100 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.100 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.100 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.100 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:30:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.103 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.103 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:30:56.103 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:56.104 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:56.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-10-17T18:30:56.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.106 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.171 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.199 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:56.200 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cdeca9c7-380a-46c2-a79d-8458babe25d6", 2023-10-17T18:30:56.200 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove_zg", 2023-10-17T18:30:56.200 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "remove_zg", 2023-10-17T18:30:56.201 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:56.201 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [], 2023-10-17T18:30:56.201 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.201 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.202 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "", 2023-10-17T18:30:56.202 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [], 2023-10-17T18:30:56.202 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [] 2023-10-17T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.204 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.204 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.205 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.205 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.206 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.206 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.207 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.207 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:56.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.210 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.271 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:56.339+0000 7f4a1d7e5780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-10-17T18:30:56.345 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:56.346 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "87e35ae4-1bc4-4369-abfb-54cc620c8d93", 2023-10-17T18:30:56.346 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove", 2023-10-17T18:30:56.346 INFO:teuthology.orchestra.run.smithi049.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-17T18:30:56.346 INFO:teuthology.orchestra.run.smithi049.stdout: "control_pool": "remove.rgw.control", 2023-10-17T18:30:56.347 INFO:teuthology.orchestra.run.smithi049.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-17T18:30:56.347 INFO:teuthology.orchestra.run.smithi049.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-17T18:30:56.347 INFO:teuthology.orchestra.run.smithi049.stdout: "log_pool": "remove.rgw.log", 2023-10-17T18:30:56.347 INFO:teuthology.orchestra.run.smithi049.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-17T18:30:56.348 INFO:teuthology.orchestra.run.smithi049.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-17T18:30:56.348 INFO:teuthology.orchestra.run.smithi049.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-17T18:30:56.348 INFO:teuthology.orchestra.run.smithi049.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-17T18:30:56.348 INFO:teuthology.orchestra.run.smithi049.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-17T18:30:56.348 INFO:teuthology.orchestra.run.smithi049.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-17T18:30:56.349 INFO:teuthology.orchestra.run.smithi049.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-17T18:30:56.349 INFO:teuthology.orchestra.run.smithi049.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-17T18:30:56.349 INFO:teuthology.orchestra.run.smithi049.stdout: "otp_pool": "remove.rgw.otp", 2023-10-17T18:30:56.349 INFO:teuthology.orchestra.run.smithi049.stdout: "system_key": { 2023-10-17T18:30:56.349 INFO:teuthology.orchestra.run.smithi049.stdout: "access_key": "", 2023-10-17T18:30:56.350 INFO:teuthology.orchestra.run.smithi049.stdout: "secret_key": "" 2023-10-17T18:30:56.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.350 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_pools": [ 2023-10-17T18:30:56.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.350 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "default-placement", 2023-10-17T18:30:56.351 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:30:56.351 INFO:teuthology.orchestra.run.smithi049.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-17T18:30:56.351 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": { 2023-10-17T18:30:56.351 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD": { 2023-10-17T18:30:56.353 INFO:teuthology.orchestra.run.smithi049.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-17T18:30:56.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.354 INFO:teuthology.orchestra.run.smithi049.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-17T18:30:56.354 INFO:teuthology.orchestra.run.smithi049.stdout: "index_type": 0, 2023-10-17T18:30:56.354 INFO:teuthology.orchestra.run.smithi049.stdout: "inline_data": true 2023-10-17T18:30:56.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.355 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.355 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.355 INFO:teuthology.orchestra.run.smithi049.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-17T18:30:56.355 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:56.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.358 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.422 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.423 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.524 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 11, 2023-10-17T18:30:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:30:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:30:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:30:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:30:56.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.526 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:30:56.526 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:30:56.526 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:30:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:30:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.530 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.530 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.533 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.533 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.533 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:30:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:30:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.542 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.542 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.542 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.543 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.543 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.543 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.544 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:30:56.544 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:30:56.544 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:56.545 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:56.545 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.545 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.545 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.546 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.546 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.546 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.546 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:56.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.547 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:56.547 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:30:56.547 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.547 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:30:56.548 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.548 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.548 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.548 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.549 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.549 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.549 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.549 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.549 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.550 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.550 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.550 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.550 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.551 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.551 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:30:56.551 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:56.552 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:56.552 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.552 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.552 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:56.553 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.553 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.553 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.553 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.553 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.554 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.554 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.554 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.554 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.555 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.555 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.555 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.556 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.556 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.556 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:56.556 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:56.556 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.557 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.557 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.557 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.557 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.558 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.558 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.558 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.558 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.559 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.559 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cdeca9c7-380a-46c2-a79d-8458babe25d6", 2023-10-17T18:30:56.559 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove_zg", 2023-10-17T18:30:56.560 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "remove_zg", 2023-10-17T18:30:56.560 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:56.560 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [], 2023-10-17T18:30:56.560 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:56.561 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:56.561 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "87e35ae4-1bc4-4369-abfb-54cc620c8d93", 2023-10-17T18:30:56.561 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:56.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.561 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "87e35ae4-1bc4-4369-abfb-54cc620c8d93", 2023-10-17T18:30:56.562 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "remove", 2023-10-17T18:30:56.562 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [], 2023-10-17T18:30:56.562 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:56.562 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": false, 2023-10-17T18:30:56.562 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:56.563 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:56.563 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:56.563 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:56.563 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:56.563 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:56.564 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:56.564 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:56.564 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.564 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.565 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.565 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:56.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.565 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:56.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:56.566 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:56.566 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:56.566 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.567 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.567 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:56.567 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.567 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:56.568 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:56.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.568 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:56.568 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:56.568 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.569 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:56.569 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:30:56.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.570 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:56.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:30:56.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.570 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.571 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:30:56.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.571 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "87e35ae4-1bc4-4369-abfb-54cc620c8d93", 2023-10-17T18:30:56.571 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1796254250 2023-10-17T18:30:56.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.572 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:56.572 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:30:56.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:56.573 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:56.573 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:30:56.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:56.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.574 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:30:56.574 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:56.574 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:30:56.576 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:30:56.576 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:56.576 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:56.576 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:30:56.577 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:56.578 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:56.578 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:56.578 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:56.578 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:56.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.579 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:30:56.579 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.579 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.580 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.580 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.580 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.580 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:30:56.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.581 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.582 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:30:56.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:56.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:56.583 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:56.583 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:56.583 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:56.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:56.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:56.584 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:56.584 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:30:56.584 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:56.584 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:56.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.586 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.648 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:56.683+0000 7f40d2512780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-10-17T18:30:56.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:56.685+0000 7f40d2512780 0 Removed zone from zonegroup remove_zg 2023-10-17T18:30:56.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.698 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.764 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:30:56.797+0000 7fec629a9780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-10-17T18:30:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-17T18:30:56.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.807 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.870 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.870 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:56.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:30:56.906 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:30:56.983 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:30:56.983 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:30:57.077 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:30:57.078 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:57.078 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12, 2023-10-17T18:30:57.078 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T18:30:57.078 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T18:30:57.079 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T18:30:57.079 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:30:57.079 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T18:30:57.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.080 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T18:30:57.080 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T18:30:57.080 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:57.080 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T18:30:57.080 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.081 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:57.081 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.081 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:57.081 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.082 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:57.083 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T18:30:57.083 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.083 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T18:30:57.083 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.083 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:57.084 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:57.084 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:57.084 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:57.084 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:57.085 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:57.085 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:57.085 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:57.085 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:57.086 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:57.086 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.087 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:57.087 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T18:30:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T18:30:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:57.088 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:57.090 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:57.090 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:57.090 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:57.090 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:57.090 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.091 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.091 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.091 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:57.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.093 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.093 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:57.093 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:57.094 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:57.094 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:57.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.094 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:57.095 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.096 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T18:30:57.096 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T18:30:57.096 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T18:30:57.096 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T18:30:57.096 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.097 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:57.097 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T18:30:57.097 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T18:30:57.098 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:57.098 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T18:30:57.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.098 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:57.098 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T18:30:57.099 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.099 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T18:30:57.099 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.099 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:57.100 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:57.100 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:57.100 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:57.100 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:57.100 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:57.102 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:57.102 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:57.102 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:57.102 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:57.104 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.104 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T18:30:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T18:30:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T18:30:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.106 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T18:30:57.106 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T18:30:57.106 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T18:30:57.106 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T18:30:57.107 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T18:30:57.107 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T18:30:57.107 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T18:30:57.107 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T18:30:57.108 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T18:30:57.108 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T18:30:57.108 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.109 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.109 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T18:30:57.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.110 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T18:30:57.110 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T18:30:57.110 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T18:30:57.110 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T18:30:57.111 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.111 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.111 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T18:30:57.111 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:57.112 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T18:30:57.112 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T18:30:57.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.112 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T18:30:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T18:30:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T18:30:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T18:30:57.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.114 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T18:30:57.114 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T18:30:57.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.115 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:57.115 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T18:30:57.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.116 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:30:57.116 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T18:30:57.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:30:57.117 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T18:30:57.117 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T18:30:57.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.117 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:30:57.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.118 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T18:30:57.118 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:30:57.118 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T18:30:57.118 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T18:30:57.119 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:57.119 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:57.119 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:57.119 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:57.120 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:57.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.120 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T18:30:57.120 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T18:30:57.120 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T18:30:57.121 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T18:30:57.121 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T18:30:57.121 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T18:30:57.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.122 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T18:30:57.122 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:57.122 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:57.122 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:57.122 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:57.123 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:57.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.123 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T18:30:57.123 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:57.124 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:57.124 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:57.124 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:57.124 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:57.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.125 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T18:30:57.125 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T18:30:57.125 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T18:30:57.125 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T18:30:57.126 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T18:30:57.126 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T18:30:57.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:30:57.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:30:57.126 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T18:30:57.127 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T18:30:57.127 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:30:57.127 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:30:57.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-10-17T18:30:57.129 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-36 2023-10-17T18:30:57.129 DEBUG:boto:path=/hbmuog-36/ 2023-10-17T18:30:57.130 DEBUG:boto:auth_path=/hbmuog-36/ 2023-10-17T18:30:57.130 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.130 DEBUG:boto:Path: /hbmuog-36/ 2023-10-17T18:30:57.131 DEBUG:boto:Data: 2023-10-17T18:30:57.131 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.132 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:30:57.132 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.132 DEBUG:boto:Params: {} 2023-10-17T18:30:57.133 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:30:57.134 DEBUG:boto:Token: None 2023-10-17T18:30:57.135 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-36/ 2023-10-17T18:30:57.135 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Li2bxwG/M1ZkSm+ERXMnKCZBTKA= 2023-10-17T18:30:57.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Li2bxwG/M1ZkSm+ERXMnKCZBTKA=', 'Content-Length': '0'} 2023-10-17T18:30:57.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb1d99aa9c346e16-00652ed2e1-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.339 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-37 2023-10-17T18:30:57.340 DEBUG:boto:path=/hbmuog-37/ 2023-10-17T18:30:57.340 DEBUG:boto:auth_path=/hbmuog-37/ 2023-10-17T18:30:57.341 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.341 DEBUG:boto:Path: /hbmuog-37/ 2023-10-17T18:30:57.342 DEBUG:boto:Data: 2023-10-17T18:30:57.342 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.343 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:30:57.343 DEBUG:boto:Port: 8001 2023-10-17T18:30:57.344 DEBUG:boto:Params: {} 2023-10-17T18:30:57.344 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:30:57.346 DEBUG:boto:Token: None 2023-10-17T18:30:57.346 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-37/ 2023-10-17T18:30:57.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1NEJr9z1S9Ee4hqfYY9xJ1hQaOg= 2023-10-17T18:30:57.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1NEJr9z1S9Ee4hqfYY9xJ1hQaOg=', 'Content-Length': '0'} 2023-10-17T18:30:57.451 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c4e4336bb04f195-00652ed2e1-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.452 INFO:rgw_multi.tests:create bucket zone=b1 name=hbmuog-38 2023-10-17T18:30:57.452 DEBUG:boto:path=/hbmuog-38/ 2023-10-17T18:30:57.453 DEBUG:boto:auth_path=/hbmuog-38/ 2023-10-17T18:30:57.453 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.453 DEBUG:boto:Path: /hbmuog-38/ 2023-10-17T18:30:57.454 DEBUG:boto:Data: 2023-10-17T18:30:57.454 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.455 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:30:57.455 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.455 DEBUG:boto:Params: {} 2023-10-17T18:30:57.456 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:30:57.456 DEBUG:boto:Token: None 2023-10-17T18:30:57.457 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-38/ 2023-10-17T18:30:57.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i+zeI0/6s9+bhvfZYWiZab3ysC0= 2023-10-17T18:30:57.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i+zeI0/6s9+bhvfZYWiZab3ysC0=', 'Content-Length': '0'} 2023-10-17T18:30:57.497 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e20a563ba12119b1-00652ed2e1-1161-b1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.498 INFO:rgw_multi.tests:create bucket zone=b2 name=hbmuog-39 2023-10-17T18:30:57.498 DEBUG:boto:path=/hbmuog-39/ 2023-10-17T18:30:57.499 DEBUG:boto:auth_path=/hbmuog-39/ 2023-10-17T18:30:57.499 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.500 DEBUG:boto:Path: /hbmuog-39/ 2023-10-17T18:30:57.500 DEBUG:boto:Data: 2023-10-17T18:30:57.501 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.502 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:30:57.502 DEBUG:boto:Port: 8001 2023-10-17T18:30:57.503 DEBUG:boto:Params: {} 2023-10-17T18:30:57.503 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:30:57.504 DEBUG:boto:Token: None 2023-10-17T18:30:57.504 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-39/ 2023-10-17T18:30:57.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pljUetcSM8mJhKUo4vLXXRZ5yJg= 2023-10-17T18:30:57.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pljUetcSM8mJhKUo4vLXXRZ5yJg=', 'Content-Length': '0'} 2023-10-17T18:30:57.540 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000048e2b7e115b17b-00652ed2e1-1165-b2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.541 DEBUG:boto:path=/hbmuog-36/ 2023-10-17T18:30:57.541 DEBUG:boto:auth_path=/hbmuog-36/ 2023-10-17T18:30:57.542 DEBUG:boto:path=/hbmuog-36/?website 2023-10-17T18:30:57.542 DEBUG:boto:auth_path=/hbmuog-36/?website 2023-10-17T18:30:57.543 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.543 DEBUG:boto:Path: /hbmuog-36/?website 2023-10-17T18:30:57.543 DEBUG:boto:Data: index.htmlerror.html 2023-10-17T18:30:57.544 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.544 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:30:57.545 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.545 DEBUG:boto:Params: {} 2023-10-17T18:30:57.546 DEBUG:boto:Token: None 2023-10-17T18:30:57.546 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-36/?website 2023-10-17T18:30:57.546 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iBHO4SDvK2FTUhEX49s7E74OsKI= 2023-10-17T18:30:57.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iBHO4SDvK2FTUhEX49s7E74OsKI=', 'Content-Length': '241'} 2023-10-17T18:30:57.548 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx000007a417f33db0826ad-00652ed2e1-162c-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-10-17T18:30:57.550 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-40 2023-10-17T18:30:57.550 DEBUG:boto:path=/hbmuog-40/ 2023-10-17T18:30:57.551 DEBUG:boto:auth_path=/hbmuog-40/ 2023-10-17T18:30:57.551 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.552 DEBUG:boto:Path: /hbmuog-40/ 2023-10-17T18:30:57.552 DEBUG:boto:Data: 2023-10-17T18:30:57.552 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:30:57.553 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.554 DEBUG:boto:Params: {} 2023-10-17T18:30:57.554 DEBUG:boto:Token: None 2023-10-17T18:30:57.555 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-40/ 2023-10-17T18:30:57.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SLE7a1zBWNrmpZVQqVt+Emk7CVk= 2023-10-17T18:30:57.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLE7a1zBWNrmpZVQqVt+Emk7CVk=', 'Content-Length': '0'} 2023-10-17T18:30:57.572 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8d8c4aa7b4f774a-00652ed2e1-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.573 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-41 2023-10-17T18:30:57.573 DEBUG:boto:path=/hbmuog-41/ 2023-10-17T18:30:57.573 DEBUG:boto:auth_path=/hbmuog-41/ 2023-10-17T18:30:57.574 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.574 DEBUG:boto:Path: /hbmuog-41/ 2023-10-17T18:30:57.575 DEBUG:boto:Data: 2023-10-17T18:30:57.575 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.575 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:30:57.576 DEBUG:boto:Port: 8001 2023-10-17T18:30:57.576 DEBUG:boto:Params: {} 2023-10-17T18:30:57.577 DEBUG:boto:Token: None 2023-10-17T18:30:57.578 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-41/ 2023-10-17T18:30:57.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lo5uXxzYS09MDhCuN7m6tn140RE= 2023-10-17T18:30:57.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lo5uXxzYS09MDhCuN7m6tn140RE=', 'Content-Length': '0'} 2023-10-17T18:30:57.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c4544b33e2e2572-00652ed2e1-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.608 INFO:rgw_multi.tests:create bucket zone=b1 name=hbmuog-42 2023-10-17T18:30:57.608 DEBUG:boto:path=/hbmuog-42/ 2023-10-17T18:30:57.609 DEBUG:boto:auth_path=/hbmuog-42/ 2023-10-17T18:30:57.609 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.610 DEBUG:boto:Path: /hbmuog-42/ 2023-10-17T18:30:57.610 DEBUG:boto:Data: 2023-10-17T18:30:57.610 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.611 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:30:57.611 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.612 DEBUG:boto:Params: {} 2023-10-17T18:30:57.612 DEBUG:boto:Token: None 2023-10-17T18:30:57.613 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-42/ 2023-10-17T18:30:57.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g2rvNox9tR9ZWfeKYDI9qjx8Evk= 2023-10-17T18:30:57.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g2rvNox9tR9ZWfeKYDI9qjx8Evk=', 'Content-Length': '0'} 2023-10-17T18:30:57.641 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2dd533ff5979bc0-00652ed2e1-1161-b1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.642 INFO:rgw_multi.tests:create bucket zone=b2 name=hbmuog-43 2023-10-17T18:30:57.643 DEBUG:boto:path=/hbmuog-43/ 2023-10-17T18:30:57.643 DEBUG:boto:auth_path=/hbmuog-43/ 2023-10-17T18:30:57.643 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.644 DEBUG:boto:Path: /hbmuog-43/ 2023-10-17T18:30:57.644 DEBUG:boto:Data: 2023-10-17T18:30:57.645 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.645 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:30:57.645 DEBUG:boto:Port: 8001 2023-10-17T18:30:57.646 DEBUG:boto:Params: {} 2023-10-17T18:30:57.646 DEBUG:boto:Token: None 2023-10-17T18:30:57.647 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-43/ 2023-10-17T18:30:57.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bu8nfS9kf3WK/DRoF6sLHAd8YGA= 2023-10-17T18:30:57.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bu8nfS9kf3WK/DRoF6sLHAd8YGA=', 'Content-Length': '0'} 2023-10-17T18:30:57.677 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6de4581f9d55c86-00652ed2e1-1165-b2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.678 DEBUG:boto:path=/hbmuog-40/ 2023-10-17T18:30:57.678 DEBUG:boto:auth_path=/hbmuog-40/ 2023-10-17T18:30:57.679 DEBUG:boto:path=/hbmuog-40/?policy 2023-10-17T18:30:57.679 DEBUG:boto:auth_path=/hbmuog-40/?policy 2023-10-17T18:30:57.680 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.681 DEBUG:boto:Path: /hbmuog-40/?policy 2023-10-17T18:30:57.681 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-17T18:30:57.682 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:30:57.683 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.684 DEBUG:boto:Params: {} 2023-10-17T18:30:57.684 DEBUG:boto:Token: None 2023-10-17T18:30:57.685 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-40/?policy 2023-10-17T18:30:57.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U+8OYBQDLOsYKTNmsgggSHVH23A= 2023-10-17T18:30:57.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U+8OYBQDLOsYKTNmsgggSHVH23A=', 'Content-Length': '97'} 2023-10-17T18:30:57.692 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c67bb4392cfe9f8e-00652ed2e1-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.693 DEBUG:boto:path=/hbmuog-40/ 2023-10-17T18:30:57.693 DEBUG:boto:auth_path=/hbmuog-40/ 2023-10-17T18:30:57.694 DEBUG:boto:path=/hbmuog-40/?policy 2023-10-17T18:30:57.694 DEBUG:boto:auth_path=/hbmuog-40/?policy 2023-10-17T18:30:57.695 DEBUG:boto:Method: GET 2023-10-17T18:30:57.695 DEBUG:boto:Path: /hbmuog-40/?policy 2023-10-17T18:30:57.695 DEBUG:boto:Data: 2023-10-17T18:30:57.696 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:30:57.697 DEBUG:boto:Port: 8000 2023-10-17T18:30:57.697 DEBUG:boto:Params: {} 2023-10-17T18:30:57.698 DEBUG:boto:Token: None 2023-10-17T18:30:57.698 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-40/?policy 2023-10-17T18:30:57.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:URnWfmK37orKGN8rOw/uFOG4z/E= 2023-10-17T18:30:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:URnWfmK37orKGN8rOw/uFOG4z/E=', 'Content-Length': '0'} 2023-10-17T18:30:57.701 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1a7585ffc9b6f2a-00652ed2e1-162c-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 17 Oct 2023 18:30:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:57.701 DEBUG:boto:path=/hbmuog-41/ 2023-10-17T18:30:57.702 DEBUG:boto:auth_path=/hbmuog-41/ 2023-10-17T18:30:57.702 DEBUG:boto:path=/hbmuog-41/?policy 2023-10-17T18:30:57.703 DEBUG:boto:auth_path=/hbmuog-41/?policy 2023-10-17T18:30:57.703 DEBUG:boto:Method: PUT 2023-10-17T18:30:57.703 DEBUG:boto:Path: /hbmuog-41/?policy 2023-10-17T18:30:57.704 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-17T18:30:57.704 DEBUG:boto:Headers: {} 2023-10-17T18:30:57.705 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:30:57.705 DEBUG:boto:Port: 8001 2023-10-17T18:30:57.705 DEBUG:boto:Params: {} 2023-10-17T18:30:57.706 DEBUG:boto:Token: None 2023-10-17T18:30:57.706 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:57 GMT /hbmuog-41/?policy 2023-10-17T18:30:57.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EQrY97YlHBxsxupIJC2r4hGoM18= 2023-10-17T18:30:57.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EQrY97YlHBxsxupIJC2r4hGoM18=', 'Content-Length': '97'} 2023-10-17T18:30:58.717 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f69438251f96f8c1-00652ed2e1-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:30:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:58.718 DEBUG:boto:path=/hbmuog-41/ 2023-10-17T18:30:58.719 DEBUG:boto:auth_path=/hbmuog-41/ 2023-10-17T18:30:58.719 DEBUG:boto:path=/hbmuog-41/?policy 2023-10-17T18:30:58.719 DEBUG:boto:auth_path=/hbmuog-41/?policy 2023-10-17T18:30:58.720 DEBUG:boto:Method: GET 2023-10-17T18:30:58.720 DEBUG:boto:Path: /hbmuog-41/?policy 2023-10-17T18:30:58.721 DEBUG:boto:Data: 2023-10-17T18:30:58.721 DEBUG:boto:Headers: {} 2023-10-17T18:30:58.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:30:58.722 DEBUG:boto:Port: 8001 2023-10-17T18:30:58.722 DEBUG:boto:Params: {} 2023-10-17T18:30:58.723 DEBUG:boto:Token: None 2023-10-17T18:30:58.724 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:30:58 GMT /hbmuog-41/?policy 2023-10-17T18:30:58.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KuM8r1Py9/cmIdSFTrJdPJaeDFU= 2023-10-17T18:30:58.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KuM8r1Py9/cmIdSFTrJdPJaeDFU=', 'Content-Length': '0'} 2023-10-17T18:30:58.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac014bb2621ef6b8-00652ed2e2-1630-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 17 Oct 2023 18:30:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:58.727 DEBUG:boto:path=/hbmuog-42/ 2023-10-17T18:30:58.728 DEBUG:boto:auth_path=/hbmuog-42/ 2023-10-17T18:30:58.728 DEBUG:boto:path=/hbmuog-42/?policy 2023-10-17T18:30:58.729 DEBUG:boto:auth_path=/hbmuog-42/?policy 2023-10-17T18:30:58.729 DEBUG:boto:Method: PUT 2023-10-17T18:30:58.730 DEBUG:boto:Path: /hbmuog-42/?policy 2023-10-17T18:30:58.730 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-17T18:30:58.730 DEBUG:boto:Headers: {} 2023-10-17T18:30:58.731 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:30:58.731 DEBUG:boto:Port: 8000 2023-10-17T18:30:58.732 DEBUG:boto:Params: {} 2023-10-17T18:30:58.732 DEBUG:boto:Token: None 2023-10-17T18:30:58.733 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:58 GMT /hbmuog-42/?policy 2023-10-17T18:30:58.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giSB1a6nSdy1MTSLuBu7Ddhqvmo= 2023-10-17T18:30:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:58 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giSB1a6nSdy1MTSLuBu7Ddhqvmo=', 'Content-Length': '97'} 2023-10-17T18:30:59.745 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077ec32b4883ac8ed-00652ed2e2-1161-b1'), ('Date', 'Tue, 17 Oct 2023 18:30:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:59.746 DEBUG:boto:path=/hbmuog-42/ 2023-10-17T18:30:59.746 DEBUG:boto:auth_path=/hbmuog-42/ 2023-10-17T18:30:59.747 DEBUG:boto:path=/hbmuog-42/?policy 2023-10-17T18:30:59.747 DEBUG:boto:auth_path=/hbmuog-42/?policy 2023-10-17T18:30:59.747 DEBUG:boto:Method: GET 2023-10-17T18:30:59.748 DEBUG:boto:Path: /hbmuog-42/?policy 2023-10-17T18:30:59.748 DEBUG:boto:Data: 2023-10-17T18:30:59.749 DEBUG:boto:Headers: {} 2023-10-17T18:30:59.749 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8000 2023-10-17T18:30:59.749 DEBUG:boto:Port: 8000 2023-10-17T18:30:59.750 DEBUG:boto:Params: {} 2023-10-17T18:30:59.750 DEBUG:boto:Token: None 2023-10-17T18:30:59.751 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:30:59 GMT /hbmuog-42/?policy 2023-10-17T18:30:59.751 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f6h7dMIldcVoDS/4P4pS1cAhyMw= 2023-10-17T18:30:59.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:59 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f6h7dMIldcVoDS/4P4pS1cAhyMw=', 'Content-Length': '0'} 2023-10-17T18:30:59.753 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071fa95baad0f7490-00652ed2e3-1161-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 17 Oct 2023 18:30:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:30:59.754 DEBUG:boto:path=/hbmuog-43/ 2023-10-17T18:30:59.754 DEBUG:boto:auth_path=/hbmuog-43/ 2023-10-17T18:30:59.754 DEBUG:boto:path=/hbmuog-43/?policy 2023-10-17T18:30:59.755 DEBUG:boto:auth_path=/hbmuog-43/?policy 2023-10-17T18:30:59.755 DEBUG:boto:Method: PUT 2023-10-17T18:30:59.756 DEBUG:boto:Path: /hbmuog-43/?policy 2023-10-17T18:30:59.756 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-17T18:30:59.756 DEBUG:boto:Headers: {} 2023-10-17T18:30:59.758 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:30:59.758 DEBUG:boto:Port: 8001 2023-10-17T18:30:59.759 DEBUG:boto:Params: {} 2023-10-17T18:30:59.759 DEBUG:boto:Token: None 2023-10-17T18:30:59.760 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:30:59 GMT /hbmuog-43/?policy 2023-10-17T18:30:59.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vijcO7RhotUnExXaBVuFjR7RtAk= 2023-10-17T18:30:59.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:30:59 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vijcO7RhotUnExXaBVuFjR7RtAk=', 'Content-Length': '97'} 2023-10-17T18:31:00.772 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bfc1514f5a92b9b-00652ed2e3-1165-b2'), ('Date', 'Tue, 17 Oct 2023 18:31:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:31:00.772 DEBUG:boto:path=/hbmuog-43/ 2023-10-17T18:31:00.773 DEBUG:boto:auth_path=/hbmuog-43/ 2023-10-17T18:31:00.773 DEBUG:boto:path=/hbmuog-43/?policy 2023-10-17T18:31:00.774 DEBUG:boto:auth_path=/hbmuog-43/?policy 2023-10-17T18:31:00.774 DEBUG:boto:Method: GET 2023-10-17T18:31:00.774 DEBUG:boto:Path: /hbmuog-43/?policy 2023-10-17T18:31:00.775 DEBUG:boto:Data: 2023-10-17T18:31:00.775 DEBUG:boto:Headers: {} 2023-10-17T18:31:00.775 DEBUG:boto:Host: smithi071.front.sepia.ceph.com:8001 2023-10-17T18:31:00.776 DEBUG:boto:Port: 8001 2023-10-17T18:31:00.776 DEBUG:boto:Params: {} 2023-10-17T18:31:00.777 DEBUG:boto:Token: None 2023-10-17T18:31:00.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:31:00 GMT /hbmuog-43/?policy 2023-10-17T18:31:00.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o77nhIWT0w1imUjTudpep8Ee14Q= 2023-10-17T18:31:00.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:31:00 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o77nhIWT0w1imUjTudpep8Ee14Q=', 'Content-Length': '0'} 2023-10-17T18:31:00.780 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1db91ae2506d982-00652ed2e4-1165-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 17 Oct 2023 18:31:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:31:00.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-10-17T18:31:00.781 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-44 2023-10-17T18:31:00.782 DEBUG:boto:path=/hbmuog-44/ 2023-10-17T18:31:00.782 DEBUG:boto:auth_path=/hbmuog-44/ 2023-10-17T18:31:00.782 DEBUG:boto:Method: PUT 2023-10-17T18:31:00.783 DEBUG:boto:Path: /hbmuog-44/ 2023-10-17T18:31:00.783 DEBUG:boto:Data: 2023-10-17T18:31:00.784 DEBUG:boto:Headers: {} 2023-10-17T18:31:00.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:31:00.784 DEBUG:boto:Port: 8000 2023-10-17T18:31:00.785 DEBUG:boto:Params: {} 2023-10-17T18:31:00.785 DEBUG:boto:Token: None 2023-10-17T18:31:00.786 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:31:00 GMT /hbmuog-44/ 2023-10-17T18:31:00.786 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Ofv0cTP4Tfpuz2fdt7V/OjQMcs= 2023-10-17T18:31:00.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:31:00 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Ofv0cTP4Tfpuz2fdt7V/OjQMcs=', 'Content-Length': '0'} 2023-10-17T18:31:00.805 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b722b15430c848f7-00652ed2e4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:31:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:31:00.805 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-45 2023-10-17T18:31:00.806 DEBUG:boto:path=/hbmuog-45/ 2023-10-17T18:31:00.806 DEBUG:boto:auth_path=/hbmuog-45/ 2023-10-17T18:31:00.807 DEBUG:boto:Method: PUT 2023-10-17T18:31:00.807 DEBUG:boto:Path: /hbmuog-45/ 2023-10-17T18:31:00.807 DEBUG:boto:Data: 2023-10-17T18:31:00.808 DEBUG:boto:Headers: {} 2023-10-17T18:31:00.808 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:31:00.809 DEBUG:boto:Port: 8001 2023-10-17T18:31:00.809 DEBUG:boto:Params: {} 2023-10-17T18:31:00.810 DEBUG:boto:Token: None 2023-10-17T18:31:00.810 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:31:00 GMT /hbmuog-45/ 2023-10-17T18:31:00.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tBQswlecRWbB1oVOgBy7EmSaZPk= 2023-10-17T18:31:00.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:31:00 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tBQswlecRWbB1oVOgBy7EmSaZPk=', 'Content-Length': '0'} 2023-10-17T18:31:00.839 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000880440517e26a31e-00652ed2e4-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:31:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:31:00.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:00.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:00.841 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:00.896 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:00.897 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:00.930 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:31:00.942 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:31:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567460.829217_508.1", 2023-10-17T18:31:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:31:00.829217Z" 2023-10-17T18:31:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:31:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:31:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:31:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:00.947 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:31:00.948 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:00.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:31:00.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:00.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:00.950 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:01.002 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:01.002 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:01.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:01.029+0000 7fc9e99b3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:01.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:01.030+0000 7fc9e99b3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:01.042 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:01.043 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:01.043 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:01.043 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:01.043 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:01.044 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:01.044 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:01.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.044 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:01.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:01.045 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:01.045 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:01.045 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:01.045 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:01.046 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:01.046 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:01.046 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:01.046 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:01.046 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:01.047 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:01.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:01.047 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:01.048 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:01.048 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:01.048 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:01.048 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:01.048 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:01.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:01.049 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:01.049 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:01.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:01.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:01.050 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:01.050 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:01.050 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:01.050 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:01.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:01.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:01.052 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:01.052 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:01.053 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:01.053 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:01.053 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:01.053 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:01.054 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:01.054 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:01.054 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:01.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:01.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:01.055 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:01.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.055 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:01.055 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:01.055 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:01.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:01.056 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:01Z" 2023-10-17T18:31:01.056 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:01.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:01Z" } 2023-10-17T18:31:01.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:01.058 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:01.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:01.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:06.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:06.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:06.061 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:06.121 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:06.122 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:06.154 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:06.153+0000 7f4e6f508780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:06.155 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:06.154+0000 7f4e6f508780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:06.168 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:06.169 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:06.169 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:06.169 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:06.169 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:06.170 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:06.170 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:06.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.171 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:06.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:06.171 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:06.171 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:06.171 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:06.172 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:06.172 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:06.172 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:06.172 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:06.173 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:06.173 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:06.173 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:06.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:06.174 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:06.174 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:06.174 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:06.174 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:06.175 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:06.175 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:06.175 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:06.175 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:06.176 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:06.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:06.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:06.177 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:06.178 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:06.178 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:06.178 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:06.178 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:06.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:06.179 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:06.179 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:06.179 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:06.180 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:06.180 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:06.180 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:06.180 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:06.180 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:06.181 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:06.181 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:06.181 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:06.181 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:06.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.182 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:06.182 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:06.182 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:06.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:06.182 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:06Z" 2023-10-17T18:31:06.183 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:06.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:06Z" } 2023-10-17T18:31:06.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:06.185 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:06.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:06.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:11.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:11.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:11.188 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:11.245 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:11.246 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:11.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:11.276+0000 7fe4d4109780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:11.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:11.277+0000 7fe4d4109780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:11.291 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:11.292 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:11.292 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:11.292 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:11.292 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:11.293 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:11.294 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:11.294 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:11.294 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:11.294 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:11.295 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:11.295 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:11.295 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:11.295 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:11.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:11.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:11.296 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:11.296 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:11.296 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:11.297 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:11.297 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:11.297 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:11.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:11.297 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:11.298 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:11.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:11.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:11.299 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:11.299 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:11.299 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:11.299 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:11.299 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:11.300 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:11.300 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:11.300 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:11.300 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:11.300 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:11.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:11.301 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:11.301 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:11.301 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:11.302 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:11.302 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:11.302 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:11.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:11.302 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:11.303 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:11.303 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:11.303 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:11.303 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:11.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.304 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:11.304 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:11.304 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:11.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:11.305 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:11Z" 2023-10-17T18:31:11.305 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:11.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:11Z" } 2023-10-17T18:31:11.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:11.307 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:11.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:11.307 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:16.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:16.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:16.309 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:16.369 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:16.369 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:16.400+0000 7f4f1c042780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:16.401+0000 7f4f1c042780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:16.415 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:16Z" 2023-10-17T18:31:16.430 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:16.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:16Z" } 2023-10-17T18:31:16.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:16.432 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:16.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:16.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:21.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:21.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:21.436 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:21.496 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:21.496 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:21.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:21.527+0000 7fc8af216780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:21.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:21.528+0000 7fc8af216780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:21.543 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:21.543 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:21.544 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:21.544 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:21.544 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:21.544 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:21.544 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:21.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.545 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:21.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:21.545 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:21.547 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:21.547 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:21.547 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:21.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:21.549 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:21.549 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:21.549 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:21.549 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:21.549 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:21.550 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:21.550 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:21.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:21.551 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:21.551 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:21.551 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:21.551 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:21.551 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:21.552 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:21.552 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:21.552 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:21.552 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:21.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:21.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:21.553 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:21.553 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:21.553 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:21.554 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:21.554 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:21.554 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:21.554 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:21.555 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:21.555 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:21.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:21.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:21.555 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:21.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.556 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:21.556 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:21.556 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:21.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:21.557 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:21Z" 2023-10-17T18:31:21.557 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:21.558 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:21Z" } 2023-10-17T18:31:21.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:21.559 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:21.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:21.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:26.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:26.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:26.562 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:26.621 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:26.621 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:26.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:26.653+0000 7f389efd3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:26.656 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:26.655+0000 7f389efd3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:26.669 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:26.670 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:26.670 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:26.670 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:26.671 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:26.671 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:26.671 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:26.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.671 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:26.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:26.672 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:26.673 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:26.673 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:26.673 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:26.673 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:26.673 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:26.674 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:26.674 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:26.674 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:26.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:26.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:26.675 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:26.675 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:26.675 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:26.676 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:26.676 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:26.676 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:26.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:26.678 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:26.679 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:26.680 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:26.681 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:26.681 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:26.681 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:26.681 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:26.681 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:26.682 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:26.682 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:26.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:26.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:26.682 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:26.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.683 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:26.683 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:26.683 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:26.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:26.684 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:26Z" 2023-10-17T18:31:26.684 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:26.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:26Z" } 2023-10-17T18:31:26.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:26.686 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:26.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:26.687 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:31.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:31.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:31.690 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:31.749 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:31.749 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:31.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:31.781+0000 7fca45de2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:31.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:31.782+0000 7fca45de2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:31.795 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:31.796 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:31.796 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:31.796 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:31.796 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:31.797 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:31.797 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:31.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.797 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:31.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:31.798 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:31.798 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:31.798 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:31.798 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:31.798 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:31.799 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:31.799 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:31.799 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:31.799 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:31.800 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:31.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:31.800 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:31.800 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:31.801 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:31.801 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:31.801 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:31.801 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:31.802 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:31.802 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:31.802 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:31.802 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:31.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:31.803 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:31.803 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:31.803 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:31.804 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:31.804 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:31.804 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:31.804 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:31.804 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:31.805 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:31.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:31.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:31.806 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:31.806 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:31.806 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:31.806 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:31.806 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:31.807 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:31.807 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:31.807 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:31.807 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:31Z" 2023-10-17T18:31:31.809 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:31.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:31Z" } 2023-10-17T18:31:31.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:31.812 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:31.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:31.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:36.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:36.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:36.815 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:36.874 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:36.908 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:36.907+0000 7f567ee1c780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:36.908 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:36.908+0000 7f567ee1c780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:36.922 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:36.922 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:36.922 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:36.923 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:36.923 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:36.923 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:36.923 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:36.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.924 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:36.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:36.924 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:36.924 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:36.925 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:36.925 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:36.925 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:36.925 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:36.925 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:36.926 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:36.926 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:36.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:36.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:36.927 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:36.927 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:36.927 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:36.927 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:36.928 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:36.928 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:36.928 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:36.928 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:36.928 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:36.929 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:36.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:36.929 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:36.930 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:36.930 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:36.930 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:36.930 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:36.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:36.932 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:36.932 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:36.932 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:36.933 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:36.933 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:36.933 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:36.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:36.936 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:36Z" 2023-10-17T18:31:36.936 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:36.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:36Z" } 2023-10-17T18:31:36.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:36.938 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:36.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:36.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:41.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:41.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:41.941 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:41.999 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:42.000 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:42.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:42.031+0000 7f75f10f7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:42.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:42.032+0000 7f75f10f7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:42.052 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:42.052 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:42.052 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:42.052 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:42.052 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:42.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:42.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:42.053 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:42.054 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:42.054 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:42.054 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:42.054 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:42.055 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:42.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:42.055 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:42.055 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:42.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:42.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:42.056 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:42.056 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:42.056 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:42.057 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:42.057 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:42.057 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:42.057 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:42.058 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:42.058 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:42.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:42.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:42.058 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:42.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.059 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:42.059 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:42.059 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:42.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:42.060 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:42Z" 2023-10-17T18:31:42.060 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:42.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:42Z" } 2023-10-17T18:31:42.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:42.062 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:42.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:42.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:47.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:47.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:47.065 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:47.127 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:47.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:47.158+0000 7f7a88a16780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:47.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:47.159+0000 7f7a88a16780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:47.182 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:47.182 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:47.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:47.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:47.183 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:47.183 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:47.183 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:47.184 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:47.184 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:47.184 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:47.184 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:47.184 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:47.185 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:47.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:47.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:47.186 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:47.186 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:47.186 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:47.186 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:47.187 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:47.187 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:47.187 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:47.187 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:47.188 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:47.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:47.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:47.188 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:47.190 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:47Z" 2023-10-17T18:31:47.190 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:47.191 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:47Z" } 2023-10-17T18:31:47.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:47.192 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:47.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:47.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:52.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:52.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:52.196 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:52.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:52.261 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:52.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:52.292+0000 7f4ac2a73780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:52.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:52.293+0000 7f4ac2a73780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:52.307 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:52Z" 2023-10-17T18:31:52.321 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:52.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:52Z" } 2023-10-17T18:31:52.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:52.323 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:52.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:52.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:31:57.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:57.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:31:57.326 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:31:57.385 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:31:57.386 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:31:57.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:57.417+0000 7f6061fda780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:57.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:31:57.418+0000 7f6061fda780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:31:57.432 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:31:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:31:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:31:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:31:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:31:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:31:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:31:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:31:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:31:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:31:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:31:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:31:57Z" 2023-10-17T18:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:31:57.446 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:31:57Z" } 2023-10-17T18:31:57.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:31:57.447 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:57.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:31:57.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:02.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:02.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:02.450 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:02.543+0000 7f9bf5412780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:02.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:02.544+0000 7f9bf5412780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:02.558 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:02.559 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:02.559 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:02.559 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:02.559 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:02.560 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:02.560 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:02.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.560 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:02.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:02.561 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:02.561 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:02.561 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:02.561 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:02.561 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:02.562 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:02.562 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:02.562 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:02.562 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:02.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:02.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:02.563 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:02.563 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:02.564 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:02.564 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:02.564 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:02.564 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:02.564 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:02.565 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:02.565 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:02.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:02.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:02.566 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:02.566 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:02.566 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:02.566 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:02.566 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:02.567 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:02.567 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:02.567 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:02.567 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:02Z" 2023-10-17T18:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:02.573 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:02Z" } 2023-10-17T18:32:02.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:02.574 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:02.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:02.575 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:07.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:07.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:07.577 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:07.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:07.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:07.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:07.668+0000 7f90a5fd3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:07.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:07.669+0000 7f90a5fd3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:07.683 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:07.683 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:07.683 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:07.684 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:07.684 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:07.684 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:07.684 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:07.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.685 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:07.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:07.685 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:07.685 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:07.697 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:07Z" 2023-10-17T18:32:07.697 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:07.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:07Z" } 2023-10-17T18:32:07.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:07.699 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:07.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:07.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:12.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:12.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:12.703 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:12.761 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:12.762 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:12.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:12.794+0000 7fc79821d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:12.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:12.795+0000 7fc79821d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:12.808 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:12.812 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:12.812 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:12.814 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:12.814 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:12.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:12.816 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:12.816 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:12.817 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:12.817 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:12.818 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:12.818 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:12.818 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:12.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:12.820 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:12.821 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:12.821 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:12.821 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:12.821 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:12.821 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:12.822 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:12.822 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:12.822 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:12.822 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:12.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:12.823 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:12.823 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:12.823 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:12Z" 2023-10-17T18:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:12.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:12Z" } 2023-10-17T18:32:12.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:12.829 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:12.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:12.830 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:17.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:17.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:17.833 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:17.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:17.924+0000 7f1e773f2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:17.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:17.925+0000 7f1e773f2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:17.939 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:17.950 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:17.950 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:17.950 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:17.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:17.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:17.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:17.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.951 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:17.951 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:17.952 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:17.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:17.952 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:17Z" 2023-10-17T18:32:17.952 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:17.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:17Z" } 2023-10-17T18:32:17.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:17.954 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:17.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:17.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:22.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:22.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:22.958 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:23.018 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:23.018 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:23.050 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:23.049+0000 7f1729737780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:23.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:23.050+0000 7f1729737780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:23.065 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:23.065 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:23.065 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:23.065 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:23.066 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:23.066 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:23.066 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:23.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.066 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:23.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:23.067 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:23.067 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:23.067 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:23.068 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:23.068 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:23.068 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:23.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:23.068 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:23.069 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:23.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:23.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:23.070 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:23.070 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:23.070 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:23.071 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:23.071 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:23.072 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:23.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:23.072 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:23.072 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:23.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:23.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:23.073 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:23.074 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:23.074 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:23.074 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:23.074 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:23.075 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:23.075 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:23.075 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:23.075 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:23.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:23.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:23.076 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:23.076 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:23.076 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:23.077 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:23.077 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:23.077 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:23.077 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:23.077 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:23.078 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:23.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:23.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:23.078 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:23.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.079 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:23.079 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:23.079 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:23.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:23.080 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:23Z" 2023-10-17T18:32:23.080 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:23.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:23Z" } 2023-10-17T18:32:23.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:23.083 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:23.083 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:23.083 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:28.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:28.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:28.085 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:28.145 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:28.145 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:28.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:28.177+0000 7f11bbb65780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:28.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:28.179+0000 7f11bbb65780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:28.193 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:28.193 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:28.193 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:28.194 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:28.194 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:28.194 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:28.194 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:28.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.195 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:28.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:28.195 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:28.195 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:28.196 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:28.196 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:28.196 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:28.196 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:28.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:28.197 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:28.197 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:28.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:28.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:28.198 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:28.198 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:28.198 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:28.198 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:28.198 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:28.199 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:28.199 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:28.199 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:28.199 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:28.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:28.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:28.200 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:28.200 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:28.200 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:28.201 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:28.201 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:28.201 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:28.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:28.202 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:28.202 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:28.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:28.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:28.203 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:28.203 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:28.205 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:28.205 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:28.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:28.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:28.205 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:28.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.206 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:28.206 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:28.206 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:28.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:28.207 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:28Z" 2023-10-17T18:32:28.207 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:28.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:28Z" } 2023-10-17T18:32:28.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:28.209 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:28.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:28.210 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:33.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:33.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:33.213 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:33.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:33.273 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:33.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:33.305+0000 7ff619c93780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:33.307 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:33.306+0000 7ff619c93780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:33.320 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:33.321 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:33.321 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:33.321 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:33.321 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:33.322 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:33.322 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:33.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.322 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:33.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:33.323 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:33.323 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:33.323 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:33.324 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:33.324 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:33.324 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:33.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:33.325 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:33.325 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:33.325 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:33.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:33.326 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:33.326 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:33.326 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:33.326 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:33.327 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:33.327 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:33.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:33.327 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:33.328 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:33.328 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:33.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:33.330 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:33.330 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:33.331 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:33.331 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:33.331 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:33.331 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:33.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:33.332 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:33.332 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:33.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:33.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:33.333 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:33.333 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:33.333 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:33.333 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:33.334 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:33.334 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:33.334 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:33.334 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.336 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:33.336 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:33.336 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:33.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:33.337 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:33Z" 2023-10-17T18:32:33.337 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:33.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:33Z" } 2023-10-17T18:32:33.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:33.339 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:33.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:33.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:38.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:38.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:38.342 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:38.401 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:38.401 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:38.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:38.433+0000 7f375a5f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:38.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:38.434+0000 7f375a5f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:38.448 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:38.448 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:38.449 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:38.449 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:38.449 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:38.449 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:38.450 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:38.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.451 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:38.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:38.451 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:38.451 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:38.452 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:38.452 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:38.452 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:38.452 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:38.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:38.453 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:38.453 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:38.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:38.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:38.454 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:38.454 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:38.454 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:38.455 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:38.455 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:38.455 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:38.455 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:38.456 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:38.456 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:38.456 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:38.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:38.457 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:38.457 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:38.457 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:38.457 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:38.458 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:38.458 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:38.458 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:38.458 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:38.459 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:38.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:38.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:38.459 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:38.460 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:38.460 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:38.460 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:38.460 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:38.461 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:38.461 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:38.461 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:38.461 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:38.461 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:38.462 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:38.462 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:38.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.462 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:38.463 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:38.463 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:38.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:38.463 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:38Z" 2023-10-17T18:32:38.463 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:38.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:38Z" } 2023-10-17T18:32:38.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:38.466 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:38.466 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:38.466 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:43.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:43.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:43.469 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:43.528 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:43.529 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:43.561 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:43.561+0000 7f9275067780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:43.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:43.562+0000 7f9275067780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:43.576 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:43.576 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:43.576 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:43.576 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:43.577 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:43.577 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:43.577 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:43.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.577 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:43.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:43.578 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:43.578 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:43.578 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:43.579 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:43.579 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:43.579 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:43.579 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:43.579 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:43Z" 2023-10-17T18:32:43.589 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:43.590 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:43Z" } 2023-10-17T18:32:43.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:43.591 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:43.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:43.592 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:48.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:48.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:48.594 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:48.653 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:48.654 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:48.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:48.686+0000 7f7132389780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:48.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:48.687+0000 7f7132389780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:48.701 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:48.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.703 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:48.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:48.703 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:48.703 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:48.703 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:48.704 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:48.704 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:48.704 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:48.704 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:48.704 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:48.705 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:48.705 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:48.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:48.705 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:48.706 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:48.706 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:48.706 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:48.706 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:48.707 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:48.707 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:48.707 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:48.707 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:48.707 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:48.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:48.708 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:48.708 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:48.708 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:48.709 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:48.709 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:48.709 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:48.709 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:48.709 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:48.710 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:48.710 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:48.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:48.710 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:48.711 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:48.711 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:48.711 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:48.711 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:48.712 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:48.712 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:48.712 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:48.712 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:48.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:48.713 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:48.713 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:48.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.713 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:48.713 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:48.714 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:48.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:48.714 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:48Z" 2023-10-17T18:32:48.714 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:48.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:48Z" } 2023-10-17T18:32:48.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:48.716 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:48.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:48.717 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:53.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:53.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:53.719 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:53.779 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:53.779 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:53.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:53.812+0000 7f70297c3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:53.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:53.813+0000 7f70297c3780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:53.827 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:53.828 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:53.828 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:53.828 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:53.829 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:53.829 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:53.829 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:53.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.829 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:53.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:53.830 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:53.830 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:53.830 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:53.830 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:53.831 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:53.832 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:53.833 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:53.833 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:53.833 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:53.833 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:53.833 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:53.835 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:53.835 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:53.835 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:53.835 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:53.836 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:53.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:53.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:53.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:53.838 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:53.838 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:53.838 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:53.838 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:53.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:53.839 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:53.839 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:53.839 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:53.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:53.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:53.840 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:53.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.840 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:53.840 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:53.840 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:53.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:53.841 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:53Z" 2023-10-17T18:32:53.841 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:53.842 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:53Z" } 2023-10-17T18:32:53.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:53.843 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:53.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:53.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:32:58.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:58.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:32:58.847 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:32:58.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:32:58.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:32:58.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:58.937+0000 7f4219f82780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:58.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:32:58.938+0000 7f4219f82780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:32:58.953 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:32:58.953 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:32:58.953 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:32:58.954 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:32:58.954 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:32:58.954 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:32:58.954 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:58.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:32:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:32:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:58.956 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567236.394899_344.1", 2023-10-17T18:32:58.956 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:58.956 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:32:58.956 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:58.956 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:27:16.394899Z", 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:32:58.957 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:58.958 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:58.958 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:32:58.958 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:58.958 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:58.958 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:58.959 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:32:58.959 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:58.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:58.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:58.960 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:58.961 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:58.961 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:58.961 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:58.961 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:58.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:58.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:32:58.962 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:32:58.962 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:32:58.962 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:32:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:32:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:32:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:32:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:32:58.963 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.964 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:32:58.965 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:32:58.965 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:32:58.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:32:58.965 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:32:58Z" 2023-10-17T18:32:58.965 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:32:58.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567236.394899_344.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:27:16.394899Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:32:58Z" } 2023-10-17T18:32:58.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567236.394899_344.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:27:16.394899Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:32:58.967 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:58.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567236.394899_344.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:32:58.968 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567236.394899_344.1 2023-10-17T18:33:03.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:03.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:03.972 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:04.030 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:04.031 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:04.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:04.112+0000 7f63070c1780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:04.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:04.113+0000 7f63070c1780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:04.127 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:04.127 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:04.127 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:04.128 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:04.128 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:04.128 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:33:04.128 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:04.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.129 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:04.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:04.129 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:04.129 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:04.130 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:04.130 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567457.312820_445.1", 2023-10-17T18:33:04.130 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:04.130 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:33:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:30:57.312820Z", 2023-10-17T18:33:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:04.132 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:04.132 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:04.132 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:33:04.134 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:04.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:04.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:04.136 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:04.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:04.136 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:04.136 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:04.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:04.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:04.137 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:04.137 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:04.137 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:04.138 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:04.138 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:04.138 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:04.138 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:04.139 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:04.139 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:04.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:04.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:04.139 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:04.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.140 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:04.140 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:33:04.140 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:33:04.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:04.141 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:04Z" 2023-10-17T18:33:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:04.142 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567457.312820_445.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:30:57.312820Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:33:04Z" } 2023-10-17T18:33:04.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567457.312820_445.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:30:57.312820Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:33:04.143 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:04.143 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567457.312820_445.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:04.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567457.312820_445.1 2023-10-17T18:33:09.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:09.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:09.146 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:09.201 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:09.202 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:09.231+0000 7ff64ca17780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:09.232+0000 7ff64ca17780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:09.245 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:09.245 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:09.246 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:09.246 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:09.246 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:09.246 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:33:09.247 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:09.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.247 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:09.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:09.247 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:09.248 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:09.248 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:09.248 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567457.473565_465.1", 2023-10-17T18:33:09.248 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:09.249 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:33:09.249 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:09.249 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:30:57.473565Z", 2023-10-17T18:33:09.249 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:09.250 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:09.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:09.250 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:09.251 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:09.251 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:09.251 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:09.251 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:09.251 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:09.252 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:09.252 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:33:09.252 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:09.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:09.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:09.253 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:09.253 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:09.253 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:09.254 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:09.254 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:09.254 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:09.254 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:09.255 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:09.255 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:09.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:09.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:09.256 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:09.256 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:09.256 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:09.256 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:09.256 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:09.257 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:09.257 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:09.257 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:09.257 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:09.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:09.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:09.258 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:09.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.258 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:09.259 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:33:09.259 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:33:09.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:09.259 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:09Z" 2023-10-17T18:33:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:09.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567457.473565_465.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:30:57.473565Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:33:09Z" } 2023-10-17T18:33:09.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567457.473565_465.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:30:57.473565Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:33:09.262 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:09.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567457.473565_465.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:09.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567457.473565_465.1 2023-10-17T18:33:14.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:14.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:14.266 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:14.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:14.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:14.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:14.354+0000 7f15d1bcb780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:14.356 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:14.355+0000 7f15d1bcb780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:14.369 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:14.370 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:14.370 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:14.370 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:14.370 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:14.371 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:33:14.371 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:14.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.372 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:14.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:14.372 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:14.372 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:14.373 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:14.373 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567457.473565_465.1", 2023-10-17T18:33:14.373 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:14.373 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:33:14.374 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:14.374 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:30:57.473565Z", 2023-10-17T18:33:14.374 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:14.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:14.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:14.375 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:14.375 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:14.375 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:14.376 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:14.376 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:14.376 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:14.376 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:14.376 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:33:14.377 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:14.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:14.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:14.378 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:14.378 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:14.378 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:14.378 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:14.378 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:14.379 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:14.379 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:14.379 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:14.379 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:14.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:14.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:14.380 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:14.380 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:14.381 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:14.381 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:14.381 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:14.381 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:14.381 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:14.382 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:14.382 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:14.382 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:14.382 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:14.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:14.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.383 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:14.383 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:33:14.383 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:33:14.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:14.384 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:14Z" 2023-10-17T18:33:14.384 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:14.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567457.473565_465.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:30:57.473565Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:33:14Z" } 2023-10-17T18:33:14.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567457.473565_465.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:30:57.473565Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:33:14.386 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:14.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567457.473565_465.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:14.387 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697567460.829217_508.1 target=1_1697567457.473565_465.1 2023-10-17T18:33:19.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.389 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:19.449 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:19.449 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:19.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:19.481+0000 7f67965a0780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:19.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:19.482+0000 7f67965a0780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:19.496 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:19.497 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:19.497 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:19.497 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:19.497 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:19.498 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:33:19.499 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:19.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.500 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:19.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:19.500 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:19.501 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:19.501 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:19.501 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567460.829217_508.1", 2023-10-17T18:33:19.502 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:19.502 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:33:19.502 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:19.502 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:31:00.829217Z", 2023-10-17T18:33:19.503 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:19.503 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:19.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:19.504 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:19.505 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:19.505 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:19.505 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:33:19.505 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:19.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:19.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:19.506 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:19.506 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:19.507 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:19.507 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:19.507 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:19.507 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:19.508 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:19.508 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:19.508 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:19.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:19.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:19.509 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:19.509 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:19.509 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:19.510 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:19.510 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:19.510 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:19.510 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:19.511 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:19.511 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:19.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:19.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:19.511 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:19.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.512 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:19.512 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:33:19.512 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:33:19.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:19.513 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:19Z" 2023-10-17T18:33:19.513 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:19.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567460.829217_508.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:31:00.829217Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:33:19Z" } 2023-10-17T18:33:19.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567460.829217_508.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:31:00.829217Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:33:19.515 DEBUG:rgw_multi.tests:log_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:19.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567460.829217_508.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:19.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:33:19.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:33:19.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:33:19.517 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:33:19.571 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:19.571 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:19.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:33:19.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:33:19.640 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:33:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:19.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.759 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:19.806 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:19.806 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:19.849 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:33:19.850 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000078", 2023-10-17T18:33:19.850 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.history", 2023-10-17T18:33:19.850 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:5", 2023-10-17T18:33:19.850 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:1", 2023-10-17T18:33:19.851 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:5", 2023-10-17T18:33:19.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000070", 2023-10-17T18:33:19.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000104", 2023-10-17T18:33:19.851 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13", 2023-10-17T18:33:19.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000026", 2023-10-17T18:33:19.852 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:3", 2023-10-17T18:33:19.852 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:10", 2023-10-17T18:33:19.852 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.8", 2023-10-17T18:33:19.852 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:7", 2023-10-17T18:33:19.852 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000028", 2023-10-17T18:33:19.853 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:10", 2023-10-17T18:33:19.853 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:2", 2023-10-17T18:33:19.853 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:1", 2023-10-17T18:33:19.853 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:6", 2023-10-17T18:33:19.853 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:1", 2023-10-17T18:33:19.854 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:0", 2023-10-17T18:33:19.854 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14", 2023-10-17T18:33:19.854 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:7", 2023-10-17T18:33:19.854 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000040", 2023-10-17T18:33:19.855 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:3", 2023-10-17T18:33:19.855 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:2", 2023-10-17T18:33:19.855 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.6", 2023-10-17T18:33:19.855 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:1", 2023-10-17T18:33:19.855 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000015", 2023-10-17T18:33:19.856 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000069", 2023-10-17T18:33:19.856 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000095", 2023-10-17T18:33:19.856 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000003", 2023-10-17T18:33:19.856 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000047", 2023-10-17T18:33:19.856 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000035", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000037", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:6", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:8", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000024", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000000", 2023-10-17T18:33:19.857 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000031", 2023-10-17T18:33:19.858 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:10", 2023-10-17T18:33:19.858 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:2", 2023-10-17T18:33:19.858 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:10", 2023-10-17T18:33:19.858 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.3", 2023-10-17T18:33:19.859 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:1", 2023-10-17T18:33:19.859 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:0", 2023-10-17T18:33:19.859 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:10", 2023-10-17T18:33:19.859 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:3", 2023-10-17T18:33:19.859 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:0", 2023-10-17T18:33:19.860 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:2", 2023-10-17T18:33:19.860 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:7", 2023-10-17T18:33:19.860 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:4", 2023-10-17T18:33:19.860 INFO:teuthology.orchestra.run.smithi049.stdout: "data_loggenerations_metadata", 2023-10-17T18:33:19.860 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:4", 2023-10-17T18:33:19.861 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:3", 2023-10-17T18:33:19.861 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000076", 2023-10-17T18:33:19.861 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000113", 2023-10-17T18:33:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000005", 2023-10-17T18:33:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:3", 2023-10-17T18:33:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000011", 2023-10-17T18:33:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000058", 2023-10-17T18:33:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000052", 2023-10-17T18:33:19.863 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:8", 2023-10-17T18:33:19.863 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:8", 2023-10-17T18:33:19.863 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:6", 2023-10-17T18:33:19.863 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:9", 2023-10-17T18:33:19.864 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:8", 2023-10-17T18:33:19.864 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:5", 2023-10-17T18:33:19.864 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:9", 2023-10-17T18:33:19.864 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:4", 2023-10-17T18:33:19.864 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:6", 2023-10-17T18:33:19.865 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000088", 2023-10-17T18:33:19.865 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000080", 2023-10-17T18:33:19.865 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000090", 2023-10-17T18:33:19.865 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000110", 2023-10-17T18:33:19.865 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15", 2023-10-17T18:33:19.866 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000096", 2023-10-17T18:33:19.866 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:3", 2023-10-17T18:33:19.866 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8", 2023-10-17T18:33:19.866 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000087", 2023-10-17T18:33:19.867 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7", 2023-10-17T18:33:19.867 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:6", 2023-10-17T18:33:19.867 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000008", 2023-10-17T18:33:19.867 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000006", 2023-10-17T18:33:19.867 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000029", 2023-10-17T18:33:19.868 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:5", 2023-10-17T18:33:19.868 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000089", 2023-10-17T18:33:19.868 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:0", 2023-10-17T18:33:19.868 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000022", 2023-10-17T18:33:19.868 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:4", 2023-10-17T18:33:19.869 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog.trim", 2023-10-17T18:33:19.869 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:5", 2023-10-17T18:33:19.869 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000016", 2023-10-17T18:33:19.869 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:7", 2023-10-17T18:33:19.870 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:10", 2023-10-17T18:33:19.870 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000041", 2023-10-17T18:33:19.870 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:5", 2023-10-17T18:33:19.870 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:8", 2023-10-17T18:33:19.870 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:2", 2023-10-17T18:33:19.871 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:5", 2023-10-17T18:33:19.871 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:3", 2023-10-17T18:33:19.871 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000018", 2023-10-17T18:33:19.871 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000092", 2023-10-17T18:33:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:0", 2023-10-17T18:33:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:9", 2023-10-17T18:33:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:4", 2023-10-17T18:33:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.4", 2023-10-17T18:33:19.872 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:2", 2023-10-17T18:33:19.873 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:1", 2023-10-17T18:33:19.873 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:2", 2023-10-17T18:33:19.873 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:4", 2023-10-17T18:33:19.873 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:9", 2023-10-17T18:33:19.874 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000014", 2023-10-17T18:33:19.874 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:4", 2023-10-17T18:33:19.874 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:6", 2023-10-17T18:33:19.874 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000112", 2023-10-17T18:33:19.874 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000007", 2023-10-17T18:33:19.875 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:9", 2023-10-17T18:33:19.875 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000021", 2023-10-17T18:33:19.875 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000064", 2023-10-17T18:33:19.875 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:9", 2023-10-17T18:33:19.875 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:5", 2023-10-17T18:33:19.876 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000071", 2023-10-17T18:33:19.876 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:8", 2023-10-17T18:33:19.876 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000074", 2023-10-17T18:33:19.876 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:5", 2023-10-17T18:33:19.877 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:2", 2023-10-17T18:33:19.877 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:8", 2023-10-17T18:33:19.877 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:1", 2023-10-17T18:33:19.877 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:10", 2023-10-17T18:33:19.877 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.1.retry", 2023-10-17T18:33:19.878 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000081", 2023-10-17T18:33:19.878 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000009", 2023-10-17T18:33:19.878 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000121", 2023-10-17T18:33:19.878 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.1", 2023-10-17T18:33:19.879 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:6", 2023-10-17T18:33:19.879 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:9", 2023-10-17T18:33:19.879 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000125", 2023-10-17T18:33:19.879 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000082", 2023-10-17T18:33:19.879 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:6", 2023-10-17T18:33:19.880 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000105", 2023-10-17T18:33:19.880 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000059", 2023-10-17T18:33:19.880 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000077", 2023-10-17T18:33:19.880 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:7", 2023-10-17T18:33:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000032", 2023-10-17T18:33:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:7", 2023-10-17T18:33:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:1", 2023-10-17T18:33:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:9", 2023-10-17T18:33:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.2", 2023-10-17T18:33:19.882 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:6", 2023-10-17T18:33:19.882 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.1", 2023-10-17T18:33:19.882 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000053", 2023-10-17T18:33:19.882 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000091", 2023-10-17T18:33:19.882 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18", 2023-10-17T18:33:19.883 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000065", 2023-10-17T18:33:19.883 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000083", 2023-10-17T18:33:19.883 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000010", 2023-10-17T18:33:19.883 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:4", 2023-10-17T18:33:19.883 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.7", 2023-10-17T18:33:19.884 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.3", 2023-10-17T18:33:19.884 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:10", 2023-10-17T18:33:19.884 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:5", 2023-10-17T18:33:19.884 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.1.lnkbaKNNopl.0", 2023-10-17T18:33:19.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:8", 2023-10-17T18:33:19.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:8", 2023-10-17T18:33:19.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:4", 2023-10-17T18:33:19.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:8", 2023-10-17T18:33:19.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:10", 2023-10-17T18:33:19.886 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000045", 2023-10-17T18:33:19.886 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000002", 2023-10-17T18:33:19.886 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:8", 2023-10-17T18:33:19.886 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:4", 2023-10-17T18:33:19.887 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000116", 2023-10-17T18:33:19.887 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000034", 2023-10-17T18:33:19.887 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000101", 2023-10-17T18:33:19.887 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:9", 2023-10-17T18:33:19.888 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:0", 2023-10-17T18:33:19.888 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:3", 2023-10-17T18:33:19.888 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000079", 2023-10-17T18:33:19.888 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:1", 2023-10-17T18:33:19.888 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000049", 2023-10-17T18:33:19.889 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000117", 2023-10-17T18:33:19.889 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000044", 2023-10-17T18:33:19.889 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:6", 2023-10-17T18:33:19.889 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000066", 2023-10-17T18:33:19.889 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:3", 2023-10-17T18:33:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:7", 2023-10-17T18:33:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31", 2023-10-17T18:33:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:1", 2023-10-17T18:33:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000068", 2023-10-17T18:33:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:5", 2023-10-17T18:33:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000085", 2023-10-17T18:33:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:1", 2023-10-17T18:33:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17", 2023-10-17T18:33:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:0", 2023-10-17T18:33:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000073", 2023-10-17T18:33:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:10", 2023-10-17T18:33:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000038", 2023-10-17T18:33:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000118", 2023-10-17T18:33:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:2", 2023-10-17T18:33:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.2.ZrzZlVxEG+D.0", 2023-10-17T18:33:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:5", 2023-10-17T18:33:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:3", 2023-10-17T18:33:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000036", 2023-10-17T18:33:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16", 2023-10-17T18:33:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000103", 2023-10-17T18:33:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:1", 2023-10-17T18:33:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.0", 2023-10-17T18:33:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000119", 2023-10-17T18:33:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:9", 2023-10-17T18:33:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:1", 2023-10-17T18:33:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000098", 2023-10-17T18:33:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:6", 2023-10-17T18:33:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:8", 2023-10-17T18:33:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:0", 2023-10-17T18:33:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:0", 2023-10-17T18:33:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:0", 2023-10-17T18:33:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:7", 2023-10-17T18:33:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000027", 2023-10-17T18:33:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:7", 2023-10-17T18:33:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12", 2023-10-17T18:33:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:1", 2023-10-17T18:33:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000019", 2023-10-17T18:33:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:2", 2023-10-17T18:33:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000039", 2023-10-17T18:33:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:2", 2023-10-17T18:33:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000100", 2023-10-17T18:33:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:10", 2023-10-17T18:33:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.3.retry", 2023-10-17T18:33:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:10", 2023-10-17T18:33:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:5", 2023-10-17T18:33:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:0", 2023-10-17T18:33:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000093", 2023-10-17T18:33:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:6", 2023-10-17T18:33:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:3", 2023-10-17T18:33:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000004", 2023-10-17T18:33:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000063", 2023-10-17T18:33:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000122", 2023-10-17T18:33:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:8", 2023-10-17T18:33:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000057", 2023-10-17T18:33:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000054", 2023-10-17T18:33:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000114", 2023-10-17T18:33:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:5", 2023-10-17T18:33:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.2", 2023-10-17T18:33:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:10", 2023-10-17T18:33:19.903 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.0", 2023-10-17T18:33:19.903 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:6", 2023-10-17T18:33:19.903 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:7", 2023-10-17T18:33:19.903 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000012", 2023-10-17T18:33:19.903 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:9", 2023-10-17T18:33:19.904 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:8", 2023-10-17T18:33:19.904 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:4", 2023-10-17T18:33:19.904 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000084", 2023-10-17T18:33:19.904 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000043", 2023-10-17T18:33:19.904 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:2", 2023-10-17T18:33:19.905 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000111", 2023-10-17T18:33:19.905 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:6", 2023-10-17T18:33:19.905 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000048", 2023-10-17T18:33:19.905 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:0", 2023-10-17T18:33:19.905 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000020", 2023-10-17T18:33:19.906 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000099", 2023-10-17T18:33:19.906 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000056", 2023-10-17T18:33:19.906 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:9", 2023-10-17T18:33:19.906 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:0", 2023-10-17T18:33:19.906 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:7", 2023-10-17T18:33:19.907 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11", 2023-10-17T18:33:19.907 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000072", 2023-10-17T18:33:19.907 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:7", 2023-10-17T18:33:19.907 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1", 2023-10-17T18:33:19.907 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000062", 2023-10-17T18:33:19.908 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:0", 2023-10-17T18:33:19.908 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000115", 2023-10-17T18:33:19.908 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.1", 2023-10-17T18:33:19.908 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000033", 2023-10-17T18:33:19.908 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:9", 2023-10-17T18:33:19.909 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:7", 2023-10-17T18:33:19.909 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:7", 2023-10-17T18:33:19.909 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000025", 2023-10-17T18:33:19.909 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000123", 2023-10-17T18:33:19.910 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.3.gBydDewx7Q+.0", 2023-10-17T18:33:19.910 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000108", 2023-10-17T18:33:19.910 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000094", 2023-10-17T18:33:19.910 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:6", 2023-10-17T18:33:19.910 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000060", 2023-10-17T18:33:19.911 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10", 2023-10-17T18:33:19.911 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000109", 2023-10-17T18:33:19.911 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:4", 2023-10-17T18:33:19.911 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000013", 2023-10-17T18:33:19.911 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:4", 2023-10-17T18:33:19.912 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:9", 2023-10-17T18:33:19.913 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:8", 2023-10-17T18:33:19.913 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:2", 2023-10-17T18:33:19.913 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:2", 2023-10-17T18:33:19.914 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.0.x5KzNaXvZk+.0", 2023-10-17T18:33:19.914 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:4", 2023-10-17T18:33:19.914 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:1", 2023-10-17T18:33:19.914 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:8", 2023-10-17T18:33:19.915 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000051", 2023-10-17T18:33:19.915 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:0", 2023-10-17T18:33:19.915 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000106", 2023-10-17T18:33:19.915 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:6", 2023-10-17T18:33:19.915 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:10", 2023-10-17T18:33:19.916 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000017", 2023-10-17T18:33:19.916 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:3", 2023-10-17T18:33:19.916 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000023", 2023-10-17T18:33:19.916 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000107", 2023-10-17T18:33:19.916 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000046", 2023-10-17T18:33:19.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34", 2023-10-17T18:33:19.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:9", 2023-10-17T18:33:19.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:6", 2023-10-17T18:33:19.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:4", 2023-10-17T18:33:19.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:1", 2023-10-17T18:33:19.918 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.2", 2023-10-17T18:33:19.918 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:2", 2023-10-17T18:33:19.918 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000067", 2023-10-17T18:33:19.918 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:10", 2023-10-17T18:33:19.919 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.2.retry", 2023-10-17T18:33:19.919 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T18:33:19.919 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:0", 2023-10-17T18:33:19.919 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.8f644c5a-db9c-4565-b32b-e3a6790e3bcc.3", 2023-10-17T18:33:19.919 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:8", 2023-10-17T18:33:19.920 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.9", 2023-10-17T18:33:19.920 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000086", 2023-10-17T18:33:19.920 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:2", 2023-10-17T18:33:19.920 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.5", 2023-10-17T18:33:19.920 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:5", 2023-10-17T18:33:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:7", 2023-10-17T18:33:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.log.cfc9d53d-2765-4da9-b7a8-37407f517c55.0", 2023-10-17T18:33:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.log.cfc9d53d-2765-4da9-b7a8-37407f517c55.1", 2023-10-17T18:33:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:2", 2023-10-17T18:33:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:5", 2023-10-17T18:33:19.922 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:3", 2023-10-17T18:33:19.922 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:3", 2023-10-17T18:33:19.922 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:9", 2023-10-17T18:33:19.922 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33", 2023-10-17T18:33:19.922 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:3", 2023-10-17T18:33:19.923 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:3", 2023-10-17T18:33:19.923 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000050", 2023-10-17T18:33:19.923 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:10", 2023-10-17T18:33:19.923 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000102", 2023-10-17T18:33:19.923 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:0", 2023-10-17T18:33:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:3", 2023-10-17T18:33:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:7", 2023-10-17T18:33:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:9", 2023-10-17T18:33:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000124", 2023-10-17T18:33:19.925 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9", 2023-10-17T18:33:19.925 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:5", 2023-10-17T18:33:19.925 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32", 2023-10-17T18:33:19.925 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000001", 2023-10-17T18:33:19.925 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000126", 2023-10-17T18:33:19.926 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000055", 2023-10-17T18:33:19.926 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000061", 2023-10-17T18:33:19.926 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:1", 2023-10-17T18:33:19.926 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000097", 2023-10-17T18:33:19.926 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000042", 2023-10-17T18:33:19.927 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:10", 2023-10-17T18:33:19.927 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:4", 2023-10-17T18:33:19.927 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000075", 2023-10-17T18:33:19.927 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:7", 2023-10-17T18:33:19.927 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000120", 2023-10-17T18:33:19.928 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000030", 2023-10-17T18:33:19.928 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:4" 2023-10-17T18:33:19.928 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:33:19.928 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:33:19.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:19.931 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:19.978 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:19.979 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:20.020 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:33:20.020 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000078", 2023-10-17T18:33:20.021 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.history", 2023-10-17T18:33:20.021 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.0", 2023-10-17T18:33:20.021 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:1", 2023-10-17T18:33:20.021 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000070", 2023-10-17T18:33:20.021 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000104", 2023-10-17T18:33:20.022 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11", 2023-10-17T18:33:20.022 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000026", 2023-10-17T18:33:20.022 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:6", 2023-10-17T18:33:20.022 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:1", 2023-10-17T18:33:20.023 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:8", 2023-10-17T18:33:20.023 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.8", 2023-10-17T18:33:20.023 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000028", 2023-10-17T18:33:20.023 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:9", 2023-10-17T18:33:20.023 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:1", 2023-10-17T18:33:20.024 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:9", 2023-10-17T18:33:20.024 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000040", 2023-10-17T18:33:20.024 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.6", 2023-10-17T18:33:20.024 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16", 2023-10-17T18:33:20.024 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:8", 2023-10-17T18:33:20.025 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1", 2023-10-17T18:33:20.025 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000015", 2023-10-17T18:33:20.025 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000069", 2023-10-17T18:33:20.025 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000095", 2023-10-17T18:33:20.025 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:7", 2023-10-17T18:33:20.026 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:2", 2023-10-17T18:33:20.026 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000003", 2023-10-17T18:33:20.026 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000047", 2023-10-17T18:33:20.026 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:6", 2023-10-17T18:33:20.027 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:6", 2023-10-17T18:33:20.027 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:6", 2023-10-17T18:33:20.027 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:10", 2023-10-17T18:33:20.027 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000035", 2023-10-17T18:33:20.027 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000037", 2023-10-17T18:33:20.028 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:3", 2023-10-17T18:33:20.028 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:7", 2023-10-17T18:33:20.028 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:9", 2023-10-17T18:33:20.028 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:9", 2023-10-17T18:33:20.028 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:2", 2023-10-17T18:33:20.029 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:9", 2023-10-17T18:33:20.029 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:0", 2023-10-17T18:33:20.029 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000024", 2023-10-17T18:33:20.029 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:3", 2023-10-17T18:33:20.030 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:8", 2023-10-17T18:33:20.030 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:1", 2023-10-17T18:33:20.030 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000000", 2023-10-17T18:33:20.030 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000031", 2023-10-17T18:33:20.030 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:2", 2023-10-17T18:33:20.031 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:7", 2023-10-17T18:33:20.031 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.3", 2023-10-17T18:33:20.031 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:10", 2023-10-17T18:33:20.031 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:3", 2023-10-17T18:33:20.031 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:2", 2023-10-17T18:33:20.032 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:8", 2023-10-17T18:33:20.032 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:7", 2023-10-17T18:33:20.032 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.2.retry", 2023-10-17T18:33:20.032 INFO:teuthology.orchestra.run.smithi049.stdout: "data_loggenerations_metadata", 2023-10-17T18:33:20.032 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:5", 2023-10-17T18:33:20.033 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:4", 2023-10-17T18:33:20.033 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000076", 2023-10-17T18:33:20.033 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:1", 2023-10-17T18:33:20.033 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000113", 2023-10-17T18:33:20.033 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000005", 2023-10-17T18:33:20.034 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000011", 2023-10-17T18:33:20.034 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000058", 2023-10-17T18:33:20.034 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:8", 2023-10-17T18:33:20.034 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000052", 2023-10-17T18:33:20.034 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:0", 2023-10-17T18:33:20.035 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:7", 2023-10-17T18:33:20.035 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:2", 2023-10-17T18:33:20.035 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:5", 2023-10-17T18:33:20.035 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:1", 2023-10-17T18:33:20.036 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:4", 2023-10-17T18:33:20.036 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000088", 2023-10-17T18:33:20.036 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000080", 2023-10-17T18:33:20.036 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000090", 2023-10-17T18:33:20.036 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000110", 2023-10-17T18:33:20.037 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000096", 2023-10-17T18:33:20.037 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:6", 2023-10-17T18:33:20.037 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000087", 2023-10-17T18:33:20.037 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15", 2023-10-17T18:33:20.037 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:7", 2023-10-17T18:33:20.038 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000008", 2023-10-17T18:33:20.038 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.0.retry", 2023-10-17T18:33:20.038 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:4", 2023-10-17T18:33:20.039 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:4", 2023-10-17T18:33:20.039 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:3", 2023-10-17T18:33:20.039 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:1", 2023-10-17T18:33:20.039 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000006", 2023-10-17T18:33:20.040 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000029", 2023-10-17T18:33:20.040 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:5", 2023-10-17T18:33:20.040 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000089", 2023-10-17T18:33:20.040 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000022", 2023-10-17T18:33:20.040 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:1", 2023-10-17T18:33:20.041 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:2", 2023-10-17T18:33:20.041 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog.trim", 2023-10-17T18:33:20.041 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000016", 2023-10-17T18:33:20.041 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:7", 2023-10-17T18:33:20.041 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:0", 2023-10-17T18:33:20.042 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000041", 2023-10-17T18:33:20.042 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:0", 2023-10-17T18:33:20.042 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000018", 2023-10-17T18:33:20.042 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:4", 2023-10-17T18:33:20.043 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000092", 2023-10-17T18:33:20.043 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:3", 2023-10-17T18:33:20.043 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.4", 2023-10-17T18:33:20.043 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:3", 2023-10-17T18:33:20.043 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.3", 2023-10-17T18:33:20.044 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:5", 2023-10-17T18:33:20.044 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000014", 2023-10-17T18:33:20.044 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:7", 2023-10-17T18:33:20.044 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000112", 2023-10-17T18:33:20.044 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000007", 2023-10-17T18:33:20.045 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000021", 2023-10-17T18:33:20.045 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:4", 2023-10-17T18:33:20.045 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:0", 2023-10-17T18:33:20.045 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000064", 2023-10-17T18:33:20.045 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:6", 2023-10-17T18:33:20.046 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000071", 2023-10-17T18:33:20.046 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000074", 2023-10-17T18:33:20.046 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:2", 2023-10-17T18:33:20.046 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:5", 2023-10-17T18:33:20.047 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:9", 2023-10-17T18:33:20.047 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:6", 2023-10-17T18:33:20.047 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31", 2023-10-17T18:33:20.047 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.2", 2023-10-17T18:33:20.047 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:2", 2023-10-17T18:33:20.048 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34", 2023-10-17T18:33:20.048 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000081", 2023-10-17T18:33:20.048 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000009", 2023-10-17T18:33:20.048 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:9", 2023-10-17T18:33:20.048 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000121", 2023-10-17T18:33:20.049 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.1", 2023-10-17T18:33:20.049 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:7", 2023-10-17T18:33:20.049 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:6", 2023-10-17T18:33:20.049 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:2", 2023-10-17T18:33:20.049 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000125", 2023-10-17T18:33:20.050 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:10", 2023-10-17T18:33:20.050 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000082", 2023-10-17T18:33:20.050 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:2", 2023-10-17T18:33:20.050 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:0", 2023-10-17T18:33:20.051 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:0", 2023-10-17T18:33:20.051 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:5", 2023-10-17T18:33:20.051 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:4", 2023-10-17T18:33:20.051 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:0", 2023-10-17T18:33:20.052 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000105", 2023-10-17T18:33:20.052 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000059", 2023-10-17T18:33:20.053 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:4", 2023-10-17T18:33:20.053 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:10", 2023-10-17T18:33:20.053 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000077", 2023-10-17T18:33:20.053 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000032", 2023-10-17T18:33:20.053 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:10", 2023-10-17T18:33:20.054 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12", 2023-10-17T18:33:20.054 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:7", 2023-10-17T18:33:20.054 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:7", 2023-10-17T18:33:20.054 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:8", 2023-10-17T18:33:20.055 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.1.retry", 2023-10-17T18:33:20.055 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000053", 2023-10-17T18:33:20.055 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:5", 2023-10-17T18:33:20.055 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000091", 2023-10-17T18:33:20.055 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000065", 2023-10-17T18:33:20.056 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.3.retry", 2023-10-17T18:33:20.056 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:0", 2023-10-17T18:33:20.056 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000083", 2023-10-17T18:33:20.056 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000010", 2023-10-17T18:33:20.056 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.7", 2023-10-17T18:33:20.057 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:1", 2023-10-17T18:33:20.057 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.3", 2023-10-17T18:33:20.057 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:4", 2023-10-17T18:33:20.057 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:3", 2023-10-17T18:33:20.058 INFO:teuthology.orchestra.run.smithi049.stdout: "mdlog.sync-status.shard.2", 2023-10-17T18:33:20.058 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:9", 2023-10-17T18:33:20.058 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:4", 2023-10-17T18:33:20.058 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:9", 2023-10-17T18:33:20.058 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000045", 2023-10-17T18:33:20.059 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:3", 2023-10-17T18:33:20.059 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:7", 2023-10-17T18:33:20.059 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000002", 2023-10-17T18:33:20.059 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.12", 2023-10-17T18:33:20.059 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:4", 2023-10-17T18:33:20.060 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:2", 2023-10-17T18:33:20.060 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:4", 2023-10-17T18:33:20.060 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000116", 2023-10-17T18:33:20.060 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:7", 2023-10-17T18:33:20.061 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:10", 2023-10-17T18:33:20.061 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000034", 2023-10-17T18:33:20.061 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:5", 2023-10-17T18:33:20.061 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000101", 2023-10-17T18:33:20.061 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.13", 2023-10-17T18:33:20.062 INFO:teuthology.orchestra.run.smithi049.stdout: "mdlog.sync-status.shard.1", 2023-10-17T18:33:20.062 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:8", 2023-10-17T18:33:20.062 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:0", 2023-10-17T18:33:20.062 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000079", 2023-10-17T18:33:20.062 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000049", 2023-10-17T18:33:20.063 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:7", 2023-10-17T18:33:20.063 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000117", 2023-10-17T18:33:20.063 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000044", 2023-10-17T18:33:20.063 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:9", 2023-10-17T18:33:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:2", 2023-10-17T18:33:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000066", 2023-10-17T18:33:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:5", 2023-10-17T18:33:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:8", 2023-10-17T18:33:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:6", 2023-10-17T18:33:20.065 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000068", 2023-10-17T18:33:20.065 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:0", 2023-10-17T18:33:20.065 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000085", 2023-10-17T18:33:20.065 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33", 2023-10-17T18:33:20.065 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:4", 2023-10-17T18:33:20.066 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000073", 2023-10-17T18:33:20.066 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000038", 2023-10-17T18:33:20.066 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.10", 2023-10-17T18:33:20.066 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000118", 2023-10-17T18:33:20.067 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000036", 2023-10-17T18:33:20.067 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:6", 2023-10-17T18:33:20.067 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:6", 2023-10-17T18:33:20.067 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000103", 2023-10-17T18:33:20.067 INFO:teuthology.orchestra.run.smithi049.stdout: "mdlog.sync-status.shard.3", 2023-10-17T18:33:20.068 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:6", 2023-10-17T18:33:20.068 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.3.DPVkNwyWJgM.0", 2023-10-17T18:33:20.068 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000119", 2023-10-17T18:33:20.068 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000098", 2023-10-17T18:33:20.068 INFO:teuthology.orchestra.run.smithi049.stdout: "mdlog.sync-status.shard.0", 2023-10-17T18:33:20.069 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:0", 2023-10-17T18:33:20.069 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000027", 2023-10-17T18:33:20.069 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:3", 2023-10-17T18:33:20.069 INFO:teuthology.orchestra.run.smithi049.stdout: "mdlog.sync-status", 2023-10-17T18:33:20.069 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000019", 2023-10-17T18:33:20.070 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000039", 2023-10-17T18:33:20.070 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:7", 2023-10-17T18:33:20.070 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:2", 2023-10-17T18:33:20.070 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:6", 2023-10-17T18:33:20.071 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14", 2023-10-17T18:33:20.071 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-15:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.15:5", 2023-10-17T18:33:20.071 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000100", 2023-10-17T18:33:20.071 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.0.qjn126Kabgm.0", 2023-10-17T18:33:20.071 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:3", 2023-10-17T18:33:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000093", 2023-10-17T18:33:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:5", 2023-10-17T18:33:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:10", 2023-10-17T18:33:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17", 2023-10-17T18:33:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000004", 2023-10-17T18:33:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000063", 2023-10-17T18:33:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000122", 2023-10-17T18:33:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:8", 2023-10-17T18:33:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000057", 2023-10-17T18:33:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:2", 2023-10-17T18:33:20.074 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:5", 2023-10-17T18:33:20.074 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:10", 2023-10-17T18:33:20.074 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000054", 2023-10-17T18:33:20.074 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000114", 2023-10-17T18:33:20.075 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.2", 2023-10-17T18:33:20.075 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.0", 2023-10-17T18:33:20.075 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000012", 2023-10-17T18:33:20.075 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:6", 2023-10-17T18:33:20.075 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:9", 2023-10-17T18:33:20.076 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:10", 2023-10-17T18:33:20.076 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:9", 2023-10-17T18:33:20.076 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000084", 2023-10-17T18:33:20.076 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000043", 2023-10-17T18:33:20.076 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:5", 2023-10-17T18:33:20.077 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:3", 2023-10-17T18:33:20.077 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:4", 2023-10-17T18:33:20.077 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:0", 2023-10-17T18:33:20.077 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.11", 2023-10-17T18:33:20.077 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:9", 2023-10-17T18:33:20.078 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:10", 2023-10-17T18:33:20.078 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000111", 2023-10-17T18:33:20.078 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000048", 2023-10-17T18:33:20.078 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:4", 2023-10-17T18:33:20.079 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000020", 2023-10-17T18:33:20.079 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:5", 2023-10-17T18:33:20.079 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000099", 2023-10-17T18:33:20.079 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000056", 2023-10-17T18:33:20.079 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32", 2023-10-17T18:33:20.080 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:10", 2023-10-17T18:33:20.080 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:6", 2023-10-17T18:33:20.080 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10:2", 2023-10-17T18:33:20.080 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000072", 2023-10-17T18:33:20.080 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:4", 2023-10-17T18:33:20.081 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:10", 2023-10-17T18:33:20.081 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000062", 2023-10-17T18:33:20.081 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000115", 2023-10-17T18:33:20.081 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.1", 2023-10-17T18:33:20.082 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000033", 2023-10-17T18:33:20.082 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:8", 2023-10-17T18:33:20.082 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000025", 2023-10-17T18:33:20.082 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-10:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.10", 2023-10-17T18:33:20.082 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:10", 2023-10-17T18:33:20.083 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000123", 2023-10-17T18:33:20.083 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:9", 2023-10-17T18:33:20.083 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:3", 2023-10-17T18:33:20.083 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:9", 2023-10-17T18:33:20.083 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000108", 2023-10-17T18:33:20.084 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:1", 2023-10-17T18:33:20.084 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:0", 2023-10-17T18:33:20.084 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000094", 2023-10-17T18:33:20.084 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:8", 2023-10-17T18:33:20.085 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000060", 2023-10-17T18:33:20.085 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:6", 2023-10-17T18:33:20.085 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:6", 2023-10-17T18:33:20.085 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000109", 2023-10-17T18:33:20.085 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:8", 2023-10-17T18:33:20.086 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7:4", 2023-10-17T18:33:20.086 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-7:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.7", 2023-10-17T18:33:20.086 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:0", 2023-10-17T18:33:20.086 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000013", 2023-10-17T18:33:20.086 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13", 2023-10-17T18:33:20.087 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:1", 2023-10-17T18:33:20.087 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:3", 2023-10-17T18:33:20.087 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:1", 2023-10-17T18:33:20.087 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:9", 2023-10-17T18:33:20.087 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000051", 2023-10-17T18:33:20.088 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:3", 2023-10-17T18:33:20.088 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.1.Leu1KOHRVCW.0", 2023-10-17T18:33:20.088 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000106", 2023-10-17T18:33:20.088 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:10", 2023-10-17T18:33:20.089 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:1", 2023-10-17T18:33:20.089 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000017", 2023-10-17T18:33:20.089 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:1", 2023-10-17T18:33:20.089 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18", 2023-10-17T18:33:20.089 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.2.Mht+KBJu9CP.0", 2023-10-17T18:33:20.090 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:5", 2023-10-17T18:33:20.090 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000023", 2023-10-17T18:33:20.090 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T18:33:20.090 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:0", 2023-10-17T18:33:20.090 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000107", 2023-10-17T18:33:20.091 INFO:teuthology.orchestra.run.smithi049.stdout: "datalog.sync-status.shard.4c84373e-d451-49bb-a8c9-c37d5e5e2160.1", 2023-10-17T18:33:20.091 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:1", 2023-10-17T18:33:20.091 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000046", 2023-10-17T18:33:20.091 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:10", 2023-10-17T18:33:20.092 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-35:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5397.1:8", 2023-10-17T18:33:20.092 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-34:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.34:0", 2023-10-17T18:33:20.092 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-16:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.16:7", 2023-10-17T18:33:20.092 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:3", 2023-10-17T18:33:20.092 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:8", 2023-10-17T18:33:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: "data_log.2", 2023-10-17T18:33:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000067", 2023-10-17T18:33:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:8", 2023-10-17T18:33:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-14:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.14:5", 2023-10-17T18:33:20.093 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.9", 2023-10-17T18:33:20.094 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000086", 2023-10-17T18:33:20.094 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-17:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.17:2", 2023-10-17T18:33:20.094 INFO:teuthology.orchestra.run.smithi049.stdout: "sync.error-log.5", 2023-10-17T18:33:20.094 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:1", 2023-10-17T18:33:20.095 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.log.cfc9d53d-2765-4da9-b7a8-37407f517c55.0", 2023-10-17T18:33:20.095 INFO:teuthology.orchestra.run.smithi049.stdout: "meta.log.cfc9d53d-2765-4da9-b7a8-37407f517c55.1", 2023-10-17T18:33:20.095 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-12:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.12:7", 2023-10-17T18:33:20.095 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:8", 2023-10-17T18:33:20.095 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-31:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.31:3", 2023-10-17T18:33:20.096 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8:10", 2023-10-17T18:33:20.096 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000050", 2023-10-17T18:33:20.096 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000102", 2023-10-17T18:33:20.096 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:1", 2023-10-17T18:33:20.096 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-8:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.8", 2023-10-17T18:33:20.097 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-11:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.11:5", 2023-10-17T18:33:20.097 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000124", 2023-10-17T18:33:20.097 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-33:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.33:9", 2023-10-17T18:33:20.097 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000001", 2023-10-17T18:33:20.097 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.full-sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9", 2023-10-17T18:33:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000126", 2023-10-17T18:33:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000055", 2023-10-17T18:33:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000061", 2023-10-17T18:33:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000097", 2023-10-17T18:33:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000042", 2023-10-17T18:33:20.099 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-13:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.13:2", 2023-10-17T18:33:20.099 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-9:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.9:3", 2023-10-17T18:33:20.099 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000075", 2023-10-17T18:33:20.099 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000120", 2023-10-17T18:33:20.100 INFO:teuthology.orchestra.run.smithi049.stdout: "obj_delete_at_hint.0000000030", 2023-10-17T18:33:20.100 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-18:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.18:8", 2023-10-17T18:33:20.100 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-32:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.32:10" 2023-10-17T18:33:20.100 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:33:20.100 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-17T18:33:20.102 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T18:33:20.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:20.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:20.103 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:20.201 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:33:20.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.201 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-10-17T18:33:20.201 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:19.740062Z" 2023-10-17T18:33:20.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.202 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-17T18:33:20.202 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:19.740934Z" 2023-10-17T18:33:20.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.203 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-10-17T18:33:20.203 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:19.738037Z" 2023-10-17T18:33:20.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.204 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-17T18:33:20.204 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:19.739063Z" 2023-10-17T18:33:20.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.205 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:33:20.206 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:20.206 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:33:20.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:20.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:20.208 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:20.264 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:20.265 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:20.308 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:33:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-17T18:33:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.134621Z" 2023-10-17T18:33:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:20.314 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:20.314 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:20.314 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:20.314 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:33:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:33:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:33:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:33:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:20Z" 2023-10-17T18:33:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:20.325 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.134621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.459915Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.592314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:20Z" } 2023-10-17T18:33:20.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.134621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.459915Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.592314Z'}}] 2023-10-17T18:33:20.326 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:20.327 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002625', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-17T18:33:20.327 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324 2023-10-17T18:33:25.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:25.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:25.330 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:25.390 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:25.390 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:25.438 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:25.439 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:25.439 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:25.440 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:25.440 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:25.441 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:33:25.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.441 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:25.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:25.442 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:25.443 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:25.443 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:25.443 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-17T18:33:25.444 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:25.444 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:25.445 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:25.445 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.134621Z" 2023-10-17T18:33:25.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:25.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:25.446 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:25.447 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:25.447 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:25.448 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:25.448 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:25.448 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:25.449 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:25.449 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:33:25.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:25.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:25.451 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:25.451 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:25.451 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:25.452 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:25.452 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:25.453 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:25.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:25.453 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:33:25.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:25.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:25.458 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:25.458 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:25.458 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:25.459 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:33:25.459 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:25.459 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:25.460 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:25.460 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:33:25.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:25.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:25.461 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:25.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.461 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:25.461 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:25.462 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:25.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:25.463 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:25Z" 2023-10-17T18:33:25.463 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:25.465 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.134621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.459915Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.592314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:25Z" } 2023-10-17T18:33:25.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.134621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.459915Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.592314Z'}}] 2023-10-17T18:33:25.466 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:25.467 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002625', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-17T18:33:25.467 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324 2023-10-17T18:33:30.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:30.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:30.469 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:30.533 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:30.533 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:30.582 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:30.582 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:30.582 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:30.583 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:30.583 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:30.583 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:33:30.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.585 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:30.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:30.586 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:30.586 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:30.586 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:30.586 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-17T18:33:30.587 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:30.588 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:30.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:30.589 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.134621Z" 2023-10-17T18:33:30.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:30.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:30.591 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:30.592 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:30.592 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:30.592 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:30.592 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:30.593 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:30.593 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:30.593 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:33:30.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:30.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:30.594 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:30.594 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:30.595 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:30.595 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:30.595 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:30.595 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:30.595 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:30.596 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:33:30.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:30.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:30.596 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:30.598 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:30.598 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:33:30.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:30.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:30.599 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:30.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.599 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:30.599 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:30.599 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:30.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:30.600 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:30Z" 2023-10-17T18:33:30.600 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:30.602 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.134621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.459915Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.592314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:30Z" } 2023-10-17T18:33:30.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.134621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.459915Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.592314Z'}}] 2023-10-17T18:33:30.603 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:30.604 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002625', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-17T18:33:30.605 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324 2023-10-17T18:33:35.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:35.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:35.609 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:35.666 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:35.667 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:35.713 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:35.713 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:35.714 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:35.714 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:35.714 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:35.714 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:33:35.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.715 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:35.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:35.715 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-17T18:33:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:35.717 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:35.717 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.134621Z" 2023-10-17T18:33:35.717 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:35.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:35.718 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:35.718 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:35.718 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:35.718 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:35.719 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:35.719 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:35.719 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:35.719 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.459915Z" 2023-10-17T18:33:35.720 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:35.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:35.720 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:35.720 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:35.721 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:35.721 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:35.721 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:35.721 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:35.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:35.722 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:16:29.368051Z" 2023-10-17T18:33:35.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:35.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:35.723 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:35.723 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:35.723 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:35.723 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-17T18:33:35.723 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:35.724 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:35.724 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:35.724 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.592314Z" 2023-10-17T18:33:35.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:35.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:35.725 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:35.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.725 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:35.725 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:35.726 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:35.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:35.726 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:35Z" 2023-10-17T18:33:35.726 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:35.727 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.134621Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.459915Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:16:29.368051Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.592314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:35Z" } 2023-10-17T18:33:35.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.134621Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.459915Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:16:29.368051Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.592314Z'}}] 2023-10-17T18:33:35.728 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:35.729 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002625', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-17T18:33:35.729 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003467 target=00000000000000000000:00000000000000002625, shard=1 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001211 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324 2023-10-17T18:33:38.396 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.395+0000 7fc7bbc0f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.397 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.395+0000 7fc7bf416700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.397 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.395+0000 7fc7c141a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.397 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.395+0000 7fc7c0c19700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.400 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.398+0000 7fc7c3c1f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.401 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7bcc11700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.401 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c5c23700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.401 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7b940a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.401 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c2c1d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.402 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7bc410700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.402 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c1c1b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.402 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c1c1b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.402 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c341e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.402 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c341e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.403 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c341e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:38.403 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T18:33:38.399+0000 7fc7c5422700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T18:33:40.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.731 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:40.792 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:40.793 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:40.842 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:40.842 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:40.843 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:40.843 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:40.843 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:40.843 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:33:40.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.844 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:40.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.844 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:40.844 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:40.845 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:40.845 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003467", 2023-10-17T18:33:40.845 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:40.845 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:40.845 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:40.846 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:19.740062Z" 2023-10-17T18:33:40.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.846 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:40.847 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:40.847 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:40.847 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-17T18:33:40.847 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:40.847 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:19.740934Z" 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.848 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:40.849 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:40.849 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:40.849 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001211", 2023-10-17T18:33:40.849 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:40.850 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:40.850 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:40.850 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:19.738037Z" 2023-10-17T18:33:40.850 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.851 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:40.851 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:40.851 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:40.852 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-17T18:33:40.852 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:40.852 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:40.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:40.853 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:19.739063Z" 2023-10-17T18:33:40.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:40.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.854 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:40.854 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:40.854 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:40.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.855 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:40Z" 2023-10-17T18:33:40.855 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:40.856 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003467", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:19.740062Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:19.740934Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001211", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:19.738037Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004167", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:19.739063Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:40Z" } 2023-10-17T18:33:40.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003467', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:19.740062Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:19.740934Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001211', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:19.738037Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004167', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:19.739063Z'}}] 2023-10-17T18:33:40.857 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:40.857 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003467', 1: '00000000000000000000:00000000000000002055', 2: '00000000000000000000:00000000000000001211', 3: '00000000000000000000:00000000000000004167'} 2023-10-17T18:33:40.858 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:33:40.858 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T18:33:40.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.859 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:40.911 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:40.911 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:40.958 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:33:40.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.959 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002483", 2023-10-17T18:33:40.959 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:36.965951Z" 2023-10-17T18:33:40.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002056", 2023-10-17T18:33:40.960 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:36.967181Z" 2023-10-17T18:33:40.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.961 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001212", 2023-10-17T18:33:40.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:36.963542Z" 2023-10-17T18:33:40.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:40.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:40.961 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000005014", 2023-10-17T18:33:40.962 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:36.964668Z" 2023-10-17T18:33:40.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:40.962 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:33:40.963 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002056', 2: '00000000000000000000:00000000000000001212', 3: '00000000000000000000:00000000000000005014'} 2023-10-17T18:33:40.964 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:33:40.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:40.965 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:41.061 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:41.062 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:41.062 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:41.063 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:41.063 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:41.063 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:33:41.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.064 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:41.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:41.064 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:41.064 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:41.065 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:41.065 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-17T18:33:41.065 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:41.065 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:41.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:41.066 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.432433Z" 2023-10-17T18:33:41.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:41.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:41.066 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:41.067 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:41.067 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:41.067 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:41.067 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:41.068 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:41.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:41.068 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.471975Z" 2023-10-17T18:33:41.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:41.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:41.069 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:41.069 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:41.069 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:41.069 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:41.070 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:41.070 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:41.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:41.070 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:33:41.071 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:41.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:41.071 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:41.071 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:41.072 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:41.072 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-17T18:33:41.072 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:41.072 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:41.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:41.073 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.629973Z" 2023-10-17T18:33:41.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:41.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:41.073 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:41.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.074 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:41.074 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:41.074 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:41.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:41.075 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:41Z" 2023-10-17T18:33:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:41.076 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.432433Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.471975Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.629973Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:41Z" } 2023-10-17T18:33:41.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.432433Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.471975Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.629973Z'}}] 2023-10-17T18:33:41.077 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002056', 2: '00000000000000000000:00000000000000001212', 3: '00000000000000000000:00000000000000005014'} 2023-10-17T18:33:41.078 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:33:41.078 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002483 target=00000000000000000000:00000000000000001640, shard=1 master=00000000000000000000:00000000000000002056 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001212 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000005014 target=00000000000000000000:00000000000000004170 2023-10-17T18:33:46.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:46.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:46.081 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:46.144 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:46.145 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:46.192 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:46.193 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:46.194 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:33:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:46.196 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:46.197 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-17T18:33:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:46.198 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:28:48.432433Z" 2023-10-17T18:33:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:46.199 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-17T18:33:46.200 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:49.471975Z" 2023-10-17T18:33:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:46.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:46.202 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-17T18:33:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:14:29.388077Z" 2023-10-17T18:33:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:46.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:46.205 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-17T18:33:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:46.206 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:25:50.629973Z" 2023-10-17T18:33:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:46.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:46.208 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:46Z" 2023-10-17T18:33:46.209 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:46.210 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:28:48.432433Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:49.471975Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:14:29.388077Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:25:50.629973Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:46Z" } 2023-10-17T18:33:46.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:28:48.432433Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:49.471975Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:14:29.388077Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:25:50.629973Z'}}] 2023-10-17T18:33:46.211 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002056', 2: '00000000000000000000:00000000000000001212', 3: '00000000000000000000:00000000000000005014'} 2023-10-17T18:33:46.211 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-17T18:33:46.211 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002483 target=00000000000000000000:00000000000000001640, shard=1 master=00000000000000000000:00000000000000002056 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001212 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000005014 target=00000000000000000000:00000000000000004170 2023-10-17T18:33:51.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.214 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:51.276 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:51.276 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:51.325 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:51.326 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:51.326 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:51.326 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:51.326 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:51.327 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:33:51.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.327 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:51.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.327 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:51.328 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.328 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:51.328 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002483", 2023-10-17T18:33:51.328 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.328 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.329 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:36.965951Z" 2023-10-17T18:33:51.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.330 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:51.330 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.330 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:51.330 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002056", 2023-10-17T18:33:51.331 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.331 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.331 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:36.967181Z" 2023-10-17T18:33:51.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.332 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:51.332 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.332 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:51.333 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001212", 2023-10-17T18:33:51.333 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.333 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.333 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.334 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:36.963542Z" 2023-10-17T18:33:51.334 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.335 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:51.335 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.335 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:33:51.335 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000005014", 2023-10-17T18:33:51.335 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.336 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.336 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.336 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:36.964668Z" 2023-10-17T18:33:51.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.337 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:51.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.337 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:51.337 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:33:51.337 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:33:51.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.338 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:51Z" 2023-10-17T18:33:51.338 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:51.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002483", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:36.965951Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002056", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:36.967181Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001212", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:36.963542Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005014", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:36.964668Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:33:51Z" } 2023-10-17T18:33:51.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002483', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:36.965951Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002056', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:36.967181Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001212', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:36.963542Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005014', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:36.964668Z'}}] 2023-10-17T18:33:51.340 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002056', 2: '00000000000000000000:00000000000000001212', 3: '00000000000000000000:00000000000000005014'} 2023-10-17T18:33:51.341 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002056', 2: '00000000000000000000:00000000000000001212', 3: '00000000000000000000:00000000000000005014'} 2023-10-17T18:33:51.341 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:33:51.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-10-17T18:33:51.343 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-46 2023-10-17T18:33:51.343 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:33:51.344 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:33:51.344 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.345 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:33:51.345 DEBUG:boto:Data: 2023-10-17T18:33:51.346 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.346 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.347 DEBUG:boto:Params: {} 2023-10-17T18:33:51.347 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:33:51.348 DEBUG:boto:Token: None 2023-10-17T18:33:51.349 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/ 2023-10-17T18:33:51.349 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t+xU1cH8vfwRz3Y6YjYoxSw6mBU= 2023-10-17T18:33:51.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t+xU1cH8vfwRz3Y6YjYoxSw6mBU=', 'Content-Length': '0'} 2023-10-17T18:33:51.366 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6cb7e22d861dc57-00652ed38f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.366 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-47 2023-10-17T18:33:51.367 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:33:51.367 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:33:51.368 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.368 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:33:51.368 DEBUG:boto:Data: 2023-10-17T18:33:51.369 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.369 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.370 DEBUG:boto:Params: {} 2023-10-17T18:33:51.370 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:33:51.371 DEBUG:boto:Token: None 2023-10-17T18:33:51.371 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/ 2023-10-17T18:33:51.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X5K6Eh9Yqt4j9FAG308MPKY3A3Y= 2023-10-17T18:33:51.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X5K6Eh9Yqt4j9FAG308MPKY3A3Y=', 'Content-Length': '0'} 2023-10-17T18:33:51.403 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ac8955282651087-00652ed38f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.403 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:33:51.404 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:33:51.404 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.404 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:33:51.405 DEBUG:boto:Data: 2023-10-17T18:33:51.405 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.406 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.406 DEBUG:boto:Params: {} 2023-10-17T18:33:51.407 DEBUG:boto:Token: None 2023-10-17T18:33:51.407 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/ 2023-10-17T18:33:51.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc= 2023-10-17T18:33:51.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc=', 'Content-Length': '0'} 2023-10-17T18:33:51.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b566e814189aaa1-00652ed38f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.411 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:33:51.412 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:33:51.412 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.413 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:33:51.413 DEBUG:boto:Data: 2023-10-17T18:33:51.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.414 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.415 DEBUG:boto:Params: {} 2023-10-17T18:33:51.415 DEBUG:boto:Token: None 2023-10-17T18:33:51.416 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/obj1 2023-10-17T18:33:51.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/iJzLE0Kb830MZpLlKmvVEuKV+g= 2023-10-17T18:33:51.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/iJzLE0Kb830MZpLlKmvVEuKV+g='} 2023-10-17T18:33:51.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025cb925698bb3a91-00652ed38f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.423 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:33:51.423 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:33:51.424 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.424 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:33:51.424 DEBUG:boto:Data: 2023-10-17T18:33:51.425 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.425 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.426 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.426 DEBUG:boto:Params: {} 2023-10-17T18:33:51.426 DEBUG:boto:Token: None 2023-10-17T18:33:51.427 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/ 2023-10-17T18:33:51.427 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc= 2023-10-17T18:33:51.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc=', 'Content-Length': '0'} 2023-10-17T18:33:51.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080c818cd1e8a61e5-00652ed38f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.431 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:33:51.431 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:33:51.431 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.432 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:33:51.432 DEBUG:boto:Data: 2023-10-17T18:33:51.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.433 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.433 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.434 DEBUG:boto:Params: {} 2023-10-17T18:33:51.434 DEBUG:boto:Token: None 2023-10-17T18:33:51.435 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/obj2 2023-10-17T18:33:51.435 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+5GQ5G5UCY1xiLXWwT1x1ulsIpk= 2023-10-17T18:33:51.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+5GQ5G5UCY1xiLXWwT1x1ulsIpk='} 2023-10-17T18:33:51.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a4dc8d3284106bd-00652ed38f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.443 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:33:51.444 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:33:51.444 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.445 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:33:51.445 DEBUG:boto:Data: 2023-10-17T18:33:51.445 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.446 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.446 DEBUG:boto:Params: {} 2023-10-17T18:33:51.447 DEBUG:boto:Token: None 2023-10-17T18:33:51.447 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/ 2023-10-17T18:33:51.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc= 2023-10-17T18:33:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc=', 'Content-Length': '0'} 2023-10-17T18:33:51.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000462c15cd57e782d4-00652ed38f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.451 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:33:51.451 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:33:51.452 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.452 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:33:51.453 DEBUG:boto:Data: 2023-10-17T18:33:51.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.454 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.454 DEBUG:boto:Params: {} 2023-10-17T18:33:51.455 DEBUG:boto:Token: None 2023-10-17T18:33:51.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/obj3 2023-10-17T18:33:51.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uXJed0M73jAdZZqpghwHk4T7M4g= 2023-10-17T18:33:51.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uXJed0M73jAdZZqpghwHk4T7M4g='} 2023-10-17T18:33:51.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dddf45496323afc-00652ed38f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.499 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:33:51.499 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:33:51.500 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.500 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:33:51.501 DEBUG:boto:Data: 2023-10-17T18:33:51.501 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.502 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.502 DEBUG:boto:Params: {} 2023-10-17T18:33:51.503 DEBUG:boto:Token: None 2023-10-17T18:33:51.503 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/ 2023-10-17T18:33:51.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc= 2023-10-17T18:33:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vyr9VPdib56oMt3nrufAfEdEabc=', 'Content-Length': '0'} 2023-10-17T18:33:51.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006461210ab7ff25a-00652ed38f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.507 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:33:51.507 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:33:51.508 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.508 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:33:51.509 DEBUG:boto:Data: 2023-10-17T18:33:51.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.509 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:33:51.510 DEBUG:boto:Port: 8000 2023-10-17T18:33:51.510 DEBUG:boto:Params: {} 2023-10-17T18:33:51.511 DEBUG:boto:Token: None 2023-10-17T18:33:51.511 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-46/obj4 2023-10-17T18:33:51.511 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EmtmAbs0+YkpmGyOMEAhnsewPYg= 2023-10-17T18:33:51.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EmtmAbs0+YkpmGyOMEAhnsewPYg='} 2023-10-17T18:33:51.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054a7db9ddde8b31a-00652ed38f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.517 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:33:51.518 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:33:51.518 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.519 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:33:51.521 DEBUG:boto:Data: 2023-10-17T18:33:51.521 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.522 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.523 DEBUG:boto:Params: {} 2023-10-17T18:33:51.523 DEBUG:boto:Token: None 2023-10-17T18:33:51.524 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/ 2023-10-17T18:33:51.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo= 2023-10-17T18:33:51.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo=', 'Content-Length': '0'} 2023-10-17T18:33:51.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a53a5ffaa21c1b8-00652ed38f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.528 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:33:51.528 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:33:51.529 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.529 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:33:51.529 DEBUG:boto:Data: 2023-10-17T18:33:51.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.531 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.531 DEBUG:boto:Params: {} 2023-10-17T18:33:51.531 DEBUG:boto:Token: None 2023-10-17T18:33:51.532 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/obj1 2023-10-17T18:33:51.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:42IpG5ZVF2igkjO9A2z/0Zsp3zY= 2023-10-17T18:33:51.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:42IpG5ZVF2igkjO9A2z/0Zsp3zY='} 2023-10-17T18:33:51.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d657c57adf8c24e-00652ed38f-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.539 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:33:51.540 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:33:51.540 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.540 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:33:51.541 DEBUG:boto:Data: 2023-10-17T18:33:51.541 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.542 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.542 DEBUG:boto:Params: {} 2023-10-17T18:33:51.543 DEBUG:boto:Token: None 2023-10-17T18:33:51.543 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/ 2023-10-17T18:33:51.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo= 2023-10-17T18:33:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo=', 'Content-Length': '0'} 2023-10-17T18:33:51.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c14baa2197f26de-00652ed38f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.547 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:33:51.547 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:33:51.547 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.548 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:33:51.548 DEBUG:boto:Data: 2023-10-17T18:33:51.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.549 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.550 DEBUG:boto:Params: {} 2023-10-17T18:33:51.550 DEBUG:boto:Token: None 2023-10-17T18:33:51.551 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/obj2 2023-10-17T18:33:51.551 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B+Wed4+GspCDPMtzLS18YxombzA= 2023-10-17T18:33:51.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+Wed4+GspCDPMtzLS18YxombzA='} 2023-10-17T18:33:51.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085d1647fd4f6a028-00652ed38f-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.557 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:33:51.558 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:33:51.558 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.558 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:33:51.559 DEBUG:boto:Data: 2023-10-17T18:33:51.559 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.560 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.560 DEBUG:boto:Params: {} 2023-10-17T18:33:51.561 DEBUG:boto:Token: None 2023-10-17T18:33:51.561 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/ 2023-10-17T18:33:51.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo= 2023-10-17T18:33:51.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo=', 'Content-Length': '0'} 2023-10-17T18:33:51.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008caa4feb5022f867-00652ed38f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.565 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:33:51.565 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:33:51.565 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.566 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:33:51.566 DEBUG:boto:Data: 2023-10-17T18:33:51.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.567 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.568 DEBUG:boto:Params: {} 2023-10-17T18:33:51.568 DEBUG:boto:Token: None 2023-10-17T18:33:51.569 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/obj3 2023-10-17T18:33:51.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9CVf1d45/f5k1xGdnGOs9lPZpoY= 2023-10-17T18:33:51.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9CVf1d45/f5k1xGdnGOs9lPZpoY='} 2023-10-17T18:33:51.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022c6283d73df7f98-00652ed38f-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.615 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:33:51.616 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:33:51.616 DEBUG:boto:Method: HEAD 2023-10-17T18:33:51.616 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:33:51.617 DEBUG:boto:Data: 2023-10-17T18:33:51.617 DEBUG:boto:Headers: {} 2023-10-17T18:33:51.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.618 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.618 DEBUG:boto:Params: {} 2023-10-17T18:33:51.619 DEBUG:boto:Token: None 2023-10-17T18:33:51.619 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/ 2023-10-17T18:33:51.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo= 2023-10-17T18:33:51.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFDMdwE9R8DJ1RFEG4/ZWUz3Eeo=', 'Content-Length': '0'} 2023-10-17T18:33:51.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aee24253555aa736-00652ed38f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.623 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:33:51.623 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:33:51.624 DEBUG:boto:Method: PUT 2023-10-17T18:33:51.624 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:33:51.624 DEBUG:boto:Data: 2023-10-17T18:33:51.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:33:51.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:33:51.626 DEBUG:boto:Port: 8001 2023-10-17T18:33:51.626 DEBUG:boto:Params: {} 2023-10-17T18:33:51.627 DEBUG:boto:Token: None 2023-10-17T18:33:51.627 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:33:51 GMT /hbmuog-47/obj4 2023-10-17T18:33:51.627 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kOKYNVEVhqoCOijTL8a8ME0HhR8= 2023-10-17T18:33:51.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:33:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kOKYNVEVhqoCOijTL8a8ME0HhR8='} 2023-10-17T18:33:51.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002db4357a6d75d716-00652ed38f-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:33:51.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.634 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:51.683 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:51.684 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:51.714 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:33:51.726 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:33:51.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.726 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567631.389945_529.1", 2023-10-17T18:33:51.727 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:51.389945Z" 2023-10-17T18:33:51.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.727 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:51.727 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:33:51.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.728 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:51.728 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:33:51.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.729 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:51.729 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:33:51.729 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.729 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:33:51.730 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567631.389945_529.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:51.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:33:51.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:33:51.732 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:33:51.783 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:51.783 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:51.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:51.812+0000 7f8a0b6cd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:51.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:51.813+0000 7f8a0b6cd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:33:51.827 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:33:51.827 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:33:51.827 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:33:51.827 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:33:51.827 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:33:51.828 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:33:51.828 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:51.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.828 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:33:51.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.829 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:33:51.829 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.829 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:51.829 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567631.389945_529.1", 2023-10-17T18:33:51.830 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.830 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:33:51.830 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.830 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:51.389945Z", 2023-10-17T18:33:51.831 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:51.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.831 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:33:51.832 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.832 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:51.832 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:33:51.832 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.832 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.833 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.833 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:33:51.833 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:51.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.834 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:33:51.834 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.834 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:51.835 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:51.835 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.835 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.835 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.836 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:51.836 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:51.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:33:51.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:33:51.837 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:33:51.837 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:33:51.837 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:33:51.837 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:33:51.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:33:51.838 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:33:51.838 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:33:51.838 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:33:51.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:33:51.839 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:33:51.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.840 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:33:51.840 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:33:51.840 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:33:51.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:33:51.840 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:33:51Z" 2023-10-17T18:33:51.841 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:33:51.842 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567631.389945_529.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:33:51.389945Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:33:51Z" } 2023-10-17T18:33:51.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567631.389945_529.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:33:51.389945Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:33:51.843 DEBUG:rgw_multi.tests:log_status={0: '1_1697567631.389945_529.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:51.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567631.389945_529.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:33:51.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:33:51.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-46 2023-10-17T18:33:51.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:33:51.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:33:51.845 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:33:51.903 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:33:51.904 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:33:51.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:51.939+0000 7f99bc4dc780 1 waiting to reach incremental sync.. 2023-10-17T18:33:56.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:33:56.940+0000 7f99bc4dc780 1 waiting to reach incremental sync.. 2023-10-17T18:34:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:01.942+0000 7f99bc4dc780 1 bucket sync caught up with source: 2023-10-17T18:34:01.943 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.54.6, , 00000000001.9.6, , 00000000001.35.6, , , 00000000001.14.6, , , ] 2023-10-17T18:34:01.943 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.54.6, , 00000000001.9.6, , 00000000001.35.6, , , 00000000001.14.6, , , ] 2023-10-17T18:34:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:01.942+0000 7f99bc4dc780 0 bucket checkpoint complete 2023-10-17T18:34:01.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-46 2023-10-17T18:34:01.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:34:01.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:34:01.961 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:34:02.024 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:34:02.025 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:34:02.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:02.061+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:07.060+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:12.062 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:12.061+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:17.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:17.061+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:22.062+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:27.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:27.063+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:32.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:32.064+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:37.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:37.065+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:42.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:42.065+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:47.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:47.065+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:52.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:52.066+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:34:57.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:34:57.066+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:02.066+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:07.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:07.068+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:12.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:12.068+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:17.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:17.069+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:22.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:22.069+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:27.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:27.070+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:32.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:32.070+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:37.071+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:42.072+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:47.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:47.073+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:52.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:52.074+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:35:57.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:35:57.073+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:02.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:02.074+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:07.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:07.074+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:12.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:12.075+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:17.076+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:22.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:22.077+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:27.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:27.078+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:32.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:32.078+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:37.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:37.078+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:42.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:42.079+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:47.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:47.080+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:52.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:52.081+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:36:57.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:36:57.082+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:37:02.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:02.083+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:37:07.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:07.083+0000 7f58fc54d780 1 waiting to reach incremental sync.. 2023-10-17T18:37:12.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:12.084+0000 7f58fc54d780 1 bucket sync caught up with source: 2023-10-17T18:37:12.086 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.142.6, , 00000000001.39.6, , 00000000001.128.6, , , 00000000001.106.6, , , ] 2023-10-17T18:37:12.086 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.142.6, , 00000000001.39.6, , 00000000001.128.6, , , 00000000001.106.6, , , ] 2023-10-17T18:37:12.087 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:12.084+0000 7f58fc54d780 0 bucket checkpoint complete 2023-10-17T18:37:12.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-46 zones={a2, a1} 2023-10-17T18:37:12.105 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.106 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.106 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.107 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:12.107 DEBUG:boto:Data: 2023-10-17T18:37:12.107 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.108 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.109 DEBUG:boto:Params: {} 2023-10-17T18:37:12.109 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:37:12.110 DEBUG:boto:Token: None 2023-10-17T18:37:12.111 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7LSfKzx4LLIXY67DMYRxOZ5F8L0= 2023-10-17T18:37:12.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7LSfKzx4LLIXY67DMYRxOZ5F8L0=', 'Content-Length': '0'} 2023-10-17T18:37:12.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000080db6005818e5e8-00652ed458-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.117 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.117 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.117 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.118 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:12.118 DEBUG:boto:Data: 2023-10-17T18:37:12.119 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.120 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.120 DEBUG:boto:Params: {} 2023-10-17T18:37:12.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:37:12.121 DEBUG:boto:Token: None 2023-10-17T18:37:12.121 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.122 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7LSfKzx4LLIXY67DMYRxOZ5F8L0= 2023-10-17T18:37:12.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7LSfKzx4LLIXY67DMYRxOZ5F8L0=', 'Content-Length': '0'} 2023-10-17T18:37:12.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6634804e690184e-00652ed458-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:37:12.127 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.128 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.128 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.129 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.129 DEBUG:boto:Method: GET 2023-10-17T18:37:12.129 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.130 DEBUG:boto:Data: 2023-10-17T18:37:12.130 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.131 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.132 DEBUG:boto:Params: {} 2023-10-17T18:37:12.132 DEBUG:boto:Token: None 2023-10-17T18:37:12.133 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/?versions 2023-10-17T18:37:12.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408= 2023-10-17T18:37:12.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408=', 'Content-Length': '0'} 2023-10-17T18:37:12.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ac3dc54967a975-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.137 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:12.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:12.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:12.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:12.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:37:12.140 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.140 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.141 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.141 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.141 DEBUG:boto:Method: GET 2023-10-17T18:37:12.142 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.142 DEBUG:boto:Data: 2023-10-17T18:37:12.143 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.144 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.144 DEBUG:boto:Params: {} 2023-10-17T18:37:12.144 DEBUG:boto:Token: None 2023-10-17T18:37:12.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/?versions 2023-10-17T18:37:12.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408= 2023-10-17T18:37:12.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408=', 'Content-Length': '0'} 2023-10-17T18:37:12.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002df08d03dfb28193-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.149 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:12.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:12.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:12.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:12.151 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.152 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.152 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.153 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.153 DEBUG:boto:Method: GET 2023-10-17T18:37:12.153 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.154 DEBUG:boto:Data: 2023-10-17T18:37:12.154 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.155 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.156 DEBUG:boto:Params: {} 2023-10-17T18:37:12.156 DEBUG:boto:Token: None 2023-10-17T18:37:12.156 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/?versions 2023-10-17T18:37:12.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408= 2023-10-17T18:37:12.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408=', 'Content-Length': '0'} 2023-10-17T18:37:12.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a746da399c3aeba-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.160 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.161 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.162 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.162 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.163 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.163 DEBUG:boto:Method: GET 2023-10-17T18:37:12.163 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:12.164 DEBUG:boto:Data: 2023-10-17T18:37:12.164 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.165 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.165 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.166 DEBUG:boto:Params: {} 2023-10-17T18:37:12.166 DEBUG:boto:Token: None 2023-10-17T18:37:12.167 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/?versions 2023-10-17T18:37:12.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408= 2023-10-17T18:37:12.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0OQp3T4igz+Q2WdZU25eebtp408=', 'Content-Length': '0'} 2023-10-17T18:37:12.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9049aea169df4bb-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.170 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:12.172 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.172 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.173 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.173 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.173 DEBUG:boto:Method: GET 2023-10-17T18:37:12.174 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.174 DEBUG:boto:Data: 2023-10-17T18:37:12.175 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.176 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.176 DEBUG:boto:Params: {} 2023-10-17T18:37:12.177 DEBUG:boto:Token: None 2023-10-17T18:37:12.177 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.178 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA= 2023-10-17T18:37:12.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA=', 'Content-Length': '0'} 2023-10-17T18:37:12.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cb4979cb5ca9012-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.181 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.181 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.182 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.182 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.183 DEBUG:boto:Method: GET 2023-10-17T18:37:12.183 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.184 DEBUG:boto:Data: 2023-10-17T18:37:12.184 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.185 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.185 DEBUG:boto:Params: {} 2023-10-17T18:37:12.186 DEBUG:boto:Token: None 2023-10-17T18:37:12.187 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA= 2023-10-17T18:37:12.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA=', 'Content-Length': '0'} 2023-10-17T18:37:12.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0b10166a6e5a426-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.190 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.190 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.191 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.191 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.192 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.192 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.192 DEBUG:boto:Data: 2023-10-17T18:37:12.193 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.193 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.194 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.194 DEBUG:boto:Params: {} 2023-10-17T18:37:12.194 DEBUG:boto:Token: None 2023-10-17T18:37:12.195 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.195 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fAR0nsSTdWoTQPz8WuZGK550/2c= 2023-10-17T18:37:12.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fAR0nsSTdWoTQPz8WuZGK550/2c=', 'Content-Length': '0'} 2023-10-17T18:37:12.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8b036bf6ef68c13-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.198 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.199 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.199 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.200 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.200 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.201 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.201 DEBUG:boto:Data: 2023-10-17T18:37:12.201 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.202 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.203 DEBUG:boto:Params: {} 2023-10-17T18:37:12.203 DEBUG:boto:Token: None 2023-10-17T18:37:12.204 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fAR0nsSTdWoTQPz8WuZGK550/2c= 2023-10-17T18:37:12.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fAR0nsSTdWoTQPz8WuZGK550/2c=', 'Content-Length': '0'} 2023-10-17T18:37:12.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4da993fe39f7752-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:12.207 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.208 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.208 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.209 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.209 DEBUG:boto:Method: GET 2023-10-17T18:37:12.210 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.210 DEBUG:boto:Data: 2023-10-17T18:37:12.211 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.211 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.211 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.212 DEBUG:boto:Params: {} 2023-10-17T18:37:12.212 DEBUG:boto:Token: None 2023-10-17T18:37:12.213 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.213 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA= 2023-10-17T18:37:12.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA=', 'Content-Length': '0'} 2023-10-17T18:37:12.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2e9cc995b5c275-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.216 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.217 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.217 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.218 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.218 DEBUG:boto:Method: GET 2023-10-17T18:37:12.219 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.219 DEBUG:boto:Data: 2023-10-17T18:37:12.219 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.220 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.220 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.221 DEBUG:boto:Params: {} 2023-10-17T18:37:12.221 DEBUG:boto:Token: None 2023-10-17T18:37:12.222 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:12.222 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA= 2023-10-17T18:37:12.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RuGZ5OmSpJdlXgSaOyyAlRA9zcA=', 'Content-Length': '0'} 2023-10-17T18:37:12.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bf48717b5d1b9db-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.226 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.227 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.227 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.227 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.228 DEBUG:boto:Method: GET 2023-10-17T18:37:12.228 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.229 DEBUG:boto:Data: 2023-10-17T18:37:12.229 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.230 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.230 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.231 DEBUG:boto:Params: {} 2023-10-17T18:37:12.231 DEBUG:boto:Token: None 2023-10-17T18:37:12.232 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.232 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a55c34f3fd26d1-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.236 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.237 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.237 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.238 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.238 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.238 DEBUG:boto:Method: GET 2023-10-17T18:37:12.239 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.239 DEBUG:boto:Data: 2023-10-17T18:37:12.240 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.240 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.241 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.241 DEBUG:boto:Params: {} 2023-10-17T18:37:12.241 DEBUG:boto:Token: None 2023-10-17T18:37:12.242 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.242 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002289852f5ae8334e-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.246 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.247 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.247 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.247 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.248 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:12.248 DEBUG:boto:Data: 2023-10-17T18:37:12.249 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.250 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.250 DEBUG:boto:Params: {} 2023-10-17T18:37:12.251 DEBUG:boto:Token: None 2023-10-17T18:37:12.251 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1 2023-10-17T18:37:12.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r5JmXBCVIGLulCvauSgHmDmHwOU= 2023-10-17T18:37:12.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r5JmXBCVIGLulCvauSgHmDmHwOU=', 'Content-Length': '0'} 2023-10-17T18:37:12.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011e03742671754b5-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.255 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.255 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.256 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.256 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:12.256 DEBUG:boto:Data: 2023-10-17T18:37:12.257 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.258 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.258 DEBUG:boto:Params: {} 2023-10-17T18:37:12.259 DEBUG:boto:Token: None 2023-10-17T18:37:12.259 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1 2023-10-17T18:37:12.260 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r5JmXBCVIGLulCvauSgHmDmHwOU= 2023-10-17T18:37:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r5JmXBCVIGLulCvauSgHmDmHwOU=', 'Content-Length': '0'} 2023-10-17T18:37:12.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074c76207c8ce2e57-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:12.263 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.264 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.264 DEBUG:boto:Method: GET 2023-10-17T18:37:12.264 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:12.265 DEBUG:boto:Data: 2023-10-17T18:37:12.265 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.266 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.266 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.267 DEBUG:boto:Params: {} 2023-10-17T18:37:12.267 DEBUG:boto:Token: None 2023-10-17T18:37:12.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1 2023-10-17T18:37:12.268 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NggEZ7gqXJr6xWItxG+Lg0XJMJI= 2023-10-17T18:37:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NggEZ7gqXJr6xWItxG+Lg0XJMJI=', 'Content-Length': '0'} 2023-10-17T18:37:12.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c08a67467af0fc3-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.271 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:12.271 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:12.272 DEBUG:boto:Method: GET 2023-10-17T18:37:12.272 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:12.273 DEBUG:boto:Data: 2023-10-17T18:37:12.273 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.274 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.274 DEBUG:boto:Params: {} 2023-10-17T18:37:12.275 DEBUG:boto:Token: None 2023-10-17T18:37:12.275 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj1 2023-10-17T18:37:12.276 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NggEZ7gqXJr6xWItxG+Lg0XJMJI= 2023-10-17T18:37:12.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NggEZ7gqXJr6xWItxG+Lg0XJMJI=', 'Content-Length': '0'} 2023-10-17T18:37:12.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0e576bd4e1ab20a-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.279 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.279 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.280 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.280 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.281 DEBUG:boto:Method: GET 2023-10-17T18:37:12.281 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.282 DEBUG:boto:Data: 2023-10-17T18:37:12.282 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.282 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.283 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.283 DEBUG:boto:Params: {} 2023-10-17T18:37:12.284 DEBUG:boto:Token: None 2023-10-17T18:37:12.284 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.285 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb1bc0c1c9abcb31-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.288 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.289 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.289 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.290 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.290 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.290 DEBUG:boto:Method: GET 2023-10-17T18:37:12.291 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:12.291 DEBUG:boto:Data: 2023-10-17T18:37:12.292 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.292 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.293 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.293 DEBUG:boto:Params: {} 2023-10-17T18:37:12.293 DEBUG:boto:Token: None 2023-10-17T18:37:12.294 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.294 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd2fe189707b240-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.342 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:12.344 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.344 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.345 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.345 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.345 DEBUG:boto:Method: GET 2023-10-17T18:37:12.346 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.346 DEBUG:boto:Data: 2023-10-17T18:37:12.347 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.347 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.348 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.348 DEBUG:boto:Params: {} 2023-10-17T18:37:12.348 DEBUG:boto:Token: None 2023-10-17T18:37:12.349 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU= 2023-10-17T18:37:12.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU=', 'Content-Length': '0'} 2023-10-17T18:37:12.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1b3213b07cb8f84-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.353 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.353 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.354 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.354 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.355 DEBUG:boto:Method: GET 2023-10-17T18:37:12.355 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.355 DEBUG:boto:Data: 2023-10-17T18:37:12.356 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.356 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.357 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.357 DEBUG:boto:Params: {} 2023-10-17T18:37:12.358 DEBUG:boto:Token: None 2023-10-17T18:37:12.358 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU= 2023-10-17T18:37:12.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU=', 'Content-Length': '0'} 2023-10-17T18:37:12.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000839953277fe19272-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.362 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.363 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.363 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.364 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.364 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.364 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.365 DEBUG:boto:Data: 2023-10-17T18:37:12.365 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.366 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.367 DEBUG:boto:Params: {} 2023-10-17T18:37:12.367 DEBUG:boto:Token: None 2023-10-17T18:37:12.368 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZsSooLhD2TewXAsm2bh19TnP/tE= 2023-10-17T18:37:12.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZsSooLhD2TewXAsm2bh19TnP/tE=', 'Content-Length': '0'} 2023-10-17T18:37:12.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af23b57be633aaae-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.371 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.371 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.372 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.372 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.373 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.373 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.373 DEBUG:boto:Data: 2023-10-17T18:37:12.374 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.375 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.375 DEBUG:boto:Params: {} 2023-10-17T18:37:12.376 DEBUG:boto:Token: None 2023-10-17T18:37:12.376 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZsSooLhD2TewXAsm2bh19TnP/tE= 2023-10-17T18:37:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZsSooLhD2TewXAsm2bh19TnP/tE=', 'Content-Length': '0'} 2023-10-17T18:37:12.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000507dfdfd3d653b4a-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:12.380 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.381 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.381 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.381 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.382 DEBUG:boto:Method: GET 2023-10-17T18:37:12.382 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.383 DEBUG:boto:Data: 2023-10-17T18:37:12.383 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.384 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.384 DEBUG:boto:Params: {} 2023-10-17T18:37:12.385 DEBUG:boto:Token: None 2023-10-17T18:37:12.385 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.386 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU= 2023-10-17T18:37:12.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU=', 'Content-Length': '0'} 2023-10-17T18:37:12.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ce2e32353d32ed1-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.430 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.431 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.432 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.432 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.433 DEBUG:boto:Method: GET 2023-10-17T18:37:12.433 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.433 DEBUG:boto:Data: 2023-10-17T18:37:12.434 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.435 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.435 DEBUG:boto:Params: {} 2023-10-17T18:37:12.436 DEBUG:boto:Token: None 2023-10-17T18:37:12.436 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:12.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU= 2023-10-17T18:37:12.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaGvwlQLUlAP39tCDw3044p1PgU=', 'Content-Length': '0'} 2023-10-17T18:37:12.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095eb2b6bad35368a-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.440 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.440 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.441 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.441 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.442 DEBUG:boto:Method: GET 2023-10-17T18:37:12.442 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.443 DEBUG:boto:Data: 2023-10-17T18:37:12.443 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.444 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.444 DEBUG:boto:Params: {} 2023-10-17T18:37:12.445 DEBUG:boto:Token: None 2023-10-17T18:37:12.445 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea5d020656efabe9-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.449 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.450 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.451 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.451 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.452 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.452 DEBUG:boto:Method: GET 2023-10-17T18:37:12.452 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.453 DEBUG:boto:Data: 2023-10-17T18:37:12.453 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.454 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.455 DEBUG:boto:Params: {} 2023-10-17T18:37:12.455 DEBUG:boto:Token: None 2023-10-17T18:37:12.456 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c6b68074d495e79-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.460 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.460 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.461 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.461 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.462 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:12.462 DEBUG:boto:Data: 2023-10-17T18:37:12.462 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.463 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.464 DEBUG:boto:Params: {} 2023-10-17T18:37:12.464 DEBUG:boto:Token: None 2023-10-17T18:37:12.465 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2 2023-10-17T18:37:12.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:znVoOawLbconFbKNh6c1hg43oFg= 2023-10-17T18:37:12.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:znVoOawLbconFbKNh6c1hg43oFg=', 'Content-Length': '0'} 2023-10-17T18:37:12.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba81307285256517-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.468 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.469 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.469 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.469 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:12.470 DEBUG:boto:Data: 2023-10-17T18:37:12.470 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.471 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.471 DEBUG:boto:Params: {} 2023-10-17T18:37:12.472 DEBUG:boto:Token: None 2023-10-17T18:37:12.472 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2 2023-10-17T18:37:12.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:znVoOawLbconFbKNh6c1hg43oFg= 2023-10-17T18:37:12.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:znVoOawLbconFbKNh6c1hg43oFg=', 'Content-Length': '0'} 2023-10-17T18:37:12.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ab879529f68e80d-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:12.476 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.477 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.477 DEBUG:boto:Method: GET 2023-10-17T18:37:12.478 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:12.478 DEBUG:boto:Data: 2023-10-17T18:37:12.479 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.479 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.480 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.480 DEBUG:boto:Params: {} 2023-10-17T18:37:12.481 DEBUG:boto:Token: None 2023-10-17T18:37:12.481 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2 2023-10-17T18:37:12.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mENesKAVcVdm0UM5Z1bror+y22I= 2023-10-17T18:37:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mENesKAVcVdm0UM5Z1bror+y22I=', 'Content-Length': '0'} 2023-10-17T18:37:12.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a808987ef887cb42-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.485 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:12.485 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:12.486 DEBUG:boto:Method: GET 2023-10-17T18:37:12.486 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:12.486 DEBUG:boto:Data: 2023-10-17T18:37:12.487 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.488 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.488 DEBUG:boto:Params: {} 2023-10-17T18:37:12.489 DEBUG:boto:Token: None 2023-10-17T18:37:12.489 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj2 2023-10-17T18:37:12.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mENesKAVcVdm0UM5Z1bror+y22I= 2023-10-17T18:37:12.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mENesKAVcVdm0UM5Z1bror+y22I=', 'Content-Length': '0'} 2023-10-17T18:37:12.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a00a3a72fee02df1-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.538 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.539 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.539 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.540 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.540 DEBUG:boto:Method: GET 2023-10-17T18:37:12.541 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.541 DEBUG:boto:Data: 2023-10-17T18:37:12.542 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.542 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.543 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.543 DEBUG:boto:Params: {} 2023-10-17T18:37:12.544 DEBUG:boto:Token: None 2023-10-17T18:37:12.544 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.545 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e361eab4d58fd7d-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.550 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.551 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.552 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.552 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.553 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.554 DEBUG:boto:Method: GET 2023-10-17T18:37:12.554 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:12.555 DEBUG:boto:Data: 2023-10-17T18:37:12.555 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.556 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.556 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.557 DEBUG:boto:Params: {} 2023-10-17T18:37:12.557 DEBUG:boto:Token: None 2023-10-17T18:37:12.558 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.558 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060cbb4ec387980b-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.562 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:12.564 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.564 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.564 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.565 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.565 DEBUG:boto:Method: GET 2023-10-17T18:37:12.566 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.566 DEBUG:boto:Data: 2023-10-17T18:37:12.567 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.567 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.568 DEBUG:boto:Params: {} 2023-10-17T18:37:12.568 DEBUG:boto:Token: None 2023-10-17T18:37:12.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck= 2023-10-17T18:37:12.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck=', 'Content-Length': '0'} 2023-10-17T18:37:12.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc6905d3d45a8d71-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.573 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.573 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.574 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.574 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.575 DEBUG:boto:Method: GET 2023-10-17T18:37:12.575 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.576 DEBUG:boto:Data: 2023-10-17T18:37:12.576 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.577 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.578 DEBUG:boto:Params: {} 2023-10-17T18:37:12.578 DEBUG:boto:Token: None 2023-10-17T18:37:12.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck= 2023-10-17T18:37:12.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck=', 'Content-Length': '0'} 2023-10-17T18:37:12.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006771511ae343f527-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.582 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.583 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.583 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.584 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.584 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.584 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.585 DEBUG:boto:Data: 2023-10-17T18:37:12.585 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.586 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.586 DEBUG:boto:Params: {} 2023-10-17T18:37:12.587 DEBUG:boto:Token: None 2023-10-17T18:37:12.588 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JwzDik6xt2f0FCovEmXth0g7QUU= 2023-10-17T18:37:12.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JwzDik6xt2f0FCovEmXth0g7QUU=', 'Content-Length': '0'} 2023-10-17T18:37:12.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be63291346b1aa9d-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.591 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.592 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.592 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.592 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.593 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.593 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.594 DEBUG:boto:Data: 2023-10-17T18:37:12.594 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.595 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.595 DEBUG:boto:Params: {} 2023-10-17T18:37:12.596 DEBUG:boto:Token: None 2023-10-17T18:37:12.596 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JwzDik6xt2f0FCovEmXth0g7QUU= 2023-10-17T18:37:12.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JwzDik6xt2f0FCovEmXth0g7QUU=', 'Content-Length': '0'} 2023-10-17T18:37:12.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd6edf2c7e441e74-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:12.600 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.601 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.601 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.601 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.602 DEBUG:boto:Method: GET 2023-10-17T18:37:12.602 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.603 DEBUG:boto:Data: 2023-10-17T18:37:12.603 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.603 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.604 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.604 DEBUG:boto:Params: {} 2023-10-17T18:37:12.605 DEBUG:boto:Token: None 2023-10-17T18:37:12.605 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.606 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck= 2023-10-17T18:37:12.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck=', 'Content-Length': '0'} 2023-10-17T18:37:12.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a6e23611ff18bdb-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.609 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.609 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.610 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.610 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.611 DEBUG:boto:Method: GET 2023-10-17T18:37:12.611 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.611 DEBUG:boto:Data: 2023-10-17T18:37:12.612 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.613 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.613 DEBUG:boto:Params: {} 2023-10-17T18:37:12.614 DEBUG:boto:Token: None 2023-10-17T18:37:12.614 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:12.615 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck= 2023-10-17T18:37:12.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aZtR7N0NuAQkdx3eOGzxwbt+5Ck=', 'Content-Length': '0'} 2023-10-17T18:37:12.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed3b78fc821df3f6-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.618 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.618 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.619 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.619 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.620 DEBUG:boto:Method: GET 2023-10-17T18:37:12.620 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.620 DEBUG:boto:Data: 2023-10-17T18:37:12.621 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.622 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.622 DEBUG:boto:Params: {} 2023-10-17T18:37:12.622 DEBUG:boto:Token: None 2023-10-17T18:37:12.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005096ad26662a19e3-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.627 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.628 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.628 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.629 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.629 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.630 DEBUG:boto:Method: GET 2023-10-17T18:37:12.630 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.631 DEBUG:boto:Data: 2023-10-17T18:37:12.631 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.631 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.632 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.632 DEBUG:boto:Params: {} 2023-10-17T18:37:12.633 DEBUG:boto:Token: None 2023-10-17T18:37:12.633 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc9b16a1e16d461-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.637 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.638 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.639 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.639 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.639 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:12.640 DEBUG:boto:Data: 2023-10-17T18:37:12.640 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.641 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.641 DEBUG:boto:Params: {} 2023-10-17T18:37:12.642 DEBUG:boto:Token: None 2023-10-17T18:37:12.642 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3 2023-10-17T18:37:12.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nIIrQYCfjQtnZ48u8ifbzwd4uHI= 2023-10-17T18:37:12.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nIIrQYCfjQtnZ48u8ifbzwd4uHI=', 'Content-Length': '0'} 2023-10-17T18:37:12.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000711017786aab2e8c-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.646 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.646 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.647 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.647 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:12.647 DEBUG:boto:Data: 2023-10-17T18:37:12.648 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.649 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.649 DEBUG:boto:Params: {} 2023-10-17T18:37:12.649 DEBUG:boto:Token: None 2023-10-17T18:37:12.650 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3 2023-10-17T18:37:12.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nIIrQYCfjQtnZ48u8ifbzwd4uHI= 2023-10-17T18:37:12.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nIIrQYCfjQtnZ48u8ifbzwd4uHI=', 'Content-Length': '0'} 2023-10-17T18:37:12.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095c32e622b2a3522-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:12.654 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.654 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.655 DEBUG:boto:Method: GET 2023-10-17T18:37:12.655 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:12.655 DEBUG:boto:Data: 2023-10-17T18:37:12.656 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.656 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.657 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.657 DEBUG:boto:Params: {} 2023-10-17T18:37:12.657 DEBUG:boto:Token: None 2023-10-17T18:37:12.658 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3 2023-10-17T18:37:12.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R0HTasmqfXkRVE0oOqQpHUbvM7Q= 2023-10-17T18:37:12.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R0HTasmqfXkRVE0oOqQpHUbvM7Q=', 'Content-Length': '0'} 2023-10-17T18:37:12.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003472cbcde77a0ca4-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.706 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:12.707 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:12.707 DEBUG:boto:Method: GET 2023-10-17T18:37:12.708 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:12.708 DEBUG:boto:Data: 2023-10-17T18:37:12.709 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.709 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.710 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.710 DEBUG:boto:Params: {} 2023-10-17T18:37:12.710 DEBUG:boto:Token: None 2023-10-17T18:37:12.711 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj3 2023-10-17T18:37:12.711 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R0HTasmqfXkRVE0oOqQpHUbvM7Q= 2023-10-17T18:37:12.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R0HTasmqfXkRVE0oOqQpHUbvM7Q=', 'Content-Length': '0'} 2023-10-17T18:37:12.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bab9bbbf04c5ceee-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.715 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.715 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.716 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.716 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.716 DEBUG:boto:Method: GET 2023-10-17T18:37:12.717 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.717 DEBUG:boto:Data: 2023-10-17T18:37:12.718 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.718 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.718 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.719 DEBUG:boto:Params: {} 2023-10-17T18:37:12.719 DEBUG:boto:Token: None 2023-10-17T18:37:12.720 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05f852e1cc66f85-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.723 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.724 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.725 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.725 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.725 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.726 DEBUG:boto:Method: GET 2023-10-17T18:37:12.726 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:12.727 DEBUG:boto:Data: 2023-10-17T18:37:12.727 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.727 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.728 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.728 DEBUG:boto:Params: {} 2023-10-17T18:37:12.729 DEBUG:boto:Token: None 2023-10-17T18:37:12.729 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.730 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000365a4834e90cb95e-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.733 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:12.734 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.735 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.735 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.735 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.736 DEBUG:boto:Method: GET 2023-10-17T18:37:12.736 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.737 DEBUG:boto:Data: 2023-10-17T18:37:12.737 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.738 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.738 DEBUG:boto:Params: {} 2023-10-17T18:37:12.739 DEBUG:boto:Token: None 2023-10-17T18:37:12.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA= 2023-10-17T18:37:12.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA=', 'Content-Length': '0'} 2023-10-17T18:37:12.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3b68993bb2dfad8-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.743 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.743 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.744 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.744 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.744 DEBUG:boto:Method: GET 2023-10-17T18:37:12.745 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.745 DEBUG:boto:Data: 2023-10-17T18:37:12.746 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.747 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.747 DEBUG:boto:Params: {} 2023-10-17T18:37:12.747 DEBUG:boto:Token: None 2023-10-17T18:37:12.748 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA= 2023-10-17T18:37:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA=', 'Content-Length': '0'} 2023-10-17T18:37:12.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f51102b18b4f6951-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.751 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.752 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.752 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.753 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.753 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.753 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.754 DEBUG:boto:Data: 2023-10-17T18:37:12.754 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.755 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.755 DEBUG:boto:Params: {} 2023-10-17T18:37:12.756 DEBUG:boto:Token: None 2023-10-17T18:37:12.756 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UlD0dcX+4ItdBHwZrwAZp+KtQRo= 2023-10-17T18:37:12.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UlD0dcX+4ItdBHwZrwAZp+KtQRo=', 'Content-Length': '0'} 2023-10-17T18:37:12.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b00340e18b26e76a-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.760 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.760 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.761 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.761 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.761 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.762 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.762 DEBUG:boto:Data: 2023-10-17T18:37:12.762 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.763 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.764 DEBUG:boto:Params: {} 2023-10-17T18:37:12.764 DEBUG:boto:Token: None 2023-10-17T18:37:12.764 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UlD0dcX+4ItdBHwZrwAZp+KtQRo= 2023-10-17T18:37:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UlD0dcX+4ItdBHwZrwAZp+KtQRo=', 'Content-Length': '0'} 2023-10-17T18:37:12.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006de7cb5a6211d0a3-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:12.768 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.769 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.769 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.769 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.770 DEBUG:boto:Method: GET 2023-10-17T18:37:12.770 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.771 DEBUG:boto:Data: 2023-10-17T18:37:12.771 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.772 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.772 DEBUG:boto:Params: {} 2023-10-17T18:37:12.773 DEBUG:boto:Token: None 2023-10-17T18:37:12.773 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA= 2023-10-17T18:37:12.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA=', 'Content-Length': '0'} 2023-10-17T18:37:12.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d4dced5295517dd-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.818 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.819 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.819 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.820 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.820 DEBUG:boto:Method: GET 2023-10-17T18:37:12.820 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.821 DEBUG:boto:Data: 2023-10-17T18:37:12.821 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.822 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.823 DEBUG:boto:Params: {} 2023-10-17T18:37:12.823 DEBUG:boto:Token: None 2023-10-17T18:37:12.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:12.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA= 2023-10-17T18:37:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j++T0vw7ukBM526p60c3KMQyAEA=', 'Content-Length': '0'} 2023-10-17T18:37:12.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b701dfc6dc19057-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.827 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.828 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.828 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.829 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.829 DEBUG:boto:Method: GET 2023-10-17T18:37:12.830 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.830 DEBUG:boto:Data: 2023-10-17T18:37:12.830 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.831 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.832 DEBUG:boto:Params: {} 2023-10-17T18:37:12.832 DEBUG:boto:Token: None 2023-10-17T18:37:12.833 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078679f35d46e6b5b-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.837 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.837 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.838 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.838 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.839 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.839 DEBUG:boto:Method: GET 2023-10-17T18:37:12.840 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.840 DEBUG:boto:Data: 2023-10-17T18:37:12.840 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.841 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.842 DEBUG:boto:Params: {} 2023-10-17T18:37:12.842 DEBUG:boto:Token: None 2023-10-17T18:37:12.843 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d32c69aac725c1-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.847 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.847 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.848 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.848 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.848 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:12.849 DEBUG:boto:Data: 2023-10-17T18:37:12.849 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.850 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.850 DEBUG:boto:Params: {} 2023-10-17T18:37:12.851 DEBUG:boto:Token: None 2023-10-17T18:37:12.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4 2023-10-17T18:37:12.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2piY0za/asnwGywxTDP80hJYagI= 2023-10-17T18:37:12.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2piY0za/asnwGywxTDP80hJYagI=', 'Content-Length': '0'} 2023-10-17T18:37:12.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000982b8082bf58967f-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.855 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.855 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.856 DEBUG:boto:Method: HEAD 2023-10-17T18:37:12.856 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:12.856 DEBUG:boto:Data: 2023-10-17T18:37:12.857 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.858 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.858 DEBUG:boto:Params: {} 2023-10-17T18:37:12.858 DEBUG:boto:Token: None 2023-10-17T18:37:12.859 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4 2023-10-17T18:37:12.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2piY0za/asnwGywxTDP80hJYagI= 2023-10-17T18:37:12.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2piY0za/asnwGywxTDP80hJYagI=', 'Content-Length': '0'} 2023-10-17T18:37:12.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000089af88ac9fb0202-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:12.863 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.863 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.863 DEBUG:boto:Method: GET 2023-10-17T18:37:12.864 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:12.864 DEBUG:boto:Data: 2023-10-17T18:37:12.865 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.865 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.866 DEBUG:boto:Params: {} 2023-10-17T18:37:12.866 DEBUG:boto:Token: None 2023-10-17T18:37:12.867 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4 2023-10-17T18:37:12.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fo0BUmCucuemP8obHt3cK9twAQ= 2023-10-17T18:37:12.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fo0BUmCucuemP8obHt3cK9twAQ=', 'Content-Length': '0'} 2023-10-17T18:37:12.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caeca07d3e89bbc4-00652ed458-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.870 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:12.871 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:12.871 DEBUG:boto:Method: GET 2023-10-17T18:37:12.872 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:12.872 DEBUG:boto:Data: 2023-10-17T18:37:12.873 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.873 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.873 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.874 DEBUG:boto:Params: {} 2023-10-17T18:37:12.874 DEBUG:boto:Token: None 2023-10-17T18:37:12.875 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/obj4 2023-10-17T18:37:12.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fo0BUmCucuemP8obHt3cK9twAQ= 2023-10-17T18:37:12.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fo0BUmCucuemP8obHt3cK9twAQ=', 'Content-Length': '0'} 2023-10-17T18:37:12.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f404f400124e546f-00652ed458-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.878 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.879 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.879 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.880 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.880 DEBUG:boto:Method: GET 2023-10-17T18:37:12.881 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.881 DEBUG:boto:Data: 2023-10-17T18:37:12.881 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:12.882 DEBUG:boto:Port: 8001 2023-10-17T18:37:12.883 DEBUG:boto:Params: {} 2023-10-17T18:37:12.883 DEBUG:boto:Token: None 2023-10-17T18:37:12.883 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcabf5bf087894ed-00652ed458-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.887 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.888 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:12.888 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:12.889 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.889 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.890 DEBUG:boto:Method: GET 2023-10-17T18:37:12.890 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:12.890 DEBUG:boto:Data: 2023-10-17T18:37:12.891 DEBUG:boto:Headers: {} 2023-10-17T18:37:12.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:12.892 DEBUG:boto:Port: 8000 2023-10-17T18:37:12.892 DEBUG:boto:Params: {} 2023-10-17T18:37:12.892 DEBUG:boto:Token: None 2023-10-17T18:37:12.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:12 GMT /hbmuog-46/ 2023-10-17T18:37:12.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk= 2023-10-17T18:37:12.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WTnyhKwUufA7YIkXhCHQ5n7otIk=', 'Content-Length': '0'} 2023-10-17T18:37:12.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff3089af9ab3398-00652ed458-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:12.897 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:12.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-46 zones={a2, a1} 2023-10-17T18:37:12.899 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-47 2023-10-17T18:37:12.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:12.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:12.900 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:12.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:12.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:13.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:13.005+0000 7f260d760780 1 bucket sync caught up with source: 2023-10-17T18:37:13.007 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.37.6, , 00000000001.20.6, , 00000000001.19.6, , , 00000000001.20.6, , , ] 2023-10-17T18:37:13.007 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.37.6, , 00000000001.20.6, , 00000000001.19.6, , , 00000000001.20.6, , , ] 2023-10-17T18:37:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:13.005+0000 7f260d760780 0 bucket checkpoint complete 2023-10-17T18:37:13.020 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-47 2023-10-17T18:37:13.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:13.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:13.022 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:13.082 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:13.082 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:13.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:13.115+0000 7fb9229c6780 1 bucket sync caught up with source: 2023-10-17T18:37:13.117 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.126.6, , 00000000001.253.6, , 00000000001.142.6, , , 00000000001.152.6, , , ] 2023-10-17T18:37:13.117 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.126.6, , 00000000001.253.6, , 00000000001.142.6, , , 00000000001.152.6, , , ] 2023-10-17T18:37:13.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:13.115+0000 7fb9229c6780 0 bucket checkpoint complete 2023-10-17T18:37:13.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-47 zones={a2, a1} 2023-10-17T18:37:13.131 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.132 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.133 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.133 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:13.134 DEBUG:boto:Data: 2023-10-17T18:37:13.135 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.136 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.136 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.137 DEBUG:boto:Params: {} 2023-10-17T18:37:13.138 DEBUG:boto:Token: None 2023-10-17T18:37:13.138 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isbq+YuQ8BJjSUA7R0pVHp+wTB8= 2023-10-17T18:37:13.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isbq+YuQ8BJjSUA7R0pVHp+wTB8=', 'Content-Length': '0'} 2023-10-17T18:37:13.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb4c710041eeb8f4-00652ed459-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.142 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.143 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.144 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.144 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:13.145 DEBUG:boto:Data: 2023-10-17T18:37:13.145 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.146 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.147 DEBUG:boto:Params: {} 2023-10-17T18:37:13.147 DEBUG:boto:Token: None 2023-10-17T18:37:13.148 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isbq+YuQ8BJjSUA7R0pVHp+wTB8= 2023-10-17T18:37:13.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isbq+YuQ8BJjSUA7R0pVHp+wTB8=', 'Content-Length': '0'} 2023-10-17T18:37:13.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0b0d5cf108a83db-00652ed459-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:37:13.152 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.152 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.153 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.154 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.154 DEBUG:boto:Method: GET 2023-10-17T18:37:13.155 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.155 DEBUG:boto:Data: 2023-10-17T18:37:13.155 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.156 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.157 DEBUG:boto:Params: {} 2023-10-17T18:37:13.157 DEBUG:boto:Token: None 2023-10-17T18:37:13.158 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/?versions 2023-10-17T18:37:13.158 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k= 2023-10-17T18:37:13.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k=', 'Content-Length': '0'} 2023-10-17T18:37:13.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b17ae6bf3217c7-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.161 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:13.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:13.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:13.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:13.164 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:37:13.164 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.165 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.165 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.165 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.166 DEBUG:boto:Method: GET 2023-10-17T18:37:13.166 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.167 DEBUG:boto:Data: 2023-10-17T18:37:13.167 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.167 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.168 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.168 DEBUG:boto:Params: {} 2023-10-17T18:37:13.169 DEBUG:boto:Token: None 2023-10-17T18:37:13.169 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/?versions 2023-10-17T18:37:13.170 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k= 2023-10-17T18:37:13.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k=', 'Content-Length': '0'} 2023-10-17T18:37:13.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ab9131b8e4954c-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.173 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:13.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:13.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:13.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:13.175 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.176 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.176 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.176 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.177 DEBUG:boto:Method: GET 2023-10-17T18:37:13.177 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.178 DEBUG:boto:Data: 2023-10-17T18:37:13.178 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.179 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.179 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.179 DEBUG:boto:Params: {} 2023-10-17T18:37:13.180 DEBUG:boto:Token: None 2023-10-17T18:37:13.180 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/?versions 2023-10-17T18:37:13.181 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k= 2023-10-17T18:37:13.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k=', 'Content-Length': '0'} 2023-10-17T18:37:13.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4328ec555aec98e-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.184 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.185 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.185 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.186 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.186 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.187 DEBUG:boto:Method: GET 2023-10-17T18:37:13.187 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:13.187 DEBUG:boto:Data: 2023-10-17T18:37:13.188 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.189 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.189 DEBUG:boto:Params: {} 2023-10-17T18:37:13.190 DEBUG:boto:Token: None 2023-10-17T18:37:13.190 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/?versions 2023-10-17T18:37:13.191 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k= 2023-10-17T18:37:13.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0zUqzAa/MMNXUX/m+frg1x9/6k=', 'Content-Length': '0'} 2023-10-17T18:37:13.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008395c2022b936d17-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.194 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:13.195 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.196 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.196 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.197 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.197 DEBUG:boto:Method: GET 2023-10-17T18:37:13.197 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.198 DEBUG:boto:Data: 2023-10-17T18:37:13.198 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.199 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.200 DEBUG:boto:Params: {} 2023-10-17T18:37:13.200 DEBUG:boto:Token: None 2023-10-17T18:37:13.201 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY= 2023-10-17T18:37:13.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY=', 'Content-Length': '0'} 2023-10-17T18:37:13.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083750626e1fbcd02-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.204 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.204 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.205 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.205 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.206 DEBUG:boto:Method: GET 2023-10-17T18:37:13.206 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.207 DEBUG:boto:Data: 2023-10-17T18:37:13.207 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.208 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.209 DEBUG:boto:Params: {} 2023-10-17T18:37:13.209 DEBUG:boto:Token: None 2023-10-17T18:37:13.210 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY= 2023-10-17T18:37:13.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY=', 'Content-Length': '0'} 2023-10-17T18:37:13.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe5cde280c8ac841-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.213 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.214 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.215 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.215 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.216 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.216 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.217 DEBUG:boto:Data: 2023-10-17T18:37:13.217 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.218 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.218 DEBUG:boto:Params: {} 2023-10-17T18:37:13.219 DEBUG:boto:Token: None 2023-10-17T18:37:13.219 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uP8VvO6Y1liZRwyzE0/8CwMFrRU= 2023-10-17T18:37:13.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uP8VvO6Y1liZRwyzE0/8CwMFrRU=', 'Content-Length': '0'} 2023-10-17T18:37:13.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2391f1bdf716828-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.222 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.223 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.223 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.224 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.224 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.225 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.225 DEBUG:boto:Data: 2023-10-17T18:37:13.225 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.226 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.227 DEBUG:boto:Params: {} 2023-10-17T18:37:13.227 DEBUG:boto:Token: None 2023-10-17T18:37:13.228 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uP8VvO6Y1liZRwyzE0/8CwMFrRU= 2023-10-17T18:37:13.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uP8VvO6Y1liZRwyzE0/8CwMFrRU=', 'Content-Length': '0'} 2023-10-17T18:37:13.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007627498d42006cd7-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:13.231 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.232 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.232 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.233 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.233 DEBUG:boto:Method: GET 2023-10-17T18:37:13.233 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.234 DEBUG:boto:Data: 2023-10-17T18:37:13.234 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.235 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.236 DEBUG:boto:Params: {} 2023-10-17T18:37:13.236 DEBUG:boto:Token: None 2023-10-17T18:37:13.237 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY= 2023-10-17T18:37:13.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY=', 'Content-Length': '0'} 2023-10-17T18:37:13.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009336acf30850b9c9-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.240 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.241 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.241 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.241 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.242 DEBUG:boto:Method: GET 2023-10-17T18:37:13.242 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.243 DEBUG:boto:Data: 2023-10-17T18:37:13.243 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.244 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.244 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.244 DEBUG:boto:Params: {} 2023-10-17T18:37:13.245 DEBUG:boto:Token: None 2023-10-17T18:37:13.245 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:13.246 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY= 2023-10-17T18:37:13.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IyePm/6g90Wwk7dhI6fGxsX2hwY=', 'Content-Length': '0'} 2023-10-17T18:37:13.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006591c842759f9b26-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.290 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.291 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.291 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.292 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.292 DEBUG:boto:Method: GET 2023-10-17T18:37:13.293 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.293 DEBUG:boto:Data: 2023-10-17T18:37:13.294 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.294 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.295 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.295 DEBUG:boto:Params: {} 2023-10-17T18:37:13.296 DEBUG:boto:Token: None 2023-10-17T18:37:13.296 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208a2d681d2dfcbc-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.300 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.301 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.301 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.302 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.302 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.303 DEBUG:boto:Method: GET 2023-10-17T18:37:13.303 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.304 DEBUG:boto:Data: 2023-10-17T18:37:13.304 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.304 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.305 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.305 DEBUG:boto:Params: {} 2023-10-17T18:37:13.306 DEBUG:boto:Token: None 2023-10-17T18:37:13.306 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.307 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051857d455fb220ce-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.310 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.311 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.311 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.312 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.312 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:13.312 DEBUG:boto:Data: 2023-10-17T18:37:13.313 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.314 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.314 DEBUG:boto:Params: {} 2023-10-17T18:37:13.314 DEBUG:boto:Token: None 2023-10-17T18:37:13.315 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1 2023-10-17T18:37:13.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:axdrvNfEfP76ldMgSHRCM7Hf0Q4= 2023-10-17T18:37:13.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:axdrvNfEfP76ldMgSHRCM7Hf0Q4=', 'Content-Length': '0'} 2023-10-17T18:37:13.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bec61c9f85ecf720-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.318 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.319 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.319 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.320 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:13.320 DEBUG:boto:Data: 2023-10-17T18:37:13.320 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.321 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.321 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.322 DEBUG:boto:Params: {} 2023-10-17T18:37:13.322 DEBUG:boto:Token: None 2023-10-17T18:37:13.322 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1 2023-10-17T18:37:13.323 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:axdrvNfEfP76ldMgSHRCM7Hf0Q4= 2023-10-17T18:37:13.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:axdrvNfEfP76ldMgSHRCM7Hf0Q4=', 'Content-Length': '0'} 2023-10-17T18:37:13.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e21a70f0e63008-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:13.329 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.329 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.330 DEBUG:boto:Method: GET 2023-10-17T18:37:13.330 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:13.330 DEBUG:boto:Data: 2023-10-17T18:37:13.331 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.331 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.332 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.332 DEBUG:boto:Params: {} 2023-10-17T18:37:13.333 DEBUG:boto:Token: None 2023-10-17T18:37:13.333 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1 2023-10-17T18:37:13.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QPN97Ux7QzAAQ1MlFNHkFVskHEE= 2023-10-17T18:37:13.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QPN97Ux7QzAAQ1MlFNHkFVskHEE=', 'Content-Length': '0'} 2023-10-17T18:37:13.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022f650d3268ffd26-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.337 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:13.337 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:13.337 DEBUG:boto:Method: GET 2023-10-17T18:37:13.338 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:13.338 DEBUG:boto:Data: 2023-10-17T18:37:13.339 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.340 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.340 DEBUG:boto:Params: {} 2023-10-17T18:37:13.341 DEBUG:boto:Token: None 2023-10-17T18:37:13.341 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj1 2023-10-17T18:37:13.342 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QPN97Ux7QzAAQ1MlFNHkFVskHEE= 2023-10-17T18:37:13.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QPN97Ux7QzAAQ1MlFNHkFVskHEE=', 'Content-Length': '0'} 2023-10-17T18:37:13.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009afafa5d3aaca20a-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.344 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.345 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.345 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.346 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.346 DEBUG:boto:Method: GET 2023-10-17T18:37:13.347 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.347 DEBUG:boto:Data: 2023-10-17T18:37:13.348 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.349 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.349 DEBUG:boto:Params: {} 2023-10-17T18:37:13.349 DEBUG:boto:Token: None 2023-10-17T18:37:13.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a7a4caadccaa99-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.354 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.355 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.355 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.355 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.356 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.356 DEBUG:boto:Method: GET 2023-10-17T18:37:13.357 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:13.357 DEBUG:boto:Data: 2023-10-17T18:37:13.358 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.358 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.358 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.359 DEBUG:boto:Params: {} 2023-10-17T18:37:13.359 DEBUG:boto:Token: None 2023-10-17T18:37:13.360 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.360 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004174b4702412c21c-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.364 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:13.365 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.365 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.366 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.366 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.367 DEBUG:boto:Method: GET 2023-10-17T18:37:13.367 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.368 DEBUG:boto:Data: 2023-10-17T18:37:13.368 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.368 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.369 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.369 DEBUG:boto:Params: {} 2023-10-17T18:37:13.370 DEBUG:boto:Token: None 2023-10-17T18:37:13.370 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.371 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew= 2023-10-17T18:37:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew=', 'Content-Length': '0'} 2023-10-17T18:37:13.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000982d4aeeaa725db9-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.374 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.374 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.375 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.375 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.375 DEBUG:boto:Method: GET 2023-10-17T18:37:13.376 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.376 DEBUG:boto:Data: 2023-10-17T18:37:13.377 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.377 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.378 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.378 DEBUG:boto:Params: {} 2023-10-17T18:37:13.379 DEBUG:boto:Token: None 2023-10-17T18:37:13.379 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew= 2023-10-17T18:37:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew=', 'Content-Length': '0'} 2023-10-17T18:37:13.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aaea0fef2de59e9-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.422 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.423 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.423 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.424 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.424 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.424 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.425 DEBUG:boto:Data: 2023-10-17T18:37:13.425 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.426 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.426 DEBUG:boto:Params: {} 2023-10-17T18:37:13.427 DEBUG:boto:Token: None 2023-10-17T18:37:13.427 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.428 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:njUQWYcIbjnsWFcNbBhpNrED3pw= 2023-10-17T18:37:13.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:njUQWYcIbjnsWFcNbBhpNrED3pw=', 'Content-Length': '0'} 2023-10-17T18:37:13.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064fba686f6f6e7d9-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.431 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.431 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.432 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.432 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.433 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.433 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.433 DEBUG:boto:Data: 2023-10-17T18:37:13.434 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.435 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.435 DEBUG:boto:Params: {} 2023-10-17T18:37:13.436 DEBUG:boto:Token: None 2023-10-17T18:37:13.436 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:njUQWYcIbjnsWFcNbBhpNrED3pw= 2023-10-17T18:37:13.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:njUQWYcIbjnsWFcNbBhpNrED3pw=', 'Content-Length': '0'} 2023-10-17T18:37:13.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b19bd4c594ef54f5-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:13.441 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.441 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.441 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.442 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.442 DEBUG:boto:Method: GET 2023-10-17T18:37:13.443 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.443 DEBUG:boto:Data: 2023-10-17T18:37:13.444 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.445 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.445 DEBUG:boto:Params: {} 2023-10-17T18:37:13.445 DEBUG:boto:Token: None 2023-10-17T18:37:13.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew= 2023-10-17T18:37:13.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew=', 'Content-Length': '0'} 2023-10-17T18:37:13.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e7449556f25acc1-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.449 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.450 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.450 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.451 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.451 DEBUG:boto:Method: GET 2023-10-17T18:37:13.452 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.452 DEBUG:boto:Data: 2023-10-17T18:37:13.453 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.454 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.454 DEBUG:boto:Params: {} 2023-10-17T18:37:13.455 DEBUG:boto:Token: None 2023-10-17T18:37:13.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:13.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew= 2023-10-17T18:37:13.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t2YOkZeMvK/XyYDHE+nG/Ydkdew=', 'Content-Length': '0'} 2023-10-17T18:37:13.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d99c51afe5193e1-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.459 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.460 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.461 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.462 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.463 DEBUG:boto:Method: GET 2023-10-17T18:37:13.463 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.464 DEBUG:boto:Data: 2023-10-17T18:37:13.465 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.465 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.466 DEBUG:boto:Params: {} 2023-10-17T18:37:13.467 DEBUG:boto:Token: None 2023-10-17T18:37:13.467 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e482cff27e3cf020-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.471 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.472 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.473 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.473 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.474 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.474 DEBUG:boto:Method: GET 2023-10-17T18:37:13.475 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.475 DEBUG:boto:Data: 2023-10-17T18:37:13.476 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.478 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.479 DEBUG:boto:Params: {} 2023-10-17T18:37:13.479 DEBUG:boto:Token: None 2023-10-17T18:37:13.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d784eee552be07-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.485 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.486 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.487 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.487 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.487 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:13.488 DEBUG:boto:Data: 2023-10-17T18:37:13.488 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.489 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.489 DEBUG:boto:Params: {} 2023-10-17T18:37:13.490 DEBUG:boto:Token: None 2023-10-17T18:37:13.491 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2 2023-10-17T18:37:13.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t8sNC2O+s5wRKLa2OTUJswGVPVU= 2023-10-17T18:37:13.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t8sNC2O+s5wRKLa2OTUJswGVPVU=', 'Content-Length': '0'} 2023-10-17T18:37:13.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee3658ed2b3ac1fc-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.494 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.494 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.495 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.495 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:13.496 DEBUG:boto:Data: 2023-10-17T18:37:13.496 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.496 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.497 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.497 DEBUG:boto:Params: {} 2023-10-17T18:37:13.498 DEBUG:boto:Token: None 2023-10-17T18:37:13.498 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2 2023-10-17T18:37:13.499 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t8sNC2O+s5wRKLa2OTUJswGVPVU= 2023-10-17T18:37:13.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t8sNC2O+s5wRKLa2OTUJswGVPVU=', 'Content-Length': '0'} 2023-10-17T18:37:13.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa59d949d82d8052-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:13.502 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.503 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.503 DEBUG:boto:Method: GET 2023-10-17T18:37:13.504 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:13.504 DEBUG:boto:Data: 2023-10-17T18:37:13.504 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.505 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.505 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.506 DEBUG:boto:Params: {} 2023-10-17T18:37:13.506 DEBUG:boto:Token: None 2023-10-17T18:37:13.507 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2 2023-10-17T18:37:13.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/XVmYMIypLoKDed//fESe903DZU= 2023-10-17T18:37:13.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/XVmYMIypLoKDed//fESe903DZU=', 'Content-Length': '0'} 2023-10-17T18:37:13.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000614b1f8a10dc67ee-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.550 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:13.551 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:13.551 DEBUG:boto:Method: GET 2023-10-17T18:37:13.552 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:13.552 DEBUG:boto:Data: 2023-10-17T18:37:13.553 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.554 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.554 DEBUG:boto:Params: {} 2023-10-17T18:37:13.555 DEBUG:boto:Token: None 2023-10-17T18:37:13.555 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj2 2023-10-17T18:37:13.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/XVmYMIypLoKDed//fESe903DZU= 2023-10-17T18:37:13.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/XVmYMIypLoKDed//fESe903DZU=', 'Content-Length': '0'} 2023-10-17T18:37:13.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e922eac2c23d24fe-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.559 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.559 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.560 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.560 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.561 DEBUG:boto:Method: GET 2023-10-17T18:37:13.561 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.562 DEBUG:boto:Data: 2023-10-17T18:37:13.562 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.562 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.563 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.564 DEBUG:boto:Params: {} 2023-10-17T18:37:13.564 DEBUG:boto:Token: None 2023-10-17T18:37:13.565 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.565 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8374c840c4788f-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.569 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.570 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.570 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.570 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.571 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.571 DEBUG:boto:Method: GET 2023-10-17T18:37:13.572 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:13.572 DEBUG:boto:Data: 2023-10-17T18:37:13.573 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.573 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.574 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.574 DEBUG:boto:Params: {} 2023-10-17T18:37:13.575 DEBUG:boto:Token: None 2023-10-17T18:37:13.575 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.575 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed530fe61b2b5ee-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.622 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:13.624 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.624 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.624 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.625 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.625 DEBUG:boto:Method: GET 2023-10-17T18:37:13.626 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.626 DEBUG:boto:Data: 2023-10-17T18:37:13.627 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.627 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.627 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.628 DEBUG:boto:Params: {} 2023-10-17T18:37:13.628 DEBUG:boto:Token: None 2023-10-17T18:37:13.629 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ= 2023-10-17T18:37:13.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ=', 'Content-Length': '0'} 2023-10-17T18:37:13.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053f7213cbb732f5b-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.632 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.633 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.633 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.634 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.634 DEBUG:boto:Method: GET 2023-10-17T18:37:13.635 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.635 DEBUG:boto:Data: 2023-10-17T18:37:13.635 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.636 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.636 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.637 DEBUG:boto:Params: {} 2023-10-17T18:37:13.637 DEBUG:boto:Token: None 2023-10-17T18:37:13.638 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.638 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ= 2023-10-17T18:37:13.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ=', 'Content-Length': '0'} 2023-10-17T18:37:13.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f977a3585540c33-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.642 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.642 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.643 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.643 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.643 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.644 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.644 DEBUG:boto:Data: 2023-10-17T18:37:13.645 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.645 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.646 DEBUG:boto:Params: {} 2023-10-17T18:37:13.646 DEBUG:boto:Token: None 2023-10-17T18:37:13.647 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FcTZUTOQ6BqXOn/2EzZR+FGjg+s= 2023-10-17T18:37:13.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FcTZUTOQ6BqXOn/2EzZR+FGjg+s=', 'Content-Length': '0'} 2023-10-17T18:37:13.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095741a069ce33e38-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.650 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.651 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.651 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.652 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.652 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.653 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.653 DEBUG:boto:Data: 2023-10-17T18:37:13.653 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.654 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.654 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.655 DEBUG:boto:Params: {} 2023-10-17T18:37:13.655 DEBUG:boto:Token: None 2023-10-17T18:37:13.656 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.656 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FcTZUTOQ6BqXOn/2EzZR+FGjg+s= 2023-10-17T18:37:13.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FcTZUTOQ6BqXOn/2EzZR+FGjg+s=', 'Content-Length': '0'} 2023-10-17T18:37:13.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e5fc6eb8a7c0e17-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:13.659 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.660 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.660 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.661 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.661 DEBUG:boto:Method: GET 2023-10-17T18:37:13.661 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.662 DEBUG:boto:Data: 2023-10-17T18:37:13.662 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.663 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.663 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.664 DEBUG:boto:Params: {} 2023-10-17T18:37:13.664 DEBUG:boto:Token: None 2023-10-17T18:37:13.665 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ= 2023-10-17T18:37:13.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ=', 'Content-Length': '0'} 2023-10-17T18:37:13.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044b8f280018ee616-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.670 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.671 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.671 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.672 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.672 DEBUG:boto:Method: GET 2023-10-17T18:37:13.673 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.673 DEBUG:boto:Data: 2023-10-17T18:37:13.674 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.675 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.675 DEBUG:boto:Params: {} 2023-10-17T18:37:13.675 DEBUG:boto:Token: None 2023-10-17T18:37:13.676 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:13.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ= 2023-10-17T18:37:13.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+daLO+VEVhF9tNQz8AXYPd6OCyQ=', 'Content-Length': '0'} 2023-10-17T18:37:13.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d53464a9f0ab8252-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.680 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.680 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.680 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.681 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.681 DEBUG:boto:Method: GET 2023-10-17T18:37:13.682 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.682 DEBUG:boto:Data: 2023-10-17T18:37:13.683 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.683 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.684 DEBUG:boto:Params: {} 2023-10-17T18:37:13.684 DEBUG:boto:Token: None 2023-10-17T18:37:13.685 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b50fdb0173a9d9d8-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.689 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.690 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.690 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.691 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.691 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.691 DEBUG:boto:Method: GET 2023-10-17T18:37:13.692 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.692 DEBUG:boto:Data: 2023-10-17T18:37:13.693 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.694 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.694 DEBUG:boto:Params: {} 2023-10-17T18:37:13.695 DEBUG:boto:Token: None 2023-10-17T18:37:13.695 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de89e71565204178-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.742 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.744 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.744 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.745 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.745 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:13.746 DEBUG:boto:Data: 2023-10-17T18:37:13.746 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.747 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.747 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.747 DEBUG:boto:Params: {} 2023-10-17T18:37:13.748 DEBUG:boto:Token: None 2023-10-17T18:37:13.748 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3 2023-10-17T18:37:13.749 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KWYceexNbPG+bt6zSP/jnL/OvCU= 2023-10-17T18:37:13.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KWYceexNbPG+bt6zSP/jnL/OvCU=', 'Content-Length': '0'} 2023-10-17T18:37:13.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045cb3c7d2cb4c151-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.752 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.752 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.753 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.753 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:13.753 DEBUG:boto:Data: 2023-10-17T18:37:13.754 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.754 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.755 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.755 DEBUG:boto:Params: {} 2023-10-17T18:37:13.756 DEBUG:boto:Token: None 2023-10-17T18:37:13.756 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3 2023-10-17T18:37:13.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KWYceexNbPG+bt6zSP/jnL/OvCU= 2023-10-17T18:37:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KWYceexNbPG+bt6zSP/jnL/OvCU=', 'Content-Length': '0'} 2023-10-17T18:37:13.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087be8a013db4be11-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:13.760 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.761 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.761 DEBUG:boto:Method: GET 2023-10-17T18:37:13.761 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:13.762 DEBUG:boto:Data: 2023-10-17T18:37:13.762 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.763 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.764 DEBUG:boto:Params: {} 2023-10-17T18:37:13.764 DEBUG:boto:Token: None 2023-10-17T18:37:13.765 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3 2023-10-17T18:37:13.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3xFztCLW8474r4Tl1lPNWrbO2lA= 2023-10-17T18:37:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3xFztCLW8474r4Tl1lPNWrbO2lA=', 'Content-Length': '0'} 2023-10-17T18:37:13.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067d67f93474acc3e-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.768 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:13.769 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:13.769 DEBUG:boto:Method: GET 2023-10-17T18:37:13.769 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:13.770 DEBUG:boto:Data: 2023-10-17T18:37:13.770 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.771 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.772 DEBUG:boto:Params: {} 2023-10-17T18:37:13.772 DEBUG:boto:Token: None 2023-10-17T18:37:13.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj3 2023-10-17T18:37:13.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3xFztCLW8474r4Tl1lPNWrbO2lA= 2023-10-17T18:37:13.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3xFztCLW8474r4Tl1lPNWrbO2lA=', 'Content-Length': '0'} 2023-10-17T18:37:13.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000266a0610dbc0dcf7-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.776 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.776 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.777 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.777 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.778 DEBUG:boto:Method: GET 2023-10-17T18:37:13.778 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.779 DEBUG:boto:Data: 2023-10-17T18:37:13.779 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.780 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.780 DEBUG:boto:Params: {} 2023-10-17T18:37:13.781 DEBUG:boto:Token: None 2023-10-17T18:37:13.781 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.782 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b707bc0cdd79cbb0-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.785 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.786 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.786 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.787 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.787 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.788 DEBUG:boto:Method: GET 2023-10-17T18:37:13.788 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:13.789 DEBUG:boto:Data: 2023-10-17T18:37:13.789 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.789 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.790 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.790 DEBUG:boto:Params: {} 2023-10-17T18:37:13.791 DEBUG:boto:Token: None 2023-10-17T18:37:13.791 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fef9b6b37cd0d1c-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.795 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:13.796 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.797 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.797 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.798 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.798 DEBUG:boto:Method: GET 2023-10-17T18:37:13.799 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.799 DEBUG:boto:Data: 2023-10-17T18:37:13.799 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.800 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.801 DEBUG:boto:Params: {} 2023-10-17T18:37:13.801 DEBUG:boto:Token: None 2023-10-17T18:37:13.802 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g= 2023-10-17T18:37:13.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g=', 'Content-Length': '0'} 2023-10-17T18:37:13.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079e448c98d3dec04-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.805 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.806 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.806 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.807 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.807 DEBUG:boto:Method: GET 2023-10-17T18:37:13.808 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.808 DEBUG:boto:Data: 2023-10-17T18:37:13.809 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.809 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.810 DEBUG:boto:Params: {} 2023-10-17T18:37:13.810 DEBUG:boto:Token: None 2023-10-17T18:37:13.811 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g= 2023-10-17T18:37:13.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g=', 'Content-Length': '0'} 2023-10-17T18:37:13.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac23b19f063872b8-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.854 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.855 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.855 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.856 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.856 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.857 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.857 DEBUG:boto:Data: 2023-10-17T18:37:13.857 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.858 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.859 DEBUG:boto:Params: {} 2023-10-17T18:37:13.859 DEBUG:boto:Token: None 2023-10-17T18:37:13.860 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EpBX/zP0PZq9Sz2uPngGsgHrud8= 2023-10-17T18:37:13.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EpBX/zP0PZq9Sz2uPngGsgHrud8=', 'Content-Length': '0'} 2023-10-17T18:37:13.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf73b8452ec1f839-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.863 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.864 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.864 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.864 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.865 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.865 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.866 DEBUG:boto:Data: 2023-10-17T18:37:13.866 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.866 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.867 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.867 DEBUG:boto:Params: {} 2023-10-17T18:37:13.868 DEBUG:boto:Token: None 2023-10-17T18:37:13.868 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EpBX/zP0PZq9Sz2uPngGsgHrud8= 2023-10-17T18:37:13.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EpBX/zP0PZq9Sz2uPngGsgHrud8=', 'Content-Length': '0'} 2023-10-17T18:37:13.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019d22f5704fb9048-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:13.872 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.872 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.873 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.873 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.874 DEBUG:boto:Method: GET 2023-10-17T18:37:13.874 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.875 DEBUG:boto:Data: 2023-10-17T18:37:13.875 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.876 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.876 DEBUG:boto:Params: {} 2023-10-17T18:37:13.877 DEBUG:boto:Token: None 2023-10-17T18:37:13.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g= 2023-10-17T18:37:13.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g=', 'Content-Length': '0'} 2023-10-17T18:37:13.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075ef31c97da041f5-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.927 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.927 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.928 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.928 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.929 DEBUG:boto:Method: GET 2023-10-17T18:37:13.929 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.930 DEBUG:boto:Data: 2023-10-17T18:37:13.931 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.931 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.932 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.932 DEBUG:boto:Params: {} 2023-10-17T18:37:13.933 DEBUG:boto:Token: None 2023-10-17T18:37:13.933 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:13.934 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g= 2023-10-17T18:37:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OOQ8WsHB0jrgX7rlBefnFAJJM8g=', 'Content-Length': '0'} 2023-10-17T18:37:13.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afa63443d2cca904-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.937 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.938 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.938 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.942 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.942 DEBUG:boto:Method: GET 2023-10-17T18:37:13.943 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.943 DEBUG:boto:Data: 2023-10-17T18:37:13.943 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.944 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.944 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.945 DEBUG:boto:Params: {} 2023-10-17T18:37:13.946 DEBUG:boto:Token: None 2023-10-17T18:37:13.946 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.947 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000630b3ec1e25ed644-00652ed459-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.951 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.952 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.952 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.953 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.953 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.954 DEBUG:boto:Method: GET 2023-10-17T18:37:13.954 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.955 DEBUG:boto:Data: 2023-10-17T18:37:13.955 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.956 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.956 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.956 DEBUG:boto:Params: {} 2023-10-17T18:37:13.957 DEBUG:boto:Token: None 2023-10-17T18:37:13.958 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/ 2023-10-17T18:37:13.958 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8= 2023-10-17T18:37:13.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BYpFn6bauzrNCdvU6BBt3SbVUz8=', 'Content-Length': '0'} 2023-10-17T18:37:13.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af4336560e47b32-00652ed459-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.962 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:13.963 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.963 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.963 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.964 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:13.964 DEBUG:boto:Data: 2023-10-17T18:37:13.965 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.966 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.966 DEBUG:boto:Params: {} 2023-10-17T18:37:13.967 DEBUG:boto:Token: None 2023-10-17T18:37:13.967 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4 2023-10-17T18:37:13.968 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IkHxRQ5zCs8mQs/U8Vg4uRKEHFM= 2023-10-17T18:37:13.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IkHxRQ5zCs8mQs/U8Vg4uRKEHFM=', 'Content-Length': '0'} 2023-10-17T18:37:13.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de6ff7ee18402b18-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.970 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.971 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.971 DEBUG:boto:Method: HEAD 2023-10-17T18:37:13.972 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:13.972 DEBUG:boto:Data: 2023-10-17T18:37:13.973 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.973 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.974 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.974 DEBUG:boto:Params: {} 2023-10-17T18:37:13.975 DEBUG:boto:Token: None 2023-10-17T18:37:13.976 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4 2023-10-17T18:37:13.976 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IkHxRQ5zCs8mQs/U8Vg4uRKEHFM= 2023-10-17T18:37:13.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IkHxRQ5zCs8mQs/U8Vg4uRKEHFM=', 'Content-Length': '0'} 2023-10-17T18:37:13.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055f885a0863920ac-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:13.979 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.980 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.981 DEBUG:boto:Method: GET 2023-10-17T18:37:13.981 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:13.981 DEBUG:boto:Data: 2023-10-17T18:37:13.982 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:13.983 DEBUG:boto:Port: 8001 2023-10-17T18:37:13.983 DEBUG:boto:Params: {} 2023-10-17T18:37:13.984 DEBUG:boto:Token: None 2023-10-17T18:37:13.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4 2023-10-17T18:37:13.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lAAFXlzikQgXuhUw2G2RGeZqQlg= 2023-10-17T18:37:13.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lAAFXlzikQgXuhUw2G2RGeZqQlg=', 'Content-Length': '0'} 2023-10-17T18:37:13.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c66684a74f872944-00652ed459-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.988 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:13.989 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:13.989 DEBUG:boto:Method: GET 2023-10-17T18:37:13.989 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:13.990 DEBUG:boto:Data: 2023-10-17T18:37:13.990 DEBUG:boto:Headers: {} 2023-10-17T18:37:13.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:13.991 DEBUG:boto:Port: 8000 2023-10-17T18:37:13.992 DEBUG:boto:Params: {} 2023-10-17T18:37:13.992 DEBUG:boto:Token: None 2023-10-17T18:37:13.993 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:13 GMT /hbmuog-47/obj4 2023-10-17T18:37:13.993 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lAAFXlzikQgXuhUw2G2RGeZqQlg= 2023-10-17T18:37:13.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lAAFXlzikQgXuhUw2G2RGeZqQlg=', 'Content-Length': '0'} 2023-10-17T18:37:13.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000331b03c898bf692c-00652ed459-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:13.996 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:13.997 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:13.997 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.998 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.998 DEBUG:boto:Method: GET 2023-10-17T18:37:13.999 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:13.999 DEBUG:boto:Data: 2023-10-17T18:37:14.000 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.000 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.000 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.001 DEBUG:boto:Params: {} 2023-10-17T18:37:14.001 DEBUG:boto:Token: None 2023-10-17T18:37:14.002 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.002 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G1bKeUcldPHyGJHWcODvNS6kH4A= 2023-10-17T18:37:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G1bKeUcldPHyGJHWcODvNS6kH4A=', 'Content-Length': '0'} 2023-10-17T18:37:14.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eff09035755b68b-00652ed45a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.006 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:14.006 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:14.007 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:14.007 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:14.008 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:14.008 DEBUG:boto:Method: GET 2023-10-17T18:37:14.008 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:14.009 DEBUG:boto:Data: 2023-10-17T18:37:14.009 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.010 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.010 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.011 DEBUG:boto:Params: {} 2023-10-17T18:37:14.011 DEBUG:boto:Token: None 2023-10-17T18:37:14.012 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.012 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G1bKeUcldPHyGJHWcODvNS6kH4A= 2023-10-17T18:37:14.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G1bKeUcldPHyGJHWcODvNS6kH4A=', 'Content-Length': '0'} 2023-10-17T18:37:14.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000469ac9899de0ed43-00652ed45a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.016 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:14.016 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-47 zones={a2, a1} 2023-10-17T18:37:14.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.018 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:37:14.073 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:14.073 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:14.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hbmuog-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hbmuog-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.145 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hbmuog-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:37:14.196 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:14.197 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:14.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.264 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:37:14.317 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:14.318 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:14.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hbmuog-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hbmuog-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:37:14.385 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hbmuog-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:37:14.437 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:14.437 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:14.503 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:14.503 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:14.504 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.504 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:14.505 DEBUG:boto:Data: 2023-10-17T18:37:14.505 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.505 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.506 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.506 DEBUG:boto:Params: {} 2023-10-17T18:37:14.507 DEBUG:boto:Token: None 2023-10-17T18:37:14.508 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/ 2023-10-17T18:37:14.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA= 2023-10-17T18:37:14.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA=', 'Content-Length': '0'} 2023-10-17T18:37:14.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012dd7f90cc4e67ea-00652ed45a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.512 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:14.512 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:14.512 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.513 DEBUG:boto:Path: /hbmuog-46/obj5 2023-10-17T18:37:14.513 DEBUG:boto:Data: 2023-10-17T18:37:14.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.515 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.515 DEBUG:boto:Params: {} 2023-10-17T18:37:14.516 DEBUG:boto:Token: None 2023-10-17T18:37:14.516 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/obj5 2023-10-17T18:37:14.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WDxFDQ9HB3IHaQbeYRh0KJ44X/g= 2023-10-17T18:37:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WDxFDQ9HB3IHaQbeYRh0KJ44X/g='} 2023-10-17T18:37:14.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000561d5620e78f2777-00652ed45a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.567 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:14.568 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:14.568 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.568 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:14.569 DEBUG:boto:Data: 2023-10-17T18:37:14.569 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.570 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.570 DEBUG:boto:Params: {} 2023-10-17T18:37:14.571 DEBUG:boto:Token: None 2023-10-17T18:37:14.571 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/ 2023-10-17T18:37:14.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA= 2023-10-17T18:37:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA=', 'Content-Length': '0'} 2023-10-17T18:37:14.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b01f5af4f4c3dc5-00652ed45a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.575 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:14.576 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:14.576 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.577 DEBUG:boto:Path: /hbmuog-46/obj6 2023-10-17T18:37:14.577 DEBUG:boto:Data: 2023-10-17T18:37:14.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.578 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.578 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.579 DEBUG:boto:Params: {} 2023-10-17T18:37:14.579 DEBUG:boto:Token: None 2023-10-17T18:37:14.580 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/obj6 2023-10-17T18:37:14.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V50XImB4UbLI2nC3h+hJuwFTBgk= 2023-10-17T18:37:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V50XImB4UbLI2nC3h+hJuwFTBgk='} 2023-10-17T18:37:14.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdc26110f1df2575-00652ed45a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.586 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:14.586 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:14.587 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.587 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:14.587 DEBUG:boto:Data: 2023-10-17T18:37:14.588 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.589 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.589 DEBUG:boto:Params: {} 2023-10-17T18:37:14.589 DEBUG:boto:Token: None 2023-10-17T18:37:14.590 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/ 2023-10-17T18:37:14.590 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA= 2023-10-17T18:37:14.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA=', 'Content-Length': '0'} 2023-10-17T18:37:14.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006989e98a94611608-00652ed45a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.594 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:14.594 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:14.594 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.595 DEBUG:boto:Path: /hbmuog-46/obj7 2023-10-17T18:37:14.595 DEBUG:boto:Data: 2023-10-17T18:37:14.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.596 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.597 DEBUG:boto:Params: {} 2023-10-17T18:37:14.597 DEBUG:boto:Token: None 2023-10-17T18:37:14.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/obj7 2023-10-17T18:37:14.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n+8f92lpsNfYo8pd/jniL8Jq9nk= 2023-10-17T18:37:14.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n+8f92lpsNfYo8pd/jniL8Jq9nk='} 2023-10-17T18:37:14.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b42a2b9da898b8f-00652ed45a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.643 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:14.644 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:14.644 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.645 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:14.645 DEBUG:boto:Data: 2023-10-17T18:37:14.645 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.646 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.646 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.647 DEBUG:boto:Params: {} 2023-10-17T18:37:14.647 DEBUG:boto:Token: None 2023-10-17T18:37:14.648 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/ 2023-10-17T18:37:14.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA= 2023-10-17T18:37:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+U9GJmmJYP9167zsl+6LcuMouKA=', 'Content-Length': '0'} 2023-10-17T18:37:14.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036f957227baa9328-00652ed45a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.652 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:14.652 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:14.653 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.653 DEBUG:boto:Path: /hbmuog-46/obj8 2023-10-17T18:37:14.654 DEBUG:boto:Data: 2023-10-17T18:37:14.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.654 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:14.655 DEBUG:boto:Port: 8000 2023-10-17T18:37:14.655 DEBUG:boto:Params: {} 2023-10-17T18:37:14.656 DEBUG:boto:Token: None 2023-10-17T18:37:14.656 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-46/obj8 2023-10-17T18:37:14.657 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gjkbndXmQlBtkVRMR31P2BZ2SAs= 2023-10-17T18:37:14.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gjkbndXmQlBtkVRMR31P2BZ2SAs='} 2023-10-17T18:37:14.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3c080325741599b-00652ed45a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.663 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:14.663 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:14.663 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.664 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:14.664 DEBUG:boto:Data: 2023-10-17T18:37:14.665 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.665 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.665 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.666 DEBUG:boto:Params: {} 2023-10-17T18:37:14.666 DEBUG:boto:Token: None 2023-10-17T18:37:14.667 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.667 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4= 2023-10-17T18:37:14.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4=', 'Content-Length': '0'} 2023-10-17T18:37:14.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b919904b9648ad8-00652ed45a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.670 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:14.671 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:14.671 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.672 DEBUG:boto:Path: /hbmuog-47/obj5 2023-10-17T18:37:14.672 DEBUG:boto:Data: 2023-10-17T18:37:14.673 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.674 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.674 DEBUG:boto:Params: {} 2023-10-17T18:37:14.675 DEBUG:boto:Token: None 2023-10-17T18:37:14.675 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/obj5 2023-10-17T18:37:14.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p2nIaStUODYRheS1ofINjoPb3+c= 2023-10-17T18:37:14.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p2nIaStUODYRheS1ofINjoPb3+c='} 2023-10-17T18:37:14.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007224990b38e6dd26-00652ed45a-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.681 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:14.682 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:14.682 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.682 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:14.683 DEBUG:boto:Data: 2023-10-17T18:37:14.683 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.684 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.684 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.684 DEBUG:boto:Params: {} 2023-10-17T18:37:14.685 DEBUG:boto:Token: None 2023-10-17T18:37:14.685 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4= 2023-10-17T18:37:14.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4=', 'Content-Length': '0'} 2023-10-17T18:37:14.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e3853570eef4509-00652ed45a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.689 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:14.690 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:14.690 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.691 DEBUG:boto:Path: /hbmuog-47/obj6 2023-10-17T18:37:14.691 DEBUG:boto:Data: 2023-10-17T18:37:14.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.692 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.692 DEBUG:boto:Params: {} 2023-10-17T18:37:14.693 DEBUG:boto:Token: None 2023-10-17T18:37:14.693 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/obj6 2023-10-17T18:37:14.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zvFl43imVCW0JtJ78gLSemha/+8= 2023-10-17T18:37:14.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zvFl43imVCW0JtJ78gLSemha/+8='} 2023-10-17T18:37:14.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d785ce11c52434c3-00652ed45a-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.743 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:14.744 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:14.744 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.744 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:14.745 DEBUG:boto:Data: 2023-10-17T18:37:14.745 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.746 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.746 DEBUG:boto:Params: {} 2023-10-17T18:37:14.747 DEBUG:boto:Token: None 2023-10-17T18:37:14.747 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4= 2023-10-17T18:37:14.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4=', 'Content-Length': '0'} 2023-10-17T18:37:14.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3d989d7ce6e261b-00652ed45a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.753 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:14.753 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:14.754 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.754 DEBUG:boto:Path: /hbmuog-47/obj7 2023-10-17T18:37:14.755 DEBUG:boto:Data: 2023-10-17T18:37:14.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.756 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.756 DEBUG:boto:Params: {} 2023-10-17T18:37:14.757 DEBUG:boto:Token: None 2023-10-17T18:37:14.757 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/obj7 2023-10-17T18:37:14.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaLzz3xuatnFKjnr8oa6SBkllfE= 2023-10-17T18:37:14.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaLzz3xuatnFKjnr8oa6SBkllfE='} 2023-10-17T18:37:14.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008db94dc24a5d5ea3-00652ed45a-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.764 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:14.764 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:14.765 DEBUG:boto:Method: HEAD 2023-10-17T18:37:14.767 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:14.767 DEBUG:boto:Data: 2023-10-17T18:37:14.768 DEBUG:boto:Headers: {} 2023-10-17T18:37:14.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.768 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.769 DEBUG:boto:Params: {} 2023-10-17T18:37:14.769 DEBUG:boto:Token: None 2023-10-17T18:37:14.770 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/ 2023-10-17T18:37:14.770 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4= 2023-10-17T18:37:14.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K+zkS9OEK4O1G8SxWtbSSIDDBq4=', 'Content-Length': '0'} 2023-10-17T18:37:14.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091efdd5aed9dee8a-00652ed45a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.774 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:14.775 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:14.777 DEBUG:boto:Method: PUT 2023-10-17T18:37:14.777 DEBUG:boto:Path: /hbmuog-47/obj8 2023-10-17T18:37:14.778 DEBUG:boto:Data: 2023-10-17T18:37:14.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:14.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:14.780 DEBUG:boto:Port: 8001 2023-10-17T18:37:14.781 DEBUG:boto:Params: {} 2023-10-17T18:37:14.782 DEBUG:boto:Token: None 2023-10-17T18:37:14.783 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:14 GMT /hbmuog-47/obj8 2023-10-17T18:37:14.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U70v3iV5cbawIepSNm0+OHN9/xM= 2023-10-17T18:37:14.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U70v3iV5cbawIepSNm0+OHN9/xM='} 2023-10-17T18:37:14.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f65fd1423fec269-00652ed45a-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:14.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-46 2023-10-17T18:37:14.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:14.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:14.833 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:14.891 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:14.892 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:14.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:14.925+0000 7f93ff7bd780 1 waiting for incremental sync to catch up: 2023-10-17T18:37:14.927 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.54.6, , 00000000001.9.6, , 00000000001.35.6, , , 00000000001.14.6, , , ] 2023-10-17T18:37:14.927 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.58.6, 00000000002.30.1, 00000000004.13.6, 00000000002.17.1, 00000000004.43.6, 00000000002.22.1, 00000000002.22.1, 00000000004.18.6, 00000000002.31.1, 00000000002.20.1, 00000000002.23.1] 2023-10-17T18:37:19.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:19.926+0000 7f93ff7bd780 1 bucket sync caught up with source: 2023-10-17T18:37:19.928 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.58.6, 00000000002.30.1, 00000000004.13.6, 00000000002.17.1, 00000000004.43.6, 00000000002.22.1, 00000000002.22.1, 00000000004.18.6, 00000000002.31.1, 00000000002.20.1, 00000000002.23.1] 2023-10-17T18:37:19.929 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.58.6, 00000000002.30.1, 00000000004.13.6, 00000000002.17.1, 00000000004.43.6, 00000000002.22.1, 00000000002.22.1, 00000000004.18.6, 00000000002.31.1, 00000000002.20.1, 00000000002.23.1] 2023-10-17T18:37:19.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:19.926+0000 7f93ff7bd780 0 bucket checkpoint complete 2023-10-17T18:37:19.944 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-46 2023-10-17T18:37:19.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:19.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:19.946 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:20.011 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:20.012 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:20.048+0000 7f9eb65e3780 1 waiting for incremental sync to catch up: 2023-10-17T18:37:20.050 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.142.6, , 00000000001.39.6, , 00000000001.128.6, , , 00000000001.106.6, , , ] 2023-10-17T18:37:20.050 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000002.146.6, , 00000000002.43.6, , 00000000002.136.6, , , 00000000002.110.6, , , ] 2023-10-17T18:37:25.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:25.050+0000 7f9eb65e3780 1 waiting for incremental sync to catch up: 2023-10-17T18:37:25.051 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.142.6, , 00000000001.39.6, , 00000000001.128.6, , , 00000000001.106.6, , , ] 2023-10-17T18:37:25.052 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000002.146.6, , 00000000002.43.6, , 00000000002.136.6, , , 00000000002.110.6, , , ] 2023-10-17T18:37:30.053 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:30.051+0000 7f9eb65e3780 1 bucket sync caught up with source: 2023-10-17T18:37:30.053 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000002.146.6, , 00000000002.43.6, , 00000000002.136.6, , , 00000000002.110.6, , , ] 2023-10-17T18:37:30.053 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000002.146.6, , 00000000002.43.6, , 00000000002.136.6, , , 00000000002.110.6, , , ] 2023-10-17T18:37:30.053 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:30.051+0000 7f9eb65e3780 0 bucket checkpoint complete 2023-10-17T18:37:30.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-46 zones={a2, a1} 2023-10-17T18:37:30.070 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.071 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.071 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.072 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:30.072 DEBUG:boto:Data: 2023-10-17T18:37:30.073 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.073 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.074 DEBUG:boto:Params: {} 2023-10-17T18:37:30.074 DEBUG:boto:Token: None 2023-10-17T18:37:30.075 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vTsWvXaZxwf5S+7aglVJwK1w8EY= 2023-10-17T18:37:30.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vTsWvXaZxwf5S+7aglVJwK1w8EY=', 'Content-Length': '0'} 2023-10-17T18:37:30.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000824b0f51128b319a-00652ed46a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.080 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.080 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.081 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.081 DEBUG:boto:Path: /hbmuog-46/ 2023-10-17T18:37:30.081 DEBUG:boto:Data: 2023-10-17T18:37:30.082 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.083 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.083 DEBUG:boto:Params: {} 2023-10-17T18:37:30.083 DEBUG:boto:Token: None 2023-10-17T18:37:30.084 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vTsWvXaZxwf5S+7aglVJwK1w8EY= 2023-10-17T18:37:30.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vTsWvXaZxwf5S+7aglVJwK1w8EY=', 'Content-Length': '0'} 2023-10-17T18:37:30.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1593a30ec95aedc-00652ed46a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:37:30.089 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.090 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.090 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.090 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.091 DEBUG:boto:Method: GET 2023-10-17T18:37:30.091 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.092 DEBUG:boto:Data: 2023-10-17T18:37:30.092 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.093 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.093 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.093 DEBUG:boto:Params: {} 2023-10-17T18:37:30.094 DEBUG:boto:Token: None 2023-10-17T18:37:30.094 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/?versions 2023-10-17T18:37:30.095 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw= 2023-10-17T18:37:30.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw=', 'Content-Length': '0'} 2023-10-17T18:37:30.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009958e47a509f23bf-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.099 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:30.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:30.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:30.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:30.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:37:30.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:37:30.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:37:30.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:37:30.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:37:30.103 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.104 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.104 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.104 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.105 DEBUG:boto:Method: GET 2023-10-17T18:37:30.105 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.106 DEBUG:boto:Data: 2023-10-17T18:37:30.106 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.107 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.107 DEBUG:boto:Params: {} 2023-10-17T18:37:30.108 DEBUG:boto:Token: None 2023-10-17T18:37:30.108 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/?versions 2023-10-17T18:37:30.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw= 2023-10-17T18:37:30.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw=', 'Content-Length': '0'} 2023-10-17T18:37:30.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007986ec012bc6c1d5-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.115 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:30.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:30.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:30.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:30.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:37:30.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:37:30.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:37:30.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:37:30.120 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.120 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.120 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.121 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.121 DEBUG:boto:Method: GET 2023-10-17T18:37:30.122 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.122 DEBUG:boto:Data: 2023-10-17T18:37:30.123 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.123 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.124 DEBUG:boto:Params: {} 2023-10-17T18:37:30.124 DEBUG:boto:Token: None 2023-10-17T18:37:30.125 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/?versions 2023-10-17T18:37:30.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw= 2023-10-17T18:37:30.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw=', 'Content-Length': '0'} 2023-10-17T18:37:30.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f68e6fe628731b2b-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.129 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.130 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.130 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.131 DEBUG:boto:path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.131 DEBUG:boto:auth_path=/hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.132 DEBUG:boto:Method: GET 2023-10-17T18:37:30.132 DEBUG:boto:Path: /hbmuog-46/?versions&max-keys=999 2023-10-17T18:37:30.133 DEBUG:boto:Data: 2023-10-17T18:37:30.133 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.133 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.134 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.134 DEBUG:boto:Params: {} 2023-10-17T18:37:30.135 DEBUG:boto:Token: None 2023-10-17T18:37:30.135 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/?versions 2023-10-17T18:37:30.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw= 2023-10-17T18:37:30.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DaNmhZmdB/ttbjJe14fCwV1mrWw=', 'Content-Length': '0'} 2023-10-17T18:37:30.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a9c10eb8ddf685-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.139 DEBUG:boto:b'hbmuog-46999falseobj1nulltrue2023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:30.140 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.141 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.141 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.142 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.142 DEBUG:boto:Method: GET 2023-10-17T18:37:30.143 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.143 DEBUG:boto:Data: 2023-10-17T18:37:30.143 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.144 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.145 DEBUG:boto:Params: {} 2023-10-17T18:37:30.145 DEBUG:boto:Token: None 2023-10-17T18:37:30.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc= 2023-10-17T18:37:30.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc=', 'Content-Length': '0'} 2023-10-17T18:37:30.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df26e67969324101-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.149 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.149 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.150 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.150 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.151 DEBUG:boto:Method: GET 2023-10-17T18:37:30.151 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.152 DEBUG:boto:Data: 2023-10-17T18:37:30.152 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.153 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.153 DEBUG:boto:Params: {} 2023-10-17T18:37:30.154 DEBUG:boto:Token: None 2023-10-17T18:37:30.154 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.155 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc= 2023-10-17T18:37:30.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc=', 'Content-Length': '0'} 2023-10-17T18:37:30.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf596693af58864f-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.198 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.199 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.199 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.200 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.200 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.201 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.201 DEBUG:boto:Data: 2023-10-17T18:37:30.201 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.202 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.203 DEBUG:boto:Params: {} 2023-10-17T18:37:30.203 DEBUG:boto:Token: None 2023-10-17T18:37:30.203 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:84EuR1eUUtVRS6Bug3nwxfi1ra8= 2023-10-17T18:37:30.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:84EuR1eUUtVRS6Bug3nwxfi1ra8=', 'Content-Length': '0'} 2023-10-17T18:37:30.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000017de353d9b9c10-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.207 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.207 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.208 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.208 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.209 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.209 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.210 DEBUG:boto:Data: 2023-10-17T18:37:30.210 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.211 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.211 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.211 DEBUG:boto:Params: {} 2023-10-17T18:37:30.212 DEBUG:boto:Token: None 2023-10-17T18:37:30.212 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.213 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:84EuR1eUUtVRS6Bug3nwxfi1ra8= 2023-10-17T18:37:30.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:84EuR1eUUtVRS6Bug3nwxfi1ra8=', 'Content-Length': '0'} 2023-10-17T18:37:30.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000273eccdc7afcb8ca-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:30.216 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.217 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.217 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.218 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.218 DEBUG:boto:Method: GET 2023-10-17T18:37:30.219 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.219 DEBUG:boto:Data: 2023-10-17T18:37:30.220 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.220 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.220 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.221 DEBUG:boto:Params: {} 2023-10-17T18:37:30.221 DEBUG:boto:Token: None 2023-10-17T18:37:30.222 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.222 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc= 2023-10-17T18:37:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc=', 'Content-Length': '0'} 2023-10-17T18:37:30.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5088563b0bc8862-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.266 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.267 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.267 DEBUG:boto:path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.267 DEBUG:boto:auth_path=/hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.268 DEBUG:boto:Method: GET 2023-10-17T18:37:30.268 DEBUG:boto:Path: /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.269 DEBUG:boto:Data: 2023-10-17T18:37:30.269 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.270 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.270 DEBUG:boto:Params: {} 2023-10-17T18:37:30.271 DEBUG:boto:Token: None 2023-10-17T18:37:30.271 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1?versionId=null 2023-10-17T18:37:30.272 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc= 2023-10-17T18:37:30.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B0kkJhUwU9+2oXvJ8CUBMre+1uc=', 'Content-Length': '0'} 2023-10-17T18:37:30.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dc14be3e3b5ebaa-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.275 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.275 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.276 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.276 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.277 DEBUG:boto:Method: GET 2023-10-17T18:37:30.277 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.277 DEBUG:boto:Data: 2023-10-17T18:37:30.278 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.279 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.279 DEBUG:boto:Params: {} 2023-10-17T18:37:30.280 DEBUG:boto:Token: None 2023-10-17T18:37:30.280 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.281 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b05cb6dc6fd979-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.284 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.285 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.285 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.286 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.286 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.286 DEBUG:boto:Method: GET 2023-10-17T18:37:30.287 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.287 DEBUG:boto:Data: 2023-10-17T18:37:30.288 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.288 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.288 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.289 DEBUG:boto:Params: {} 2023-10-17T18:37:30.289 DEBUG:boto:Token: None 2023-10-17T18:37:30.290 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.290 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb930aecf71f15d-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.294 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.294 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.295 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.295 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.295 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:30.296 DEBUG:boto:Data: 2023-10-17T18:37:30.296 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.296 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.297 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.297 DEBUG:boto:Params: {} 2023-10-17T18:37:30.298 DEBUG:boto:Token: None 2023-10-17T18:37:30.298 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1 2023-10-17T18:37:30.298 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1MBzCG0E2M10gs32chj3PezX94I= 2023-10-17T18:37:30.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1MBzCG0E2M10gs32chj3PezX94I=', 'Content-Length': '0'} 2023-10-17T18:37:30.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dc64c2af44e6265-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.301 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.302 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.302 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.302 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:30.303 DEBUG:boto:Data: 2023-10-17T18:37:30.303 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.304 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.304 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.304 DEBUG:boto:Params: {} 2023-10-17T18:37:30.305 DEBUG:boto:Token: None 2023-10-17T18:37:30.305 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1 2023-10-17T18:37:30.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1MBzCG0E2M10gs32chj3PezX94I= 2023-10-17T18:37:30.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1MBzCG0E2M10gs32chj3PezX94I=', 'Content-Length': '0'} 2023-10-17T18:37:30.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3215428b5b6c851-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:30.309 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.309 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.310 DEBUG:boto:Method: GET 2023-10-17T18:37:30.310 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:30.311 DEBUG:boto:Data: 2023-10-17T18:37:30.311 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.311 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.312 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.312 DEBUG:boto:Params: {} 2023-10-17T18:37:30.313 DEBUG:boto:Token: None 2023-10-17T18:37:30.313 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1 2023-10-17T18:37:30.314 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DG4UJcErCb+dM58PfzT3UoYik4Y= 2023-10-17T18:37:30.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DG4UJcErCb+dM58PfzT3UoYik4Y=', 'Content-Length': '0'} 2023-10-17T18:37:30.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fda4a5c9927220b3-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.317 DEBUG:boto:path=/hbmuog-46/obj1 2023-10-17T18:37:30.317 DEBUG:boto:auth_path=/hbmuog-46/obj1 2023-10-17T18:37:30.318 DEBUG:boto:Method: GET 2023-10-17T18:37:30.318 DEBUG:boto:Path: /hbmuog-46/obj1 2023-10-17T18:37:30.319 DEBUG:boto:Data: 2023-10-17T18:37:30.319 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.320 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.320 DEBUG:boto:Params: {} 2023-10-17T18:37:30.321 DEBUG:boto:Token: None 2023-10-17T18:37:30.321 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj1 2023-10-17T18:37:30.322 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DG4UJcErCb+dM58PfzT3UoYik4Y= 2023-10-17T18:37:30.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DG4UJcErCb+dM58PfzT3UoYik4Y=', 'Content-Length': '0'} 2023-10-17T18:37:30.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcbbf1953af72b17-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.324 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.325 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.325 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.326 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.326 DEBUG:boto:Method: GET 2023-10-17T18:37:30.327 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.327 DEBUG:boto:Data: 2023-10-17T18:37:30.327 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.328 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.328 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.329 DEBUG:boto:Params: {} 2023-10-17T18:37:30.329 DEBUG:boto:Token: None 2023-10-17T18:37:30.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.330 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6f89d8a1c95886-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.334 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.334 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.335 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.335 DEBUG:boto:path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.336 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.336 DEBUG:boto:Method: GET 2023-10-17T18:37:30.336 DEBUG:boto:Path: /hbmuog-46/?prefix=obj1 2023-10-17T18:37:30.337 DEBUG:boto:Data: 2023-10-17T18:37:30.337 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.338 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.338 DEBUG:boto:Params: {} 2023-10-17T18:37:30.339 DEBUG:boto:Token: None 2023-10-17T18:37:30.339 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074735c5f319f51cc-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.344 DEBUG:boto:b'hbmuog-46obj11000falseobj12023-10-17T18:33:51.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:30.345 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.345 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.346 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.346 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.346 DEBUG:boto:Method: GET 2023-10-17T18:37:30.347 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.347 DEBUG:boto:Data: 2023-10-17T18:37:30.348 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.348 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.349 DEBUG:boto:Params: {} 2023-10-17T18:37:30.349 DEBUG:boto:Token: None 2023-10-17T18:37:30.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U= 2023-10-17T18:37:30.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U=', 'Content-Length': '0'} 2023-10-17T18:37:30.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021eabb22eb9d658e-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.353 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.354 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.354 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.355 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.355 DEBUG:boto:Method: GET 2023-10-17T18:37:30.356 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.356 DEBUG:boto:Data: 2023-10-17T18:37:30.356 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.357 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.358 DEBUG:boto:Params: {} 2023-10-17T18:37:30.358 DEBUG:boto:Token: None 2023-10-17T18:37:30.359 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U= 2023-10-17T18:37:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U=', 'Content-Length': '0'} 2023-10-17T18:37:30.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078a46c2682e0ba9e-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.402 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.403 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.404 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.404 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.404 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.405 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.405 DEBUG:boto:Data: 2023-10-17T18:37:30.406 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.406 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.407 DEBUG:boto:Params: {} 2023-10-17T18:37:30.407 DEBUG:boto:Token: None 2023-10-17T18:37:30.408 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zx0gEvRRIn1cZ2iTsIsgA7sS0yE= 2023-10-17T18:37:30.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zx0gEvRRIn1cZ2iTsIsgA7sS0yE=', 'Content-Length': '0'} 2023-10-17T18:37:30.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c7133c0e63680ad-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.411 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.412 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.412 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.413 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.413 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.413 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.414 DEBUG:boto:Data: 2023-10-17T18:37:30.414 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.415 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.415 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.415 DEBUG:boto:Params: {} 2023-10-17T18:37:30.416 DEBUG:boto:Token: None 2023-10-17T18:37:30.416 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.417 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zx0gEvRRIn1cZ2iTsIsgA7sS0yE= 2023-10-17T18:37:30.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zx0gEvRRIn1cZ2iTsIsgA7sS0yE=', 'Content-Length': '0'} 2023-10-17T18:37:30.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000746a8940ab096dcc-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:30.420 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.421 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.421 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.421 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.422 DEBUG:boto:Method: GET 2023-10-17T18:37:30.422 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.423 DEBUG:boto:Data: 2023-10-17T18:37:30.423 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.424 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.424 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.425 DEBUG:boto:Params: {} 2023-10-17T18:37:30.425 DEBUG:boto:Token: None 2023-10-17T18:37:30.425 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.426 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U= 2023-10-17T18:37:30.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U=', 'Content-Length': '0'} 2023-10-17T18:37:30.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2b6dc56db9f4cb7-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.474 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.475 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.475 DEBUG:boto:path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.476 DEBUG:boto:auth_path=/hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.476 DEBUG:boto:Method: GET 2023-10-17T18:37:30.477 DEBUG:boto:Path: /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.477 DEBUG:boto:Data: 2023-10-17T18:37:30.478 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.479 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.479 DEBUG:boto:Params: {} 2023-10-17T18:37:30.480 DEBUG:boto:Token: None 2023-10-17T18:37:30.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2?versionId=null 2023-10-17T18:37:30.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U= 2023-10-17T18:37:30.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iebauueLwv2Yd9oXhT5WYUEJ08U=', 'Content-Length': '0'} 2023-10-17T18:37:30.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002476ea476442b638-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.484 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.485 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.485 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.485 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.486 DEBUG:boto:Method: GET 2023-10-17T18:37:30.486 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.487 DEBUG:boto:Data: 2023-10-17T18:37:30.487 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.488 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.488 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.488 DEBUG:boto:Params: {} 2023-10-17T18:37:30.489 DEBUG:boto:Token: None 2023-10-17T18:37:30.489 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007223e9a95ef7120-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.494 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.495 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.495 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.495 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.496 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.496 DEBUG:boto:Method: GET 2023-10-17T18:37:30.497 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.497 DEBUG:boto:Data: 2023-10-17T18:37:30.497 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.498 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.499 DEBUG:boto:Params: {} 2023-10-17T18:37:30.499 DEBUG:boto:Token: None 2023-10-17T18:37:30.500 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35291c8967b892d-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.504 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.504 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.505 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.505 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.506 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:30.506 DEBUG:boto:Data: 2023-10-17T18:37:30.506 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.507 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.508 DEBUG:boto:Params: {} 2023-10-17T18:37:30.508 DEBUG:boto:Token: None 2023-10-17T18:37:30.509 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2 2023-10-17T18:37:30.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jf91EFbrp+JGnwjEqCxwcG8X594= 2023-10-17T18:37:30.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jf91EFbrp+JGnwjEqCxwcG8X594=', 'Content-Length': '0'} 2023-10-17T18:37:30.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049758e8363e9ce75-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.512 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.513 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.513 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.513 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:30.514 DEBUG:boto:Data: 2023-10-17T18:37:30.514 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.515 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.515 DEBUG:boto:Params: {} 2023-10-17T18:37:30.516 DEBUG:boto:Token: None 2023-10-17T18:37:30.516 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2 2023-10-17T18:37:30.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jf91EFbrp+JGnwjEqCxwcG8X594= 2023-10-17T18:37:30.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jf91EFbrp+JGnwjEqCxwcG8X594=', 'Content-Length': '0'} 2023-10-17T18:37:30.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df0c3e7abd8675d5-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:30.520 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.520 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.521 DEBUG:boto:Method: GET 2023-10-17T18:37:30.521 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:30.522 DEBUG:boto:Data: 2023-10-17T18:37:30.522 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.526 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.526 DEBUG:boto:Params: {} 2023-10-17T18:37:30.527 DEBUG:boto:Token: None 2023-10-17T18:37:30.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2 2023-10-17T18:37:30.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XY+iYc2msial4015dNdhHCfeQW4= 2023-10-17T18:37:30.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XY+iYc2msial4015dNdhHCfeQW4=', 'Content-Length': '0'} 2023-10-17T18:37:30.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bd7eb85e9f0fb20-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.531 DEBUG:boto:path=/hbmuog-46/obj2 2023-10-17T18:37:30.531 DEBUG:boto:auth_path=/hbmuog-46/obj2 2023-10-17T18:37:30.532 DEBUG:boto:Method: GET 2023-10-17T18:37:30.532 DEBUG:boto:Path: /hbmuog-46/obj2 2023-10-17T18:37:30.533 DEBUG:boto:Data: 2023-10-17T18:37:30.533 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.534 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.535 DEBUG:boto:Params: {} 2023-10-17T18:37:30.535 DEBUG:boto:Token: None 2023-10-17T18:37:30.535 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj2 2023-10-17T18:37:30.536 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XY+iYc2msial4015dNdhHCfeQW4= 2023-10-17T18:37:30.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XY+iYc2msial4015dNdhHCfeQW4=', 'Content-Length': '0'} 2023-10-17T18:37:30.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f779d484e11ff92-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.586 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.587 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.587 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.588 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.588 DEBUG:boto:Method: GET 2023-10-17T18:37:30.589 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.589 DEBUG:boto:Data: 2023-10-17T18:37:30.589 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.590 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.591 DEBUG:boto:Params: {} 2023-10-17T18:37:30.591 DEBUG:boto:Token: None 2023-10-17T18:37:30.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76c8fbc92f2d1ea-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.596 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.596 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.597 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.597 DEBUG:boto:path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.598 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.598 DEBUG:boto:Method: GET 2023-10-17T18:37:30.598 DEBUG:boto:Path: /hbmuog-46/?prefix=obj2 2023-10-17T18:37:30.599 DEBUG:boto:Data: 2023-10-17T18:37:30.599 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.600 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.600 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.601 DEBUG:boto:Params: {} 2023-10-17T18:37:30.601 DEBUG:boto:Token: None 2023-10-17T18:37:30.602 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.602 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49af9e902b13ecb-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.606 DEBUG:boto:b'hbmuog-46obj21000falseobj22023-10-17T18:33:51.437Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:30.607 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.607 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.608 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.608 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.608 DEBUG:boto:Method: GET 2023-10-17T18:37:30.609 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.609 DEBUG:boto:Data: 2023-10-17T18:37:30.609 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.610 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.611 DEBUG:boto:Params: {} 2023-10-17T18:37:30.611 DEBUG:boto:Token: None 2023-10-17T18:37:30.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw= 2023-10-17T18:37:30.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw=', 'Content-Length': '0'} 2023-10-17T18:37:30.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005967c1f6f040d8b3-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.615 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.616 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.616 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.617 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.617 DEBUG:boto:Method: GET 2023-10-17T18:37:30.617 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.618 DEBUG:boto:Data: 2023-10-17T18:37:30.618 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.619 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.620 DEBUG:boto:Params: {} 2023-10-17T18:37:30.620 DEBUG:boto:Token: None 2023-10-17T18:37:30.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw= 2023-10-17T18:37:30.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw=', 'Content-Length': '0'} 2023-10-17T18:37:30.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93bdbe18cd5b50e-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.624 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.625 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.625 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.625 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.626 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.626 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.627 DEBUG:boto:Data: 2023-10-17T18:37:30.627 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.627 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.628 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.628 DEBUG:boto:Params: {} 2023-10-17T18:37:30.629 DEBUG:boto:Token: None 2023-10-17T18:37:30.629 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.630 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7rv1pfF+V5kTi6dDyNUjtGADv6g= 2023-10-17T18:37:30.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7rv1pfF+V5kTi6dDyNUjtGADv6g=', 'Content-Length': '0'} 2023-10-17T18:37:30.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027dc80fa5bbde272-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.633 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.633 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.633 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.634 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.634 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.634 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.635 DEBUG:boto:Data: 2023-10-17T18:37:30.635 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.636 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.636 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.636 DEBUG:boto:Params: {} 2023-10-17T18:37:30.637 DEBUG:boto:Token: None 2023-10-17T18:37:30.637 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.638 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7rv1pfF+V5kTi6dDyNUjtGADv6g= 2023-10-17T18:37:30.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7rv1pfF+V5kTi6dDyNUjtGADv6g=', 'Content-Length': '0'} 2023-10-17T18:37:30.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055512cc1b35f9430-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:30.641 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.641 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.642 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.642 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.643 DEBUG:boto:Method: GET 2023-10-17T18:37:30.643 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.643 DEBUG:boto:Data: 2023-10-17T18:37:30.644 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.644 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.645 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.645 DEBUG:boto:Params: {} 2023-10-17T18:37:30.646 DEBUG:boto:Token: None 2023-10-17T18:37:30.646 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw= 2023-10-17T18:37:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw=', 'Content-Length': '0'} 2023-10-17T18:37:30.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b548330259121cc-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.694 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.695 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.695 DEBUG:boto:path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.696 DEBUG:boto:auth_path=/hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.696 DEBUG:boto:Method: GET 2023-10-17T18:37:30.697 DEBUG:boto:Path: /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.697 DEBUG:boto:Data: 2023-10-17T18:37:30.698 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.698 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.699 DEBUG:boto:Params: {} 2023-10-17T18:37:30.699 DEBUG:boto:Token: None 2023-10-17T18:37:30.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3?versionId=null 2023-10-17T18:37:30.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw= 2023-10-17T18:37:30.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AIz1Aykb24fabnU0YJnKqqCatSw=', 'Content-Length': '0'} 2023-10-17T18:37:30.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8aac534c449eaee-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.704 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.704 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.705 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.705 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.705 DEBUG:boto:Method: GET 2023-10-17T18:37:30.706 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.706 DEBUG:boto:Data: 2023-10-17T18:37:30.707 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.708 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.708 DEBUG:boto:Params: {} 2023-10-17T18:37:30.709 DEBUG:boto:Token: None 2023-10-17T18:37:30.709 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce3b4fa9f729717-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.713 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.714 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.714 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.715 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.715 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.716 DEBUG:boto:Method: GET 2023-10-17T18:37:30.716 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.717 DEBUG:boto:Data: 2023-10-17T18:37:30.717 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.718 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.718 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.718 DEBUG:boto:Params: {} 2023-10-17T18:37:30.719 DEBUG:boto:Token: None 2023-10-17T18:37:30.719 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7bb0c96f3864602-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.724 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.724 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.725 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.725 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.726 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:30.726 DEBUG:boto:Data: 2023-10-17T18:37:30.726 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.727 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.727 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.728 DEBUG:boto:Params: {} 2023-10-17T18:37:30.728 DEBUG:boto:Token: None 2023-10-17T18:37:30.729 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3 2023-10-17T18:37:30.729 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:a5Xx2+EMD7RBbdo+BS/8Dyyg0ZQ= 2023-10-17T18:37:30.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a5Xx2+EMD7RBbdo+BS/8Dyyg0ZQ=', 'Content-Length': '0'} 2023-10-17T18:37:30.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000116b895c10601c1c-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.732 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.733 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.733 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.734 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:30.734 DEBUG:boto:Data: 2023-10-17T18:37:30.734 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.735 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.735 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.736 DEBUG:boto:Params: {} 2023-10-17T18:37:30.736 DEBUG:boto:Token: None 2023-10-17T18:37:30.737 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3 2023-10-17T18:37:30.737 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:a5Xx2+EMD7RBbdo+BS/8Dyyg0ZQ= 2023-10-17T18:37:30.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a5Xx2+EMD7RBbdo+BS/8Dyyg0ZQ=', 'Content-Length': '0'} 2023-10-17T18:37:30.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e409892a4bf5e02e-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:30.740 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.741 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.741 DEBUG:boto:Method: GET 2023-10-17T18:37:30.742 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:30.742 DEBUG:boto:Data: 2023-10-17T18:37:30.743 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.744 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.744 DEBUG:boto:Params: {} 2023-10-17T18:37:30.745 DEBUG:boto:Token: None 2023-10-17T18:37:30.745 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3 2023-10-17T18:37:30.746 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dJCRyasRQRuJSB3LrO6gqhQ6UH0= 2023-10-17T18:37:30.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dJCRyasRQRuJSB3LrO6gqhQ6UH0=', 'Content-Length': '0'} 2023-10-17T18:37:30.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5606c022527fbfe-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.749 DEBUG:boto:path=/hbmuog-46/obj3 2023-10-17T18:37:30.749 DEBUG:boto:auth_path=/hbmuog-46/obj3 2023-10-17T18:37:30.750 DEBUG:boto:Method: GET 2023-10-17T18:37:30.750 DEBUG:boto:Path: /hbmuog-46/obj3 2023-10-17T18:37:30.751 DEBUG:boto:Data: 2023-10-17T18:37:30.751 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.752 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.752 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.753 DEBUG:boto:Params: {} 2023-10-17T18:37:30.753 DEBUG:boto:Token: None 2023-10-17T18:37:30.754 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj3 2023-10-17T18:37:30.754 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dJCRyasRQRuJSB3LrO6gqhQ6UH0= 2023-10-17T18:37:30.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dJCRyasRQRuJSB3LrO6gqhQ6UH0=', 'Content-Length': '0'} 2023-10-17T18:37:30.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5cc1a271e64dfa6-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.798 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.799 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.800 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.800 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.800 DEBUG:boto:Method: GET 2023-10-17T18:37:30.801 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.801 DEBUG:boto:Data: 2023-10-17T18:37:30.802 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.803 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.803 DEBUG:boto:Params: {} 2023-10-17T18:37:30.804 DEBUG:boto:Token: None 2023-10-17T18:37:30.804 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0322d791bf0916-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.808 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.809 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.810 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.810 DEBUG:boto:path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.811 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.811 DEBUG:boto:Method: GET 2023-10-17T18:37:30.812 DEBUG:boto:Path: /hbmuog-46/?prefix=obj3 2023-10-17T18:37:30.812 DEBUG:boto:Data: 2023-10-17T18:37:30.813 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.813 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.813 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.814 DEBUG:boto:Params: {} 2023-10-17T18:37:30.814 DEBUG:boto:Token: None 2023-10-17T18:37:30.815 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.815 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e9b69b0c02e4aa-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.819 DEBUG:boto:b'hbmuog-46obj31000falseobj32023-10-17T18:33:51.458Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:30.820 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.821 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.821 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.821 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.822 DEBUG:boto:Method: GET 2023-10-17T18:37:30.822 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.823 DEBUG:boto:Data: 2023-10-17T18:37:30.823 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.824 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.824 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.825 DEBUG:boto:Params: {} 2023-10-17T18:37:30.825 DEBUG:boto:Token: None 2023-10-17T18:37:30.826 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.826 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k= 2023-10-17T18:37:30.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k=', 'Content-Length': '0'} 2023-10-17T18:37:30.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2f6c99e6484d604-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.829 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.830 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.830 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.831 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.831 DEBUG:boto:Method: GET 2023-10-17T18:37:30.832 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.832 DEBUG:boto:Data: 2023-10-17T18:37:30.833 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.833 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.834 DEBUG:boto:Params: {} 2023-10-17T18:37:30.834 DEBUG:boto:Token: None 2023-10-17T18:37:30.835 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k= 2023-10-17T18:37:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k=', 'Content-Length': '0'} 2023-10-17T18:37:30.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7be6df56f54294f-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.839 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.839 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.840 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.840 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.841 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.841 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.841 DEBUG:boto:Data: 2023-10-17T18:37:30.842 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.842 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.843 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.843 DEBUG:boto:Params: {} 2023-10-17T18:37:30.844 DEBUG:boto:Token: None 2023-10-17T18:37:30.844 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.845 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z0XKWL/R8HJcYEDwOPbt6Vj1REA= 2023-10-17T18:37:30.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z0XKWL/R8HJcYEDwOPbt6Vj1REA=', 'Content-Length': '0'} 2023-10-17T18:37:30.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad275507ade42977-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.848 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.849 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.849 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.849 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.850 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.850 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.851 DEBUG:boto:Data: 2023-10-17T18:37:30.851 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.852 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.852 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.853 DEBUG:boto:Params: {} 2023-10-17T18:37:30.853 DEBUG:boto:Token: None 2023-10-17T18:37:30.854 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.854 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z0XKWL/R8HJcYEDwOPbt6Vj1REA= 2023-10-17T18:37:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z0XKWL/R8HJcYEDwOPbt6Vj1REA=', 'Content-Length': '0'} 2023-10-17T18:37:30.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a12be7ae0e5c603-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:30.858 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.858 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.859 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.859 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.860 DEBUG:boto:Method: GET 2023-10-17T18:37:30.860 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.860 DEBUG:boto:Data: 2023-10-17T18:37:30.861 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.861 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.862 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.862 DEBUG:boto:Params: {} 2023-10-17T18:37:30.863 DEBUG:boto:Token: None 2023-10-17T18:37:30.863 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k= 2023-10-17T18:37:30.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k=', 'Content-Length': '0'} 2023-10-17T18:37:30.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010a29d86859daf17-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.910 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.911 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.911 DEBUG:boto:path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.912 DEBUG:boto:auth_path=/hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.912 DEBUG:boto:Method: GET 2023-10-17T18:37:30.913 DEBUG:boto:Path: /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.913 DEBUG:boto:Data: 2023-10-17T18:37:30.913 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.914 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.914 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.915 DEBUG:boto:Params: {} 2023-10-17T18:37:30.915 DEBUG:boto:Token: None 2023-10-17T18:37:30.916 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4?versionId=null 2023-10-17T18:37:30.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k= 2023-10-17T18:37:30.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwNgPQak+a4CnhikAWrMLh0vZ9k=', 'Content-Length': '0'} 2023-10-17T18:37:30.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b7f31600908c39a-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.920 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.920 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.921 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.921 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.922 DEBUG:boto:Method: GET 2023-10-17T18:37:30.922 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.922 DEBUG:boto:Data: 2023-10-17T18:37:30.923 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.924 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.924 DEBUG:boto:Params: {} 2023-10-17T18:37:30.925 DEBUG:boto:Token: None 2023-10-17T18:37:30.925 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.926 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003faed9e9e20e8042-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.929 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.930 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.930 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.931 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.931 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.933 DEBUG:boto:Method: GET 2023-10-17T18:37:30.934 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.934 DEBUG:boto:Data: 2023-10-17T18:37:30.935 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.935 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.936 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.936 DEBUG:boto:Params: {} 2023-10-17T18:37:30.937 DEBUG:boto:Token: None 2023-10-17T18:37:30.937 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.938 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fd76931af434e4-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.941 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.942 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.943 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.943 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.943 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:30.944 DEBUG:boto:Data: 2023-10-17T18:37:30.944 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.945 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.945 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.946 DEBUG:boto:Params: {} 2023-10-17T18:37:30.946 DEBUG:boto:Token: None 2023-10-17T18:37:30.947 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4 2023-10-17T18:37:30.947 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:53Z9A/60HpdDuB5cCY+t4/swrgE= 2023-10-17T18:37:30.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:53Z9A/60HpdDuB5cCY+t4/swrgE=', 'Content-Length': '0'} 2023-10-17T18:37:30.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000605b39cf08f54dea-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.950 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.950 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.951 DEBUG:boto:Method: HEAD 2023-10-17T18:37:30.951 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:30.952 DEBUG:boto:Data: 2023-10-17T18:37:30.952 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.953 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.953 DEBUG:boto:Params: {} 2023-10-17T18:37:30.954 DEBUG:boto:Token: None 2023-10-17T18:37:30.954 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4 2023-10-17T18:37:30.955 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:53Z9A/60HpdDuB5cCY+t4/swrgE= 2023-10-17T18:37:30.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:53Z9A/60HpdDuB5cCY+t4/swrgE=', 'Content-Length': '0'} 2023-10-17T18:37:30.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b1a59ccf1928bc4-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:30.958 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.958 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.959 DEBUG:boto:Method: GET 2023-10-17T18:37:30.959 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:30.960 DEBUG:boto:Data: 2023-10-17T18:37:30.960 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.961 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.961 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.961 DEBUG:boto:Params: {} 2023-10-17T18:37:30.962 DEBUG:boto:Token: None 2023-10-17T18:37:30.962 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4 2023-10-17T18:37:30.963 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DmWzSYHhkkR020Mwo0Q/cbYuIcw= 2023-10-17T18:37:30.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DmWzSYHhkkR020Mwo0Q/cbYuIcw=', 'Content-Length': '0'} 2023-10-17T18:37:30.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc2f4272d4ad8095-00652ed46a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.966 DEBUG:boto:path=/hbmuog-46/obj4 2023-10-17T18:37:30.967 DEBUG:boto:auth_path=/hbmuog-46/obj4 2023-10-17T18:37:30.967 DEBUG:boto:Method: GET 2023-10-17T18:37:30.967 DEBUG:boto:Path: /hbmuog-46/obj4 2023-10-17T18:37:30.968 DEBUG:boto:Data: 2023-10-17T18:37:30.968 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.969 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.970 DEBUG:boto:Params: {} 2023-10-17T18:37:30.970 DEBUG:boto:Token: None 2023-10-17T18:37:30.971 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/obj4 2023-10-17T18:37:30.971 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DmWzSYHhkkR020Mwo0Q/cbYuIcw= 2023-10-17T18:37:30.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DmWzSYHhkkR020Mwo0Q/cbYuIcw=', 'Content-Length': '0'} 2023-10-17T18:37:30.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1cc1850d5733dfa-00652ed46a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.974 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.975 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.975 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.976 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.976 DEBUG:boto:Method: GET 2023-10-17T18:37:30.977 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.977 DEBUG:boto:Data: 2023-10-17T18:37:30.977 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.978 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.978 DEBUG:boto:Port: 8001 2023-10-17T18:37:30.979 DEBUG:boto:Params: {} 2023-10-17T18:37:30.979 DEBUG:boto:Token: None 2023-10-17T18:37:30.980 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab74bf1e4d8e55ba-00652ed46a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.984 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.985 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:30.985 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:30.986 DEBUG:boto:path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.986 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.986 DEBUG:boto:Method: GET 2023-10-17T18:37:30.987 DEBUG:boto:Path: /hbmuog-46/?prefix=obj4 2023-10-17T18:37:30.987 DEBUG:boto:Data: 2023-10-17T18:37:30.988 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.988 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:30.989 DEBUG:boto:Port: 8000 2023-10-17T18:37:30.989 DEBUG:boto:Params: {} 2023-10-17T18:37:30.990 DEBUG:boto:Token: None 2023-10-17T18:37:30.990 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:30 GMT /hbmuog-46/ 2023-10-17T18:37:30.991 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI= 2023-10-17T18:37:30.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qIgmCOZpoO32fAtzcD4/7AVcNvI=', 'Content-Length': '0'} 2023-10-17T18:37:30.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049066701e2911e14-00652ed46a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:30.994 DEBUG:boto:b'hbmuog-46obj41000falseobj42023-10-17T18:33:51.514Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:30.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:30.995 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:30.996 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:30.996 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:30.996 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:30.997 DEBUG:boto:Method: GET 2023-10-17T18:37:30.997 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:30.998 DEBUG:boto:Data: 2023-10-17T18:37:30.998 DEBUG:boto:Headers: {} 2023-10-17T18:37:30.999 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:30.999 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.000 DEBUG:boto:Params: {} 2023-10-17T18:37:31.000 DEBUG:boto:Token: None 2023-10-17T18:37:31.001 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.001 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8= 2023-10-17T18:37:31.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8=', 'Content-Length': '0'} 2023-10-17T18:37:31.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee2079f173cd9ab3-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.004 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.005 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.005 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.006 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.006 DEBUG:boto:Method: GET 2023-10-17T18:37:31.007 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.007 DEBUG:boto:Data: 2023-10-17T18:37:31.007 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.008 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.008 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.009 DEBUG:boto:Params: {} 2023-10-17T18:37:31.009 DEBUG:boto:Token: None 2023-10-17T18:37:31.010 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.010 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8= 2023-10-17T18:37:31.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8=', 'Content-Length': '0'} 2023-10-17T18:37:31.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa13c4c4c76a29f1-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.013 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.014 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.014 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.014 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.015 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.015 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.016 DEBUG:boto:Data: 2023-10-17T18:37:31.016 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.017 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.017 DEBUG:boto:Params: {} 2023-10-17T18:37:31.018 DEBUG:boto:Token: None 2023-10-17T18:37:31.018 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hPTmuz4Dw92lKDzDurycqW/wP4= 2023-10-17T18:37:31.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hPTmuz4Dw92lKDzDurycqW/wP4=', 'Content-Length': '0'} 2023-10-17T18:37:31.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001670738787a01455-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.022 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.022 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.023 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.023 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.024 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.024 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.024 DEBUG:boto:Data: 2023-10-17T18:37:31.025 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.025 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.026 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.026 DEBUG:boto:Params: {} 2023-10-17T18:37:31.026 DEBUG:boto:Token: None 2023-10-17T18:37:31.027 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.027 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hPTmuz4Dw92lKDzDurycqW/wP4= 2023-10-17T18:37:31.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hPTmuz4Dw92lKDzDurycqW/wP4=', 'Content-Length': '0'} 2023-10-17T18:37:31.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079897c7bfc7334c0-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:31.031 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.031 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.032 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.032 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.033 DEBUG:boto:Method: GET 2023-10-17T18:37:31.033 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.034 DEBUG:boto:Data: 2023-10-17T18:37:31.034 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.035 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.035 DEBUG:boto:Params: {} 2023-10-17T18:37:31.036 DEBUG:boto:Token: None 2023-10-17T18:37:31.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8= 2023-10-17T18:37:31.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8=', 'Content-Length': '0'} 2023-10-17T18:37:31.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076b6c000551583a8-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.040 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.040 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.041 DEBUG:boto:path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.041 DEBUG:boto:auth_path=/hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.042 DEBUG:boto:Method: GET 2023-10-17T18:37:31.042 DEBUG:boto:Path: /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.043 DEBUG:boto:Data: 2023-10-17T18:37:31.043 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.044 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.044 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.044 DEBUG:boto:Params: {} 2023-10-17T18:37:31.045 DEBUG:boto:Token: None 2023-10-17T18:37:31.045 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5?versionId=null 2023-10-17T18:37:31.046 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8= 2023-10-17T18:37:31.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C3X9F/27/Xch7Lz+CanKiej24f8=', 'Content-Length': '0'} 2023-10-17T18:37:31.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0a4ba4c3af67687-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.049 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.050 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.050 DEBUG:boto:path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.051 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.051 DEBUG:boto:Method: GET 2023-10-17T18:37:31.051 DEBUG:boto:Path: /hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.052 DEBUG:boto:Data: 2023-10-17T18:37:31.052 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.053 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.053 DEBUG:boto:Params: {} 2023-10-17T18:37:31.054 DEBUG:boto:Token: None 2023-10-17T18:37:31.054 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.055 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac4214ba1bb7e9a5-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.058 DEBUG:boto:b'hbmuog-46obj51000falseobj52023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.059 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.060 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.060 DEBUG:boto:path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.061 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.061 DEBUG:boto:Method: GET 2023-10-17T18:37:31.061 DEBUG:boto:Path: /hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.062 DEBUG:boto:Data: 2023-10-17T18:37:31.062 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.063 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.063 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.064 DEBUG:boto:Params: {} 2023-10-17T18:37:31.064 DEBUG:boto:Token: None 2023-10-17T18:37:31.064 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d6a3c009d6a9d95-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.068 DEBUG:boto:b'hbmuog-46obj51000falseobj52023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.069 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.069 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.070 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.070 DEBUG:boto:Path: /hbmuog-46/obj5 2023-10-17T18:37:31.071 DEBUG:boto:Data: 2023-10-17T18:37:31.071 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.072 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.072 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.072 DEBUG:boto:Params: {} 2023-10-17T18:37:31.073 DEBUG:boto:Token: None 2023-10-17T18:37:31.073 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5 2023-10-17T18:37:31.074 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xm1u9HvuaXdJEP6H5m03kCmz/FM= 2023-10-17T18:37:31.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xm1u9HvuaXdJEP6H5m03kCmz/FM=', 'Content-Length': '0'} 2023-10-17T18:37:31.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcff67a453e0f8b2-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.077 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.078 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.078 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.078 DEBUG:boto:Path: /hbmuog-46/obj5 2023-10-17T18:37:31.079 DEBUG:boto:Data: 2023-10-17T18:37:31.079 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.080 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.080 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.080 DEBUG:boto:Params: {} 2023-10-17T18:37:31.081 DEBUG:boto:Token: None 2023-10-17T18:37:31.081 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5 2023-10-17T18:37:31.082 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xm1u9HvuaXdJEP6H5m03kCmz/FM= 2023-10-17T18:37:31.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xm1u9HvuaXdJEP6H5m03kCmz/FM=', 'Content-Length': '0'} 2023-10-17T18:37:31.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c06998dda88ebc-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:31.085 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.086 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.086 DEBUG:boto:Method: GET 2023-10-17T18:37:31.087 DEBUG:boto:Path: /hbmuog-46/obj5 2023-10-17T18:37:31.087 DEBUG:boto:Data: 2023-10-17T18:37:31.088 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.088 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.089 DEBUG:boto:Params: {} 2023-10-17T18:37:31.089 DEBUG:boto:Token: None 2023-10-17T18:37:31.090 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5 2023-10-17T18:37:31.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gNRXW1vuyelYahl5n1+J3v+rAi8= 2023-10-17T18:37:31.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gNRXW1vuyelYahl5n1+J3v+rAi8=', 'Content-Length': '0'} 2023-10-17T18:37:31.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004effa6e92850d1fa-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.093 DEBUG:boto:path=/hbmuog-46/obj5 2023-10-17T18:37:31.094 DEBUG:boto:auth_path=/hbmuog-46/obj5 2023-10-17T18:37:31.094 DEBUG:boto:Method: GET 2023-10-17T18:37:31.095 DEBUG:boto:Path: /hbmuog-46/obj5 2023-10-17T18:37:31.095 DEBUG:boto:Data: 2023-10-17T18:37:31.095 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.096 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.097 DEBUG:boto:Params: {} 2023-10-17T18:37:31.097 DEBUG:boto:Token: None 2023-10-17T18:37:31.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj5 2023-10-17T18:37:31.098 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gNRXW1vuyelYahl5n1+J3v+rAi8= 2023-10-17T18:37:31.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gNRXW1vuyelYahl5n1+J3v+rAi8=', 'Content-Length': '0'} 2023-10-17T18:37:31.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8169ce3c8d4cfec-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.101 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.102 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.102 DEBUG:boto:path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.103 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.103 DEBUG:boto:Method: GET 2023-10-17T18:37:31.104 DEBUG:boto:Path: /hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.104 DEBUG:boto:Data: 2023-10-17T18:37:31.105 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.105 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.105 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.106 DEBUG:boto:Params: {} 2023-10-17T18:37:31.106 DEBUG:boto:Token: None 2023-10-17T18:37:31.107 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.107 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a52483858458a-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.111 DEBUG:boto:b'hbmuog-46obj51000falseobj52023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.112 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.112 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.113 DEBUG:boto:path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.113 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.114 DEBUG:boto:Method: GET 2023-10-17T18:37:31.114 DEBUG:boto:Path: /hbmuog-46/?prefix=obj5 2023-10-17T18:37:31.114 DEBUG:boto:Data: 2023-10-17T18:37:31.115 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.115 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.116 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.116 DEBUG:boto:Params: {} 2023-10-17T18:37:31.116 DEBUG:boto:Token: None 2023-10-17T18:37:31.117 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.117 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f4373b56db5a445-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.121 DEBUG:boto:b'hbmuog-46obj51000falseobj52023-10-17T18:37:14.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:31.122 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.122 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.123 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.123 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.124 DEBUG:boto:Method: GET 2023-10-17T18:37:31.124 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.125 DEBUG:boto:Data: 2023-10-17T18:37:31.125 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.125 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.126 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.126 DEBUG:boto:Params: {} 2023-10-17T18:37:31.127 DEBUG:boto:Token: None 2023-10-17T18:37:31.127 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo= 2023-10-17T18:37:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo=', 'Content-Length': '0'} 2023-10-17T18:37:31.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c2df7434ad27992-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.178 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.179 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.179 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.180 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.180 DEBUG:boto:Method: GET 2023-10-17T18:37:31.181 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.181 DEBUG:boto:Data: 2023-10-17T18:37:31.182 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.182 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.183 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.183 DEBUG:boto:Params: {} 2023-10-17T18:37:31.184 DEBUG:boto:Token: None 2023-10-17T18:37:31.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.185 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo= 2023-10-17T18:37:31.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo=', 'Content-Length': '0'} 2023-10-17T18:37:31.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002404dcd8e66e9dc-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.188 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.188 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.189 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.189 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.190 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.190 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.191 DEBUG:boto:Data: 2023-10-17T18:37:31.191 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.191 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.192 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.192 DEBUG:boto:Params: {} 2023-10-17T18:37:31.193 DEBUG:boto:Token: None 2023-10-17T18:37:31.193 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iWXAf1sQoDezhs7Wei9jSENR9Ig= 2023-10-17T18:37:31.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWXAf1sQoDezhs7Wei9jSENR9Ig=', 'Content-Length': '0'} 2023-10-17T18:37:31.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcf31c00859d28a5-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.197 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.197 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.198 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.198 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.198 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.199 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.199 DEBUG:boto:Data: 2023-10-17T18:37:31.200 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.200 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.200 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.201 DEBUG:boto:Params: {} 2023-10-17T18:37:31.201 DEBUG:boto:Token: None 2023-10-17T18:37:31.202 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.202 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iWXAf1sQoDezhs7Wei9jSENR9Ig= 2023-10-17T18:37:31.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWXAf1sQoDezhs7Wei9jSENR9Ig=', 'Content-Length': '0'} 2023-10-17T18:37:31.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001927b3c11676143d-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:31.205 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.206 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.206 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.207 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.207 DEBUG:boto:Method: GET 2023-10-17T18:37:31.208 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.208 DEBUG:boto:Data: 2023-10-17T18:37:31.209 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.209 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.210 DEBUG:boto:Params: {} 2023-10-17T18:37:31.210 DEBUG:boto:Token: None 2023-10-17T18:37:31.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo= 2023-10-17T18:37:31.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo=', 'Content-Length': '0'} 2023-10-17T18:37:31.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cd911d5cadfb70a-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.215 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.215 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.215 DEBUG:boto:path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.216 DEBUG:boto:auth_path=/hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.216 DEBUG:boto:Method: GET 2023-10-17T18:37:31.217 DEBUG:boto:Path: /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.217 DEBUG:boto:Data: 2023-10-17T18:37:31.218 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.218 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.219 DEBUG:boto:Params: {} 2023-10-17T18:37:31.219 DEBUG:boto:Token: None 2023-10-17T18:37:31.220 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6?versionId=null 2023-10-17T18:37:31.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo= 2023-10-17T18:37:31.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XzCE/1Ov3iqXx1f/rMH5AY4Gbpo=', 'Content-Length': '0'} 2023-10-17T18:37:31.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc77e91c16f66a5f-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.223 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.224 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.224 DEBUG:boto:path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.225 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.225 DEBUG:boto:Method: GET 2023-10-17T18:37:31.226 DEBUG:boto:Path: /hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.226 DEBUG:boto:Data: 2023-10-17T18:37:31.227 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.227 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.228 DEBUG:boto:Params: {} 2023-10-17T18:37:31.228 DEBUG:boto:Token: None 2023-10-17T18:37:31.229 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27b3e70889651df-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.233 DEBUG:boto:b'hbmuog-46obj61000falseobj62023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.233 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.234 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.234 DEBUG:boto:path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.235 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.235 DEBUG:boto:Method: GET 2023-10-17T18:37:31.236 DEBUG:boto:Path: /hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.236 DEBUG:boto:Data: 2023-10-17T18:37:31.236 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.237 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.237 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.238 DEBUG:boto:Params: {} 2023-10-17T18:37:31.238 DEBUG:boto:Token: None 2023-10-17T18:37:31.239 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008cbf69aea26e47a-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.286 DEBUG:boto:b'hbmuog-46obj61000falseobj62023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.287 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.288 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.288 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.289 DEBUG:boto:Path: /hbmuog-46/obj6 2023-10-17T18:37:31.289 DEBUG:boto:Data: 2023-10-17T18:37:31.290 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.290 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.290 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.291 DEBUG:boto:Params: {} 2023-10-17T18:37:31.291 DEBUG:boto:Token: None 2023-10-17T18:37:31.292 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6 2023-10-17T18:37:31.292 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oq/XDWT/z5XzMG6XqQOBbA9y2Cw= 2023-10-17T18:37:31.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oq/XDWT/z5XzMG6XqQOBbA9y2Cw=', 'Content-Length': '0'} 2023-10-17T18:37:31.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d6d0e2fcfca104d-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.296 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.296 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.297 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.297 DEBUG:boto:Path: /hbmuog-46/obj6 2023-10-17T18:37:31.297 DEBUG:boto:Data: 2023-10-17T18:37:31.298 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.299 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.299 DEBUG:boto:Params: {} 2023-10-17T18:37:31.300 DEBUG:boto:Token: None 2023-10-17T18:37:31.300 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6 2023-10-17T18:37:31.301 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oq/XDWT/z5XzMG6XqQOBbA9y2Cw= 2023-10-17T18:37:31.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oq/XDWT/z5XzMG6XqQOBbA9y2Cw=', 'Content-Length': '0'} 2023-10-17T18:37:31.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aeeedcf6cc23958-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:31.304 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.305 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.305 DEBUG:boto:Method: GET 2023-10-17T18:37:31.306 DEBUG:boto:Path: /hbmuog-46/obj6 2023-10-17T18:37:31.306 DEBUG:boto:Data: 2023-10-17T18:37:31.307 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.307 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.308 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.308 DEBUG:boto:Params: {} 2023-10-17T18:37:31.309 DEBUG:boto:Token: None 2023-10-17T18:37:31.309 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6 2023-10-17T18:37:31.310 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0GRzrgDBd1QDLajablEKMg/hnrk= 2023-10-17T18:37:31.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0GRzrgDBd1QDLajablEKMg/hnrk=', 'Content-Length': '0'} 2023-10-17T18:37:31.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d4f36ea62023bc6-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.313 DEBUG:boto:path=/hbmuog-46/obj6 2023-10-17T18:37:31.314 DEBUG:boto:auth_path=/hbmuog-46/obj6 2023-10-17T18:37:31.314 DEBUG:boto:Method: GET 2023-10-17T18:37:31.315 DEBUG:boto:Path: /hbmuog-46/obj6 2023-10-17T18:37:31.315 DEBUG:boto:Data: 2023-10-17T18:37:31.316 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.316 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.316 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.317 DEBUG:boto:Params: {} 2023-10-17T18:37:31.317 DEBUG:boto:Token: None 2023-10-17T18:37:31.318 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj6 2023-10-17T18:37:31.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0GRzrgDBd1QDLajablEKMg/hnrk= 2023-10-17T18:37:31.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0GRzrgDBd1QDLajablEKMg/hnrk=', 'Content-Length': '0'} 2023-10-17T18:37:31.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000362df64d2e5395c6-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.322 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.323 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.323 DEBUG:boto:path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.323 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.324 DEBUG:boto:Method: GET 2023-10-17T18:37:31.324 DEBUG:boto:Path: /hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.325 DEBUG:boto:Data: 2023-10-17T18:37:31.325 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.326 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.326 DEBUG:boto:Params: {} 2023-10-17T18:37:31.327 DEBUG:boto:Token: None 2023-10-17T18:37:31.327 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dba2bc55c3717bb-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.332 DEBUG:boto:b'hbmuog-46obj61000falseobj62023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.332 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.333 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.333 DEBUG:boto:path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.334 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.334 DEBUG:boto:Method: GET 2023-10-17T18:37:31.335 DEBUG:boto:Path: /hbmuog-46/?prefix=obj6 2023-10-17T18:37:31.335 DEBUG:boto:Data: 2023-10-17T18:37:31.335 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.336 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.337 DEBUG:boto:Params: {} 2023-10-17T18:37:31.337 DEBUG:boto:Token: None 2023-10-17T18:37:31.338 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f814c788d8209934-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.342 DEBUG:boto:b'hbmuog-46obj61000falseobj62023-10-17T18:37:14.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:31.347 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.347 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.347 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.348 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.348 DEBUG:boto:Method: GET 2023-10-17T18:37:31.349 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.349 DEBUG:boto:Data: 2023-10-17T18:37:31.350 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.351 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.351 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.351 DEBUG:boto:Params: {} 2023-10-17T18:37:31.352 DEBUG:boto:Token: None 2023-10-17T18:37:31.352 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.353 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8= 2023-10-17T18:37:31.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8=', 'Content-Length': '0'} 2023-10-17T18:37:31.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000654947f3c7021705-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.398 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.399 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.399 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.400 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.400 DEBUG:boto:Method: GET 2023-10-17T18:37:31.401 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.401 DEBUG:boto:Data: 2023-10-17T18:37:31.402 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.403 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.403 DEBUG:boto:Params: {} 2023-10-17T18:37:31.404 DEBUG:boto:Token: None 2023-10-17T18:37:31.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8= 2023-10-17T18:37:31.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8=', 'Content-Length': '0'} 2023-10-17T18:37:31.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000430a327718c6c6c2-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.408 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.408 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.409 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.409 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.409 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.411 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.411 DEBUG:boto:Data: 2023-10-17T18:37:31.411 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.412 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.412 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.413 DEBUG:boto:Params: {} 2023-10-17T18:37:31.413 DEBUG:boto:Token: None 2023-10-17T18:37:31.414 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.414 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aV47BHTs7N6/B/6VaABuJ5mVJro= 2023-10-17T18:37:31.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aV47BHTs7N6/B/6VaABuJ5mVJro=', 'Content-Length': '0'} 2023-10-17T18:37:31.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df00f909f5d2a47-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.417 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.418 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.418 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.419 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.419 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.419 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.420 DEBUG:boto:Data: 2023-10-17T18:37:31.420 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.421 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.421 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.421 DEBUG:boto:Params: {} 2023-10-17T18:37:31.422 DEBUG:boto:Token: None 2023-10-17T18:37:31.422 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aV47BHTs7N6/B/6VaABuJ5mVJro= 2023-10-17T18:37:31.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aV47BHTs7N6/B/6VaABuJ5mVJro=', 'Content-Length': '0'} 2023-10-17T18:37:31.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005caf1e6967302ca9-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:31.426 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.426 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.427 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.427 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.428 DEBUG:boto:Method: GET 2023-10-17T18:37:31.428 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.429 DEBUG:boto:Data: 2023-10-17T18:37:31.429 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.430 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.430 DEBUG:boto:Params: {} 2023-10-17T18:37:31.431 DEBUG:boto:Token: None 2023-10-17T18:37:31.431 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.432 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8= 2023-10-17T18:37:31.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8=', 'Content-Length': '0'} 2023-10-17T18:37:31.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065485d524ead9fd5-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.435 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.436 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.436 DEBUG:boto:path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.437 DEBUG:boto:auth_path=/hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.437 DEBUG:boto:Method: GET 2023-10-17T18:37:31.437 DEBUG:boto:Path: /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.438 DEBUG:boto:Data: 2023-10-17T18:37:31.438 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.439 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.440 DEBUG:boto:Params: {} 2023-10-17T18:37:31.440 DEBUG:boto:Token: None 2023-10-17T18:37:31.441 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7?versionId=null 2023-10-17T18:37:31.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8= 2023-10-17T18:37:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wpx0wSh+OkBIX/hUYv+xadVKJy8=', 'Content-Length': '0'} 2023-10-17T18:37:31.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020e0a3f174d45bae-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.486 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.489 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.490 DEBUG:boto:path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.490 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.491 DEBUG:boto:Method: GET 2023-10-17T18:37:31.491 DEBUG:boto:Path: /hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.492 DEBUG:boto:Data: 2023-10-17T18:37:31.493 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.494 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.494 DEBUG:boto:Params: {} 2023-10-17T18:37:31.495 DEBUG:boto:Token: None 2023-10-17T18:37:31.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253c49931d1426fb-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.499 DEBUG:boto:b'hbmuog-46obj71000falseobj72023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.500 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.501 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.501 DEBUG:boto:path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.502 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.502 DEBUG:boto:Method: GET 2023-10-17T18:37:31.503 DEBUG:boto:Path: /hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.503 DEBUG:boto:Data: 2023-10-17T18:37:31.504 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.505 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.505 DEBUG:boto:Params: {} 2023-10-17T18:37:31.506 DEBUG:boto:Token: None 2023-10-17T18:37:31.506 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12e97744efbcc3b-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.510 DEBUG:boto:b'hbmuog-46obj71000falseobj72023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.511 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.512 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.512 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.513 DEBUG:boto:Path: /hbmuog-46/obj7 2023-10-17T18:37:31.513 DEBUG:boto:Data: 2023-10-17T18:37:31.514 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.515 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.515 DEBUG:boto:Params: {} 2023-10-17T18:37:31.516 DEBUG:boto:Token: None 2023-10-17T18:37:31.516 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7 2023-10-17T18:37:31.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hRe2Y/o6gF9iT3p+3+2e9nTsXVc= 2023-10-17T18:37:31.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hRe2Y/o6gF9iT3p+3+2e9nTsXVc=', 'Content-Length': '0'} 2023-10-17T18:37:31.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb2d83b345906e1b-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.520 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.520 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.521 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.521 DEBUG:boto:Path: /hbmuog-46/obj7 2023-10-17T18:37:31.521 DEBUG:boto:Data: 2023-10-17T18:37:31.522 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.523 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.523 DEBUG:boto:Params: {} 2023-10-17T18:37:31.524 DEBUG:boto:Token: None 2023-10-17T18:37:31.524 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7 2023-10-17T18:37:31.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hRe2Y/o6gF9iT3p+3+2e9nTsXVc= 2023-10-17T18:37:31.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hRe2Y/o6gF9iT3p+3+2e9nTsXVc=', 'Content-Length': '0'} 2023-10-17T18:37:31.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a8f9fbecc435784-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:31.528 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.529 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.529 DEBUG:boto:Method: GET 2023-10-17T18:37:31.530 DEBUG:boto:Path: /hbmuog-46/obj7 2023-10-17T18:37:31.530 DEBUG:boto:Data: 2023-10-17T18:37:31.531 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.532 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.532 DEBUG:boto:Params: {} 2023-10-17T18:37:31.533 DEBUG:boto:Token: None 2023-10-17T18:37:31.533 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7 2023-10-17T18:37:31.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ET2jbKobFq3KzKoD9dKrstS2F+8= 2023-10-17T18:37:31.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ET2jbKobFq3KzKoD9dKrstS2F+8=', 'Content-Length': '0'} 2023-10-17T18:37:31.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fe050d400343ba0-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.578 DEBUG:boto:path=/hbmuog-46/obj7 2023-10-17T18:37:31.579 DEBUG:boto:auth_path=/hbmuog-46/obj7 2023-10-17T18:37:31.579 DEBUG:boto:Method: GET 2023-10-17T18:37:31.580 DEBUG:boto:Path: /hbmuog-46/obj7 2023-10-17T18:37:31.580 DEBUG:boto:Data: 2023-10-17T18:37:31.581 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.581 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.582 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.582 DEBUG:boto:Params: {} 2023-10-17T18:37:31.583 DEBUG:boto:Token: None 2023-10-17T18:37:31.583 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj7 2023-10-17T18:37:31.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ET2jbKobFq3KzKoD9dKrstS2F+8= 2023-10-17T18:37:31.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ET2jbKobFq3KzKoD9dKrstS2F+8=', 'Content-Length': '0'} 2023-10-17T18:37:31.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76d95859933526d-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.587 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.587 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.588 DEBUG:boto:path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.588 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.589 DEBUG:boto:Method: GET 2023-10-17T18:37:31.589 DEBUG:boto:Path: /hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.590 DEBUG:boto:Data: 2023-10-17T18:37:31.590 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.591 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.591 DEBUG:boto:Params: {} 2023-10-17T18:37:31.592 DEBUG:boto:Token: None 2023-10-17T18:37:31.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68169685a5c6ea2-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.596 DEBUG:boto:b'hbmuog-46obj71000falseobj72023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.597 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.598 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.598 DEBUG:boto:path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.599 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.599 DEBUG:boto:Method: GET 2023-10-17T18:37:31.600 DEBUG:boto:Path: /hbmuog-46/?prefix=obj7 2023-10-17T18:37:31.600 DEBUG:boto:Data: 2023-10-17T18:37:31.601 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.601 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.601 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.602 DEBUG:boto:Params: {} 2023-10-17T18:37:31.602 DEBUG:boto:Token: None 2023-10-17T18:37:31.603 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.603 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3e13a08bffe0e5-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.607 DEBUG:boto:b'hbmuog-46obj71000falseobj72023-10-17T18:37:14.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:31.608 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.609 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.609 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.610 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.610 DEBUG:boto:Method: GET 2023-10-17T18:37:31.611 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.611 DEBUG:boto:Data: 2023-10-17T18:37:31.611 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.612 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.613 DEBUG:boto:Params: {} 2023-10-17T18:37:31.613 DEBUG:boto:Token: None 2023-10-17T18:37:31.614 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs= 2023-10-17T18:37:31.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs=', 'Content-Length': '0'} 2023-10-17T18:37:31.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af52b3a9ffdaae2a-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.618 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.618 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.618 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.619 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.619 DEBUG:boto:Method: GET 2023-10-17T18:37:31.620 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.620 DEBUG:boto:Data: 2023-10-17T18:37:31.621 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.621 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.622 DEBUG:boto:Params: {} 2023-10-17T18:37:31.622 DEBUG:boto:Token: None 2023-10-17T18:37:31.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs= 2023-10-17T18:37:31.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs=', 'Content-Length': '0'} 2023-10-17T18:37:31.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052142fef8e5ec484-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.670 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.671 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.671 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.672 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.672 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.673 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.673 DEBUG:boto:Data: 2023-10-17T18:37:31.674 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.674 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.675 DEBUG:boto:Params: {} 2023-10-17T18:37:31.675 DEBUG:boto:Token: None 2023-10-17T18:37:31.676 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5gLK+N3UM9M8+4Kl1a7px7oIlsQ= 2023-10-17T18:37:31.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5gLK+N3UM9M8+4Kl1a7px7oIlsQ=', 'Content-Length': '0'} 2023-10-17T18:37:31.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e75664fc94aa25ea-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.680 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.680 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.680 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.681 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.681 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.682 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.682 DEBUG:boto:Data: 2023-10-17T18:37:31.683 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.683 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.684 DEBUG:boto:Params: {} 2023-10-17T18:37:31.684 DEBUG:boto:Token: None 2023-10-17T18:37:31.685 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5gLK+N3UM9M8+4Kl1a7px7oIlsQ= 2023-10-17T18:37:31.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5gLK+N3UM9M8+4Kl1a7px7oIlsQ=', 'Content-Length': '0'} 2023-10-17T18:37:31.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060a5a00930c688fa-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:31.689 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.689 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.690 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.690 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.691 DEBUG:boto:Method: GET 2023-10-17T18:37:31.691 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.692 DEBUG:boto:Data: 2023-10-17T18:37:31.692 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.693 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.693 DEBUG:boto:Params: {} 2023-10-17T18:37:31.694 DEBUG:boto:Token: None 2023-10-17T18:37:31.694 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.695 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs= 2023-10-17T18:37:31.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs=', 'Content-Length': '0'} 2023-10-17T18:37:31.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2fda667897cdbcc-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.699 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.700 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.701 DEBUG:boto:path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.702 DEBUG:boto:auth_path=/hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.703 DEBUG:boto:Method: GET 2023-10-17T18:37:31.704 DEBUG:boto:Path: /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.704 DEBUG:boto:Data: 2023-10-17T18:37:31.705 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.707 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.707 DEBUG:boto:Params: {} 2023-10-17T18:37:31.708 DEBUG:boto:Token: None 2023-10-17T18:37:31.708 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8?versionId=null 2023-10-17T18:37:31.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs= 2023-10-17T18:37:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pVGKpn9zcPMov6VdE0xOkh5zGxs=', 'Content-Length': '0'} 2023-10-17T18:37:31.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075986460f96c67e7-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.712 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.713 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.713 DEBUG:boto:path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.714 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.714 DEBUG:boto:Method: GET 2023-10-17T18:37:31.715 DEBUG:boto:Path: /hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.715 DEBUG:boto:Data: 2023-10-17T18:37:31.716 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.716 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.717 DEBUG:boto:Params: {} 2023-10-17T18:37:31.717 DEBUG:boto:Token: None 2023-10-17T18:37:31.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af734ff4b941fb54-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.723 DEBUG:boto:b'hbmuog-46obj81000falseobj82023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.724 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.725 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.725 DEBUG:boto:path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.726 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.727 DEBUG:boto:Method: GET 2023-10-17T18:37:31.728 DEBUG:boto:Path: /hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.729 DEBUG:boto:Data: 2023-10-17T18:37:31.729 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.731 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.731 DEBUG:boto:Params: {} 2023-10-17T18:37:31.732 DEBUG:boto:Token: None 2023-10-17T18:37:31.732 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca0536491069246-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.736 DEBUG:boto:b'hbmuog-46obj81000falseobj82023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.737 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.737 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.738 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.738 DEBUG:boto:Path: /hbmuog-46/obj8 2023-10-17T18:37:31.739 DEBUG:boto:Data: 2023-10-17T18:37:31.739 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.740 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.740 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.741 DEBUG:boto:Params: {} 2023-10-17T18:37:31.742 DEBUG:boto:Token: None 2023-10-17T18:37:31.742 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8 2023-10-17T18:37:31.743 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BsSXOgedB3F7E/5eK1wv/cfJyL8= 2023-10-17T18:37:31.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BsSXOgedB3F7E/5eK1wv/cfJyL8=', 'Content-Length': '0'} 2023-10-17T18:37:31.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd27d66a303f9b90-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.747 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.747 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.748 DEBUG:boto:Method: HEAD 2023-10-17T18:37:31.748 DEBUG:boto:Path: /hbmuog-46/obj8 2023-10-17T18:37:31.749 DEBUG:boto:Data: 2023-10-17T18:37:31.749 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.750 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.750 DEBUG:boto:Params: {} 2023-10-17T18:37:31.751 DEBUG:boto:Token: None 2023-10-17T18:37:31.751 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8 2023-10-17T18:37:31.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BsSXOgedB3F7E/5eK1wv/cfJyL8= 2023-10-17T18:37:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BsSXOgedB3F7E/5eK1wv/cfJyL8=', 'Content-Length': '0'} 2023-10-17T18:37:31.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e4437a948348640-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:31.756 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.756 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.757 DEBUG:boto:Method: GET 2023-10-17T18:37:31.757 DEBUG:boto:Path: /hbmuog-46/obj8 2023-10-17T18:37:31.758 DEBUG:boto:Data: 2023-10-17T18:37:31.758 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.759 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.760 DEBUG:boto:Params: {} 2023-10-17T18:37:31.760 DEBUG:boto:Token: None 2023-10-17T18:37:31.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8 2023-10-17T18:37:31.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J6PrQuspZXurS/L+MHB0K5xcKUU= 2023-10-17T18:37:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J6PrQuspZXurS/L+MHB0K5xcKUU=', 'Content-Length': '0'} 2023-10-17T18:37:31.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-46:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007745d559365e3af5-00652ed46b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.765 DEBUG:boto:path=/hbmuog-46/obj8 2023-10-17T18:37:31.765 DEBUG:boto:auth_path=/hbmuog-46/obj8 2023-10-17T18:37:31.765 DEBUG:boto:Method: GET 2023-10-17T18:37:31.766 DEBUG:boto:Path: /hbmuog-46/obj8 2023-10-17T18:37:31.767 DEBUG:boto:Data: 2023-10-17T18:37:31.767 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.768 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.768 DEBUG:boto:Params: {} 2023-10-17T18:37:31.769 DEBUG:boto:Token: None 2023-10-17T18:37:31.770 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/obj8 2023-10-17T18:37:31.770 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J6PrQuspZXurS/L+MHB0K5xcKUU= 2023-10-17T18:37:31.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J6PrQuspZXurS/L+MHB0K5xcKUU=', 'Content-Length': '0'} 2023-10-17T18:37:31.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091df1415d1c57187-00652ed46b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.818 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.819 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.820 DEBUG:boto:path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.820 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.820 DEBUG:boto:Method: GET 2023-10-17T18:37:31.821 DEBUG:boto:Path: /hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.821 DEBUG:boto:Data: 2023-10-17T18:37:31.822 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:31.823 DEBUG:boto:Port: 8001 2023-10-17T18:37:31.823 DEBUG:boto:Params: {} 2023-10-17T18:37:31.824 DEBUG:boto:Token: None 2023-10-17T18:37:31.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b106ba971d6ff76b-00652ed46b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.828 DEBUG:boto:b'hbmuog-46obj81000falseobj82023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.829 DEBUG:boto:path=/hbmuog-46/ 2023-10-17T18:37:31.829 DEBUG:boto:auth_path=/hbmuog-46/ 2023-10-17T18:37:31.830 DEBUG:boto:path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.830 DEBUG:boto:auth_path=/hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.831 DEBUG:boto:Method: GET 2023-10-17T18:37:31.831 DEBUG:boto:Path: /hbmuog-46/?prefix=obj8 2023-10-17T18:37:31.832 DEBUG:boto:Data: 2023-10-17T18:37:31.832 DEBUG:boto:Headers: {} 2023-10-17T18:37:31.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:31.833 DEBUG:boto:Port: 8000 2023-10-17T18:37:31.834 DEBUG:boto:Params: {} 2023-10-17T18:37:31.834 DEBUG:boto:Token: None 2023-10-17T18:37:31.835 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:31 GMT /hbmuog-46/ 2023-10-17T18:37:31.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw= 2023-10-17T18:37:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pJ1ExQ35rd1PvZO4ICXRILf/9Iw=', 'Content-Length': '0'} 2023-10-17T18:37:31.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6cbeb0b903c92b2-00652ed46b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:31.839 DEBUG:boto:b'hbmuog-46obj81000falseobj82023-10-17T18:37:14.659Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:31.840 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-46 zones={a2, a1} 2023-10-17T18:37:31.840 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-47 2023-10-17T18:37:31.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:31.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:31.841 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:31.909 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:31.909 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:31.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:31.948+0000 7f6c2f60c780 1 waiting for incremental sync to catch up: 2023-10-17T18:37:31.949 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.41.1, 00000000002.23.1, 00000000003.24.1, 00000000002.99.1, 00000000003.22.1, 00000000002.11.1, 00000000002.23.1, 00000000003.22.1, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:31.949 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.45.6, 00000000002.23.1, 00000000004.26.6, 00000000002.99.1, 00000000004.25.6, 00000000002.11.1, 00000000002.23.1, 00000000004.24.6, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:36.949 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:36.949+0000 7f6c2f60c780 1 waiting for incremental sync to catch up: 2023-10-17T18:37:36.950 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.41.1, 00000000002.23.1, 00000000003.24.1, 00000000002.99.1, 00000000003.22.1, 00000000002.11.1, 00000000002.23.1, 00000000003.22.1, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:36.950 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.45.6, 00000000002.23.1, 00000000004.26.6, 00000000002.99.1, 00000000004.25.6, 00000000002.11.1, 00000000002.23.1, 00000000004.24.6, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:41.951 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:41.950+0000 7f6c2f60c780 1 bucket sync caught up with source: 2023-10-17T18:37:41.951 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.45.6, 00000000002.23.1, 00000000004.26.6, 00000000002.99.1, 00000000004.25.6, 00000000002.11.1, 00000000002.23.1, 00000000004.24.6, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:41.951 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.45.6, 00000000002.23.1, 00000000004.26.6, 00000000002.99.1, 00000000004.25.6, 00000000002.11.1, 00000000002.23.1, 00000000004.24.6, 00000000002.19.1, 00000000002.33.1, 00000000002.20.1] 2023-10-17T18:37:41.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:41.950+0000 7f6c2f60c780 0 bucket checkpoint complete 2023-10-17T18:37:41.968 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-47 2023-10-17T18:37:41.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:41.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:41.970 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:42.035 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:42.035 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:42.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:42.073+0000 7f5e3f997780 1 bucket sync caught up with source: 2023-10-17T18:37:42.074 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000002.130.6, , 00000000002.255.6, , 00000000002.144.6, , , 00000000002.154.6, , , ] 2023-10-17T18:37:42.074 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000002.130.6, , 00000000002.255.6, , 00000000002.144.6, , , 00000000002.154.6, , , ] 2023-10-17T18:37:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:42.073+0000 7f5e3f997780 0 bucket checkpoint complete 2023-10-17T18:37:42.087 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-47 zones={a2, a1} 2023-10-17T18:37:42.088 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.088 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.088 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.089 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:42.089 DEBUG:boto:Data: 2023-10-17T18:37:42.090 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.091 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.091 DEBUG:boto:Params: {} 2023-10-17T18:37:42.091 DEBUG:boto:Token: None 2023-10-17T18:37:42.092 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dNsRu3/6ThAUsTDT448OqKB3kSk= 2023-10-17T18:37:42.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dNsRu3/6ThAUsTDT448OqKB3kSk=', 'Content-Length': '0'} 2023-10-17T18:37:42.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d844978f1d656aa8-00652ed476-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.096 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.096 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.097 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.097 DEBUG:boto:Path: /hbmuog-47/ 2023-10-17T18:37:42.097 DEBUG:boto:Data: 2023-10-17T18:37:42.098 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.099 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.099 DEBUG:boto:Params: {} 2023-10-17T18:37:42.100 DEBUG:boto:Token: None 2023-10-17T18:37:42.100 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dNsRu3/6ThAUsTDT448OqKB3kSk= 2023-10-17T18:37:42.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dNsRu3/6ThAUsTDT448OqKB3kSk=', 'Content-Length': '0'} 2023-10-17T18:37:42.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb0ed438c8e5a29a-00652ed476-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:37:42.105 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.105 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.105 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.106 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.107 DEBUG:boto:Method: GET 2023-10-17T18:37:42.107 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.107 DEBUG:boto:Data: 2023-10-17T18:37:42.108 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.109 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.109 DEBUG:boto:Params: {} 2023-10-17T18:37:42.110 DEBUG:boto:Token: None 2023-10-17T18:37:42.110 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/?versions 2023-10-17T18:37:42.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI= 2023-10-17T18:37:42.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI=', 'Content-Length': '0'} 2023-10-17T18:37:42.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003555deb25bc8711a-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.114 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:42.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:42.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:42.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:42.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:37:42.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:37:42.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:37:42.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:37:42.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:37:42.119 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.120 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.120 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.120 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.121 DEBUG:boto:Method: GET 2023-10-17T18:37:42.121 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.122 DEBUG:boto:Data: 2023-10-17T18:37:42.122 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.123 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.123 DEBUG:boto:Params: {} 2023-10-17T18:37:42.124 DEBUG:boto:Token: None 2023-10-17T18:37:42.124 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/?versions 2023-10-17T18:37:42.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI= 2023-10-17T18:37:42.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI=', 'Content-Length': '0'} 2023-10-17T18:37:42.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8e12eff4c3aee8-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.129 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:37:42.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:37:42.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:37:42.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:37:42.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:37:42.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:37:42.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:37:42.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:37:42.133 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.133 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.134 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.134 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.134 DEBUG:boto:Method: GET 2023-10-17T18:37:42.135 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.135 DEBUG:boto:Data: 2023-10-17T18:37:42.136 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.136 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.137 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.137 DEBUG:boto:Params: {} 2023-10-17T18:37:42.137 DEBUG:boto:Token: None 2023-10-17T18:37:42.138 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/?versions 2023-10-17T18:37:42.138 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI= 2023-10-17T18:37:42.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI=', 'Content-Length': '0'} 2023-10-17T18:37:42.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084eb32d4c701c2ce-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.142 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.143 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.143 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.144 DEBUG:boto:path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.144 DEBUG:boto:auth_path=/hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.144 DEBUG:boto:Method: GET 2023-10-17T18:37:42.145 DEBUG:boto:Path: /hbmuog-47/?versions&max-keys=999 2023-10-17T18:37:42.145 DEBUG:boto:Data: 2023-10-17T18:37:42.146 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.147 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.147 DEBUG:boto:Params: {} 2023-10-17T18:37:42.148 DEBUG:boto:Token: None 2023-10-17T18:37:42.148 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/?versions 2023-10-17T18:37:42.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI= 2023-10-17T18:37:42.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CHDi33vWFT4MnrKtGuxloPfjGdI=', 'Content-Length': '0'} 2023-10-17T18:37:42.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3aeb606f7d293f-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.152 DEBUG:boto:b'hbmuog-47999falseobj1nulltrue2023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:42.153 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.154 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.154 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.155 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.155 DEBUG:boto:Method: GET 2023-10-17T18:37:42.156 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.156 DEBUG:boto:Data: 2023-10-17T18:37:42.156 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.157 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.157 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.158 DEBUG:boto:Params: {} 2023-10-17T18:37:42.158 DEBUG:boto:Token: None 2023-10-17T18:37:42.159 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.159 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU= 2023-10-17T18:37:42.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU=', 'Content-Length': '0'} 2023-10-17T18:37:42.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000351297dedb17bd32-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.206 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.207 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.207 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.207 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.208 DEBUG:boto:Method: GET 2023-10-17T18:37:42.208 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.209 DEBUG:boto:Data: 2023-10-17T18:37:42.209 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.210 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.210 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.211 DEBUG:boto:Params: {} 2023-10-17T18:37:42.211 DEBUG:boto:Token: None 2023-10-17T18:37:42.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU= 2023-10-17T18:37:42.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU=', 'Content-Length': '0'} 2023-10-17T18:37:42.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027f72b6888a9b1ff-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.215 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.215 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.216 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.216 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.216 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.217 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.217 DEBUG:boto:Data: 2023-10-17T18:37:42.218 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.218 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.219 DEBUG:boto:Params: {} 2023-10-17T18:37:42.219 DEBUG:boto:Token: None 2023-10-17T18:37:42.220 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.221 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aUXRVHVrIYsGHDXgfySqTECMNhw= 2023-10-17T18:37:42.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aUXRVHVrIYsGHDXgfySqTECMNhw=', 'Content-Length': '0'} 2023-10-17T18:37:42.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce681f23720b8ef8-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.224 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.224 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.225 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.225 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.226 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.226 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.226 DEBUG:boto:Data: 2023-10-17T18:37:42.227 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.228 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.228 DEBUG:boto:Params: {} 2023-10-17T18:37:42.229 DEBUG:boto:Token: None 2023-10-17T18:37:42.229 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aUXRVHVrIYsGHDXgfySqTECMNhw= 2023-10-17T18:37:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aUXRVHVrIYsGHDXgfySqTECMNhw=', 'Content-Length': '0'} 2023-10-17T18:37:42.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d484219d0cafb0-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:42.233 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.233 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.234 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.234 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.235 DEBUG:boto:Method: GET 2023-10-17T18:37:42.235 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.236 DEBUG:boto:Data: 2023-10-17T18:37:42.236 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.237 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.237 DEBUG:boto:Params: {} 2023-10-17T18:37:42.238 DEBUG:boto:Token: None 2023-10-17T18:37:42.238 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU= 2023-10-17T18:37:42.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU=', 'Content-Length': '0'} 2023-10-17T18:37:42.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de10fc5de2108171-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.242 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.243 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.243 DEBUG:boto:path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.243 DEBUG:boto:auth_path=/hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.244 DEBUG:boto:Method: GET 2023-10-17T18:37:42.244 DEBUG:boto:Path: /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.245 DEBUG:boto:Data: 2023-10-17T18:37:42.245 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.246 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.246 DEBUG:boto:Params: {} 2023-10-17T18:37:42.247 DEBUG:boto:Token: None 2023-10-17T18:37:42.247 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1?versionId=null 2023-10-17T18:37:42.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU= 2023-10-17T18:37:42.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0ZZp/bQIGmGOUJDLI6H72v6CYU=', 'Content-Length': '0'} 2023-10-17T18:37:42.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000951d82ae34697670-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.294 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.295 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.295 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.296 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.296 DEBUG:boto:Method: GET 2023-10-17T18:37:42.297 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.297 DEBUG:boto:Data: 2023-10-17T18:37:42.298 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.298 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.299 DEBUG:boto:Params: {} 2023-10-17T18:37:42.299 DEBUG:boto:Token: None 2023-10-17T18:37:42.300 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054201db6407e5724-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.304 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.305 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.305 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.306 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.306 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.306 DEBUG:boto:Method: GET 2023-10-17T18:37:42.307 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.307 DEBUG:boto:Data: 2023-10-17T18:37:42.308 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.308 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.309 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.309 DEBUG:boto:Params: {} 2023-10-17T18:37:42.310 DEBUG:boto:Token: None 2023-10-17T18:37:42.310 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c285157554766ad5-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.314 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.315 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.315 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.316 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.316 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:42.316 DEBUG:boto:Data: 2023-10-17T18:37:42.317 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.318 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.318 DEBUG:boto:Params: {} 2023-10-17T18:37:42.319 DEBUG:boto:Token: None 2023-10-17T18:37:42.319 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1 2023-10-17T18:37:42.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9XPk7VdkS+P3BHVOKDHI6P6FVxY= 2023-10-17T18:37:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9XPk7VdkS+P3BHVOKDHI6P6FVxY=', 'Content-Length': '0'} 2023-10-17T18:37:42.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089e62fc4a6fdb54a-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.323 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.323 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.323 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.324 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:42.324 DEBUG:boto:Data: 2023-10-17T18:37:42.325 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.325 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.326 DEBUG:boto:Params: {} 2023-10-17T18:37:42.326 DEBUG:boto:Token: None 2023-10-17T18:37:42.327 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1 2023-10-17T18:37:42.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9XPk7VdkS+P3BHVOKDHI6P6FVxY= 2023-10-17T18:37:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9XPk7VdkS+P3BHVOKDHI6P6FVxY=', 'Content-Length': '0'} 2023-10-17T18:37:42.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076c5d57a116ee556-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:37:42.331 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.331 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.331 DEBUG:boto:Method: GET 2023-10-17T18:37:42.332 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:42.332 DEBUG:boto:Data: 2023-10-17T18:37:42.333 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.333 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.334 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.334 DEBUG:boto:Params: {} 2023-10-17T18:37:42.335 DEBUG:boto:Token: None 2023-10-17T18:37:42.335 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1 2023-10-17T18:37:42.336 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NeD4V1ryh5B36AMEDUpIDvpGJBc= 2023-10-17T18:37:42.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NeD4V1ryh5B36AMEDUpIDvpGJBc=', 'Content-Length': '0'} 2023-10-17T18:37:42.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f84dae15868d238b-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.339 DEBUG:boto:path=/hbmuog-47/obj1 2023-10-17T18:37:42.339 DEBUG:boto:auth_path=/hbmuog-47/obj1 2023-10-17T18:37:42.340 DEBUG:boto:Method: GET 2023-10-17T18:37:42.340 DEBUG:boto:Path: /hbmuog-47/obj1 2023-10-17T18:37:42.340 DEBUG:boto:Data: 2023-10-17T18:37:42.341 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.341 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.342 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.342 DEBUG:boto:Params: {} 2023-10-17T18:37:42.343 DEBUG:boto:Token: None 2023-10-17T18:37:42.343 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj1 2023-10-17T18:37:42.344 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NeD4V1ryh5B36AMEDUpIDvpGJBc= 2023-10-17T18:37:42.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NeD4V1ryh5B36AMEDUpIDvpGJBc=', 'Content-Length': '0'} 2023-10-17T18:37:42.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd5d790351f53910-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.346 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.347 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.347 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.348 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.348 DEBUG:boto:Method: GET 2023-10-17T18:37:42.349 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.349 DEBUG:boto:Data: 2023-10-17T18:37:42.349 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.350 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.350 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.351 DEBUG:boto:Params: {} 2023-10-17T18:37:42.351 DEBUG:boto:Token: None 2023-10-17T18:37:42.352 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.352 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e311753192e4fc-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.398 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.399 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.400 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.400 DEBUG:boto:path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.401 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.401 DEBUG:boto:Method: GET 2023-10-17T18:37:42.401 DEBUG:boto:Path: /hbmuog-47/?prefix=obj1 2023-10-17T18:37:42.402 DEBUG:boto:Data: 2023-10-17T18:37:42.402 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.403 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.403 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.404 DEBUG:boto:Params: {} 2023-10-17T18:37:42.404 DEBUG:boto:Token: None 2023-10-17T18:37:42.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db968f4184eeb52-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.408 DEBUG:boto:b'hbmuog-47obj11000falseobj12023-10-17T18:33:51.535Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:42.410 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.410 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.410 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.411 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.411 DEBUG:boto:Method: GET 2023-10-17T18:37:42.412 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.412 DEBUG:boto:Data: 2023-10-17T18:37:42.413 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.413 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.414 DEBUG:boto:Params: {} 2023-10-17T18:37:42.414 DEBUG:boto:Token: None 2023-10-17T18:37:42.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY= 2023-10-17T18:37:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY=', 'Content-Length': '0'} 2023-10-17T18:37:42.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e623c384d6cada-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.418 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.419 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.419 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.420 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.420 DEBUG:boto:Method: GET 2023-10-17T18:37:42.420 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.421 DEBUG:boto:Data: 2023-10-17T18:37:42.421 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.422 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.423 DEBUG:boto:Params: {} 2023-10-17T18:37:42.423 DEBUG:boto:Token: None 2023-10-17T18:37:42.425 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY= 2023-10-17T18:37:42.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY=', 'Content-Length': '0'} 2023-10-17T18:37:42.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcc176930f9c8ff1-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.429 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.429 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.429 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.430 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.430 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.431 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.431 DEBUG:boto:Data: 2023-10-17T18:37:42.432 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.432 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.432 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.433 DEBUG:boto:Params: {} 2023-10-17T18:37:42.433 DEBUG:boto:Token: None 2023-10-17T18:37:42.434 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.434 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ecmENR1UuVD/c5ij0fu5QyJpAZw= 2023-10-17T18:37:42.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ecmENR1UuVD/c5ij0fu5QyJpAZw=', 'Content-Length': '0'} 2023-10-17T18:37:42.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfc2994522b2028a-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.437 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.437 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.438 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.438 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.439 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.439 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.439 DEBUG:boto:Data: 2023-10-17T18:37:42.440 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.440 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.441 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.441 DEBUG:boto:Params: {} 2023-10-17T18:37:42.441 DEBUG:boto:Token: None 2023-10-17T18:37:42.442 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ecmENR1UuVD/c5ij0fu5QyJpAZw= 2023-10-17T18:37:42.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ecmENR1UuVD/c5ij0fu5QyJpAZw=', 'Content-Length': '0'} 2023-10-17T18:37:42.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0f9fb0d53bde13f-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:42.445 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.446 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.446 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.447 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.447 DEBUG:boto:Method: GET 2023-10-17T18:37:42.448 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.448 DEBUG:boto:Data: 2023-10-17T18:37:42.448 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.449 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.449 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.450 DEBUG:boto:Params: {} 2023-10-17T18:37:42.450 DEBUG:boto:Token: None 2023-10-17T18:37:42.451 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.451 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY= 2023-10-17T18:37:42.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY=', 'Content-Length': '0'} 2023-10-17T18:37:42.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000617a5ecccbe3b8d7-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.454 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.454 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.455 DEBUG:boto:path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.455 DEBUG:boto:auth_path=/hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.456 DEBUG:boto:Method: GET 2023-10-17T18:37:42.456 DEBUG:boto:Path: /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.457 DEBUG:boto:Data: 2023-10-17T18:37:42.457 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.458 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.458 DEBUG:boto:Params: {} 2023-10-17T18:37:42.459 DEBUG:boto:Token: None 2023-10-17T18:37:42.459 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2?versionId=null 2023-10-17T18:37:42.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY= 2023-10-17T18:37:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rvI+mW6z9fiQIEHwCybmsMfWDkY=', 'Content-Length': '0'} 2023-10-17T18:37:42.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056faeb090aceeb6a-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.463 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.463 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.464 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.464 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.465 DEBUG:boto:Method: GET 2023-10-17T18:37:42.465 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.465 DEBUG:boto:Data: 2023-10-17T18:37:42.466 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.467 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.467 DEBUG:boto:Params: {} 2023-10-17T18:37:42.468 DEBUG:boto:Token: None 2023-10-17T18:37:42.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045901490562cef92-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.472 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.473 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.473 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.474 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.474 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.475 DEBUG:boto:Method: GET 2023-10-17T18:37:42.475 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.475 DEBUG:boto:Data: 2023-10-17T18:37:42.476 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.477 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.477 DEBUG:boto:Params: {} 2023-10-17T18:37:42.478 DEBUG:boto:Token: None 2023-10-17T18:37:42.478 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001488c203ead077de-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.482 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.483 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.483 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.484 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.484 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:42.485 DEBUG:boto:Data: 2023-10-17T18:37:42.485 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.486 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.486 DEBUG:boto:Params: {} 2023-10-17T18:37:42.487 DEBUG:boto:Token: None 2023-10-17T18:37:42.487 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2 2023-10-17T18:37:42.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ydjBgGzm5wH+M30IU2c7VtZ2iMk= 2023-10-17T18:37:42.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ydjBgGzm5wH+M30IU2c7VtZ2iMk=', 'Content-Length': '0'} 2023-10-17T18:37:42.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdce1e58571fe58e-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.490 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.491 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.491 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.492 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:42.492 DEBUG:boto:Data: 2023-10-17T18:37:42.493 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.493 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.494 DEBUG:boto:Params: {} 2023-10-17T18:37:42.494 DEBUG:boto:Token: None 2023-10-17T18:37:42.495 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2 2023-10-17T18:37:42.495 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ydjBgGzm5wH+M30IU2c7VtZ2iMk= 2023-10-17T18:37:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ydjBgGzm5wH+M30IU2c7VtZ2iMk=', 'Content-Length': '0'} 2023-10-17T18:37:42.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007263446fdcb8d5f3-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:37:42.498 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.499 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.499 DEBUG:boto:Method: GET 2023-10-17T18:37:42.500 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:42.500 DEBUG:boto:Data: 2023-10-17T18:37:42.501 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.501 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.502 DEBUG:boto:Params: {} 2023-10-17T18:37:42.502 DEBUG:boto:Token: None 2023-10-17T18:37:42.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2 2023-10-17T18:37:42.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3J6b8xvScok6fXh3JLZJC0A7oY= 2023-10-17T18:37:42.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3J6b8xvScok6fXh3JLZJC0A7oY=', 'Content-Length': '0'} 2023-10-17T18:37:42.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009719dc50e93988f2-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.506 DEBUG:boto:path=/hbmuog-47/obj2 2023-10-17T18:37:42.507 DEBUG:boto:auth_path=/hbmuog-47/obj2 2023-10-17T18:37:42.507 DEBUG:boto:Method: GET 2023-10-17T18:37:42.507 DEBUG:boto:Path: /hbmuog-47/obj2 2023-10-17T18:37:42.508 DEBUG:boto:Data: 2023-10-17T18:37:42.508 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.509 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.509 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.510 DEBUG:boto:Params: {} 2023-10-17T18:37:42.510 DEBUG:boto:Token: None 2023-10-17T18:37:42.511 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj2 2023-10-17T18:37:42.511 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J3J6b8xvScok6fXh3JLZJC0A7oY= 2023-10-17T18:37:42.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J3J6b8xvScok6fXh3JLZJC0A7oY=', 'Content-Length': '0'} 2023-10-17T18:37:42.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000310dbb2db4c11943-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.514 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.515 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.515 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.515 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.516 DEBUG:boto:Method: GET 2023-10-17T18:37:42.516 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.516 DEBUG:boto:Data: 2023-10-17T18:37:42.517 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.518 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.518 DEBUG:boto:Params: {} 2023-10-17T18:37:42.519 DEBUG:boto:Token: None 2023-10-17T18:37:42.519 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074236a11b1859e93-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.523 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.523 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.524 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.524 DEBUG:boto:path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.525 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.525 DEBUG:boto:Method: GET 2023-10-17T18:37:42.526 DEBUG:boto:Path: /hbmuog-47/?prefix=obj2 2023-10-17T18:37:42.526 DEBUG:boto:Data: 2023-10-17T18:37:42.526 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.527 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.527 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.528 DEBUG:boto:Params: {} 2023-10-17T18:37:42.528 DEBUG:boto:Token: None 2023-10-17T18:37:42.529 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1683a0ffdd9ab60-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.532 DEBUG:boto:b'hbmuog-47obj21000falseobj22023-10-17T18:33:51.554Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:42.533 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.534 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.534 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.535 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.535 DEBUG:boto:Method: GET 2023-10-17T18:37:42.536 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.536 DEBUG:boto:Data: 2023-10-17T18:37:42.536 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.537 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.537 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.538 DEBUG:boto:Params: {} 2023-10-17T18:37:42.538 DEBUG:boto:Token: None 2023-10-17T18:37:42.539 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g= 2023-10-17T18:37:42.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g=', 'Content-Length': '0'} 2023-10-17T18:37:42.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd1b171186a67550-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.586 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.587 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.587 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.588 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.588 DEBUG:boto:Method: GET 2023-10-17T18:37:42.589 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.589 DEBUG:boto:Data: 2023-10-17T18:37:42.590 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.591 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.591 DEBUG:boto:Params: {} 2023-10-17T18:37:42.592 DEBUG:boto:Token: None 2023-10-17T18:37:42.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g= 2023-10-17T18:37:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g=', 'Content-Length': '0'} 2023-10-17T18:37:42.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006500443254fafc4b-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.596 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.596 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.597 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.597 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.598 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.598 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.598 DEBUG:boto:Data: 2023-10-17T18:37:42.599 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.600 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.600 DEBUG:boto:Params: {} 2023-10-17T18:37:42.600 DEBUG:boto:Token: None 2023-10-17T18:37:42.601 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sB6DCROT6Mhg5RTYCp4BxtGIdjU= 2023-10-17T18:37:42.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sB6DCROT6Mhg5RTYCp4BxtGIdjU=', 'Content-Length': '0'} 2023-10-17T18:37:42.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009221580fcdb6060f-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.604 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.605 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.605 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.605 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.606 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.606 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.607 DEBUG:boto:Data: 2023-10-17T18:37:42.607 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.608 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.608 DEBUG:boto:Params: {} 2023-10-17T18:37:42.609 DEBUG:boto:Token: None 2023-10-17T18:37:42.609 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.610 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sB6DCROT6Mhg5RTYCp4BxtGIdjU= 2023-10-17T18:37:42.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sB6DCROT6Mhg5RTYCp4BxtGIdjU=', 'Content-Length': '0'} 2023-10-17T18:37:42.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020325ec815106222-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:42.613 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.614 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.614 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.615 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.615 DEBUG:boto:Method: GET 2023-10-17T18:37:42.615 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.616 DEBUG:boto:Data: 2023-10-17T18:37:42.616 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.617 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.618 DEBUG:boto:Params: {} 2023-10-17T18:37:42.618 DEBUG:boto:Token: None 2023-10-17T18:37:42.619 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.619 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g= 2023-10-17T18:37:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g=', 'Content-Length': '0'} 2023-10-17T18:37:42.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d969b579f33ac9-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.622 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.623 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.623 DEBUG:boto:path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.624 DEBUG:boto:auth_path=/hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.624 DEBUG:boto:Method: GET 2023-10-17T18:37:42.624 DEBUG:boto:Path: /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.625 DEBUG:boto:Data: 2023-10-17T18:37:42.625 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.626 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.626 DEBUG:boto:Params: {} 2023-10-17T18:37:42.627 DEBUG:boto:Token: None 2023-10-17T18:37:42.629 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3?versionId=null 2023-10-17T18:37:42.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g= 2023-10-17T18:37:42.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KUHj0eIyU9mrN9ap18SAoLmDT4g=', 'Content-Length': '0'} 2023-10-17T18:37:42.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a057e4b05370dc88-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.678 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.679 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.679 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.680 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.680 DEBUG:boto:Method: GET 2023-10-17T18:37:42.681 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.681 DEBUG:boto:Data: 2023-10-17T18:37:42.681 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.682 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.683 DEBUG:boto:Params: {} 2023-10-17T18:37:42.683 DEBUG:boto:Token: None 2023-10-17T18:37:42.684 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008754a434ee80ecad-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.688 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.688 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.689 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.689 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.690 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.690 DEBUG:boto:Method: GET 2023-10-17T18:37:42.691 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.691 DEBUG:boto:Data: 2023-10-17T18:37:42.692 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.692 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.693 DEBUG:boto:Params: {} 2023-10-17T18:37:42.693 DEBUG:boto:Token: None 2023-10-17T18:37:42.694 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c856323784a04886-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.697 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.698 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.698 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.699 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.699 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:42.700 DEBUG:boto:Data: 2023-10-17T18:37:42.700 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.700 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.701 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.701 DEBUG:boto:Params: {} 2023-10-17T18:37:42.702 DEBUG:boto:Token: None 2023-10-17T18:37:42.702 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3 2023-10-17T18:37:42.703 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9xUdj82m8hIHDsdrgRTEI5nmnpg= 2023-10-17T18:37:42.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9xUdj82m8hIHDsdrgRTEI5nmnpg=', 'Content-Length': '0'} 2023-10-17T18:37:42.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad8694c207090542-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.705 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.705 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.706 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.706 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:42.707 DEBUG:boto:Data: 2023-10-17T18:37:42.707 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.708 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.708 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.708 DEBUG:boto:Params: {} 2023-10-17T18:37:42.709 DEBUG:boto:Token: None 2023-10-17T18:37:42.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3 2023-10-17T18:37:42.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9xUdj82m8hIHDsdrgRTEI5nmnpg= 2023-10-17T18:37:42.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9xUdj82m8hIHDsdrgRTEI5nmnpg=', 'Content-Length': '0'} 2023-10-17T18:37:42.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aed2596eac9425f3-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:37:42.713 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.713 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.714 DEBUG:boto:Method: GET 2023-10-17T18:37:42.714 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:42.714 DEBUG:boto:Data: 2023-10-17T18:37:42.715 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.715 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.716 DEBUG:boto:Params: {} 2023-10-17T18:37:42.716 DEBUG:boto:Token: None 2023-10-17T18:37:42.717 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3 2023-10-17T18:37:42.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RSZgAuKYw/Ffw5owGk0dCLysnrs= 2023-10-17T18:37:42.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RSZgAuKYw/Ffw5owGk0dCLysnrs=', 'Content-Length': '0'} 2023-10-17T18:37:42.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5b64b87af5ae2d5-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.720 DEBUG:boto:path=/hbmuog-47/obj3 2023-10-17T18:37:42.721 DEBUG:boto:auth_path=/hbmuog-47/obj3 2023-10-17T18:37:42.721 DEBUG:boto:Method: GET 2023-10-17T18:37:42.721 DEBUG:boto:Path: /hbmuog-47/obj3 2023-10-17T18:37:42.722 DEBUG:boto:Data: 2023-10-17T18:37:42.722 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.723 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.723 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.723 DEBUG:boto:Params: {} 2023-10-17T18:37:42.724 DEBUG:boto:Token: None 2023-10-17T18:37:42.724 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj3 2023-10-17T18:37:42.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RSZgAuKYw/Ffw5owGk0dCLysnrs= 2023-10-17T18:37:42.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RSZgAuKYw/Ffw5owGk0dCLysnrs=', 'Content-Length': '0'} 2023-10-17T18:37:42.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af5fe90c45fd8431-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.728 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.728 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.729 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.729 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.730 DEBUG:boto:Method: GET 2023-10-17T18:37:42.730 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.730 DEBUG:boto:Data: 2023-10-17T18:37:42.731 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.732 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.732 DEBUG:boto:Params: {} 2023-10-17T18:37:42.733 DEBUG:boto:Token: None 2023-10-17T18:37:42.733 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7917efa440c5184-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.737 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.738 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.738 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.739 DEBUG:boto:path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.739 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.739 DEBUG:boto:Method: GET 2023-10-17T18:37:42.740 DEBUG:boto:Path: /hbmuog-47/?prefix=obj3 2023-10-17T18:37:42.740 DEBUG:boto:Data: 2023-10-17T18:37:42.741 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.741 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.742 DEBUG:boto:Params: {} 2023-10-17T18:37:42.742 DEBUG:boto:Token: None 2023-10-17T18:37:42.743 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.743 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000587d4f2e33bf104e-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.747 DEBUG:boto:b'hbmuog-47obj31000falseobj32023-10-17T18:33:51.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:42.748 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.748 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.748 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.749 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.749 DEBUG:boto:Method: GET 2023-10-17T18:37:42.750 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.750 DEBUG:boto:Data: 2023-10-17T18:37:42.751 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.751 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.751 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.752 DEBUG:boto:Params: {} 2023-10-17T18:37:42.752 DEBUG:boto:Token: None 2023-10-17T18:37:42.753 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek= 2023-10-17T18:37:42.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek=', 'Content-Length': '0'} 2023-10-17T18:37:42.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099797e1508a59a09-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.756 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.757 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.757 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.758 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.758 DEBUG:boto:Method: GET 2023-10-17T18:37:42.758 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.759 DEBUG:boto:Data: 2023-10-17T18:37:42.759 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.760 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.761 DEBUG:boto:Params: {} 2023-10-17T18:37:42.761 DEBUG:boto:Token: None 2023-10-17T18:37:42.762 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek= 2023-10-17T18:37:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek=', 'Content-Length': '0'} 2023-10-17T18:37:42.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c048d3f1d9852e07-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.806 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.807 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.807 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.808 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.808 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.808 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.809 DEBUG:boto:Data: 2023-10-17T18:37:42.810 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.811 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.811 DEBUG:boto:Params: {} 2023-10-17T18:37:42.811 DEBUG:boto:Token: None 2023-10-17T18:37:42.812 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k3gVNRa+yiifb0ZFjNcaB2qQgpI= 2023-10-17T18:37:42.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k3gVNRa+yiifb0ZFjNcaB2qQgpI=', 'Content-Length': '0'} 2023-10-17T18:37:42.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000754ef9b1529dcf9-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.815 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.815 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.816 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.816 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.817 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.817 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.817 DEBUG:boto:Data: 2023-10-17T18:37:42.818 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.819 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.819 DEBUG:boto:Params: {} 2023-10-17T18:37:42.819 DEBUG:boto:Token: None 2023-10-17T18:37:42.820 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k3gVNRa+yiifb0ZFjNcaB2qQgpI= 2023-10-17T18:37:42.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k3gVNRa+yiifb0ZFjNcaB2qQgpI=', 'Content-Length': '0'} 2023-10-17T18:37:42.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000126083f4336c5507-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:42.824 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.824 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.825 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.825 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.826 DEBUG:boto:Method: GET 2023-10-17T18:37:42.826 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.827 DEBUG:boto:Data: 2023-10-17T18:37:42.827 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.828 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.828 DEBUG:boto:Params: {} 2023-10-17T18:37:42.829 DEBUG:boto:Token: None 2023-10-17T18:37:42.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek= 2023-10-17T18:37:42.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek=', 'Content-Length': '0'} 2023-10-17T18:37:42.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004595fdc0de36291d-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.878 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.879 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.879 DEBUG:boto:path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.880 DEBUG:boto:auth_path=/hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.880 DEBUG:boto:Method: GET 2023-10-17T18:37:42.881 DEBUG:boto:Path: /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.881 DEBUG:boto:Data: 2023-10-17T18:37:42.882 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.882 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.883 DEBUG:boto:Params: {} 2023-10-17T18:37:42.883 DEBUG:boto:Token: None 2023-10-17T18:37:42.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4?versionId=null 2023-10-17T18:37:42.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek= 2023-10-17T18:37:42.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kA1jY8WH9QjpX5n1obZQDBDxLek=', 'Content-Length': '0'} 2023-10-17T18:37:42.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8d11b826d898c94-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.887 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.888 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.888 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.889 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.889 DEBUG:boto:Method: GET 2023-10-17T18:37:42.890 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.890 DEBUG:boto:Data: 2023-10-17T18:37:42.891 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.891 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.892 DEBUG:boto:Params: {} 2023-10-17T18:37:42.892 DEBUG:boto:Token: None 2023-10-17T18:37:42.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081a6dc53dcfcbb2a-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.896 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.897 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.898 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.898 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.899 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.899 DEBUG:boto:Method: GET 2023-10-17T18:37:42.900 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.900 DEBUG:boto:Data: 2023-10-17T18:37:42.901 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.901 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.902 DEBUG:boto:Params: {} 2023-10-17T18:37:42.902 DEBUG:boto:Token: None 2023-10-17T18:37:42.903 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.903 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750db3ce58fa6580-00652ed476-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.907 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.907 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.908 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.908 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.909 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:42.909 DEBUG:boto:Data: 2023-10-17T18:37:42.909 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.910 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.911 DEBUG:boto:Params: {} 2023-10-17T18:37:42.911 DEBUG:boto:Token: None 2023-10-17T18:37:42.912 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4 2023-10-17T18:37:42.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZaRFzFK640hrXpcDlOLZg0CxN0M= 2023-10-17T18:37:42.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZaRFzFK640hrXpcDlOLZg0CxN0M=', 'Content-Length': '0'} 2023-10-17T18:37:42.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3ef1287999a1b8b-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.915 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.915 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.916 DEBUG:boto:Method: HEAD 2023-10-17T18:37:42.916 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:42.917 DEBUG:boto:Data: 2023-10-17T18:37:42.917 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.918 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.918 DEBUG:boto:Params: {} 2023-10-17T18:37:42.919 DEBUG:boto:Token: None 2023-10-17T18:37:42.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4 2023-10-17T18:37:42.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZaRFzFK640hrXpcDlOLZg0CxN0M= 2023-10-17T18:37:42.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZaRFzFK640hrXpcDlOLZg0CxN0M=', 'Content-Length': '0'} 2023-10-17T18:37:42.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008074071a012eaad1-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:37:42.923 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.924 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.924 DEBUG:boto:Method: GET 2023-10-17T18:37:42.925 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:42.925 DEBUG:boto:Data: 2023-10-17T18:37:42.926 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.927 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.927 DEBUG:boto:Params: {} 2023-10-17T18:37:42.927 DEBUG:boto:Token: None 2023-10-17T18:37:42.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4 2023-10-17T18:37:42.928 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1js9L0gD2q6ELrrmRo3ybbWb+Y= 2023-10-17T18:37:42.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1js9L0gD2q6ELrrmRo3ybbWb+Y=', 'Content-Length': '0'} 2023-10-17T18:37:42.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d86d78149425257-00652ed476-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.931 DEBUG:boto:path=/hbmuog-47/obj4 2023-10-17T18:37:42.932 DEBUG:boto:auth_path=/hbmuog-47/obj4 2023-10-17T18:37:42.932 DEBUG:boto:Method: GET 2023-10-17T18:37:42.933 DEBUG:boto:Path: /hbmuog-47/obj4 2023-10-17T18:37:42.933 DEBUG:boto:Data: 2023-10-17T18:37:42.934 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:42.934 DEBUG:boto:Port: 8000 2023-10-17T18:37:42.935 DEBUG:boto:Params: {} 2023-10-17T18:37:42.935 DEBUG:boto:Token: None 2023-10-17T18:37:42.936 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/obj4 2023-10-17T18:37:42.937 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1js9L0gD2q6ELrrmRo3ybbWb+Y= 2023-10-17T18:37:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1js9L0gD2q6ELrrmRo3ybbWb+Y=', 'Content-Length': '0'} 2023-10-17T18:37:42.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:33:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cab5306b503568d-00652ed476-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.986 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.987 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.987 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.988 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.988 DEBUG:boto:Method: GET 2023-10-17T18:37:42.989 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.989 DEBUG:boto:Data: 2023-10-17T18:37:42.990 DEBUG:boto:Headers: {} 2023-10-17T18:37:42.990 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:42.991 DEBUG:boto:Port: 8001 2023-10-17T18:37:42.991 DEBUG:boto:Params: {} 2023-10-17T18:37:42.991 DEBUG:boto:Token: None 2023-10-17T18:37:42.992 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:42 GMT /hbmuog-47/ 2023-10-17T18:37:42.992 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY= 2023-10-17T18:37:42.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZxDctFRymfcZtnmIHRedsPk5leY=', 'Content-Length': '0'} 2023-10-17T18:37:42.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca70d0b43f80d75-00652ed476-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:42.996 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:42.997 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:42.997 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:42.998 DEBUG:boto:path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.998 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj4 2023-10-17T18:37:42.999 DEBUG:boto:Method: GET 2023-10-17T18:37:42.999 DEBUG:boto:Path: /hbmuog-47/?prefix=obj4 2023-10-17T18:37:43.000 DEBUG:boto:Data: 2023-10-17T18:37:43.000 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.000 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.001 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.001 DEBUG:boto:Params: {} 2023-10-17T18:37:43.002 DEBUG:boto:Token: None 2023-10-17T18:37:43.002 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.003 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000930b4879c75bc3a3-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.006 DEBUG:boto:b'hbmuog-47obj41000falseobj42023-10-17T18:33:51.629Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:43.007 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.008 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.008 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.008 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.009 DEBUG:boto:Method: GET 2023-10-17T18:37:43.009 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.010 DEBUG:boto:Data: 2023-10-17T18:37:43.010 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.011 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.012 DEBUG:boto:Params: {} 2023-10-17T18:37:43.012 DEBUG:boto:Token: None 2023-10-17T18:37:43.012 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk= 2023-10-17T18:37:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk=', 'Content-Length': '0'} 2023-10-17T18:37:43.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006be1cdbf3b8fed1c-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.062 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.063 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.063 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.064 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.064 DEBUG:boto:Method: GET 2023-10-17T18:37:43.065 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.065 DEBUG:boto:Data: 2023-10-17T18:37:43.066 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.066 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.067 DEBUG:boto:Params: {} 2023-10-17T18:37:43.067 DEBUG:boto:Token: None 2023-10-17T18:37:43.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk= 2023-10-17T18:37:43.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk=', 'Content-Length': '0'} 2023-10-17T18:37:43.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059ed88c18ba5ef45-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.072 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.072 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.073 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.073 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.073 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.074 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.074 DEBUG:boto:Data: 2023-10-17T18:37:43.075 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.075 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.075 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.076 DEBUG:boto:Params: {} 2023-10-17T18:37:43.076 DEBUG:boto:Token: None 2023-10-17T18:37:43.077 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.077 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bCKvvAyC9ymacW4upu4AL10u7yQ= 2023-10-17T18:37:43.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bCKvvAyC9ymacW4upu4AL10u7yQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008151b096aefc5f31-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.080 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.081 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.081 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.082 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.082 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.083 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.083 DEBUG:boto:Data: 2023-10-17T18:37:43.084 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.084 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.084 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.085 DEBUG:boto:Params: {} 2023-10-17T18:37:43.085 DEBUG:boto:Token: None 2023-10-17T18:37:43.086 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.086 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bCKvvAyC9ymacW4upu4AL10u7yQ= 2023-10-17T18:37:43.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bCKvvAyC9ymacW4upu4AL10u7yQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a160806b07af7f7-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:43.090 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.090 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.091 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.091 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.092 DEBUG:boto:Method: GET 2023-10-17T18:37:43.092 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.093 DEBUG:boto:Data: 2023-10-17T18:37:43.093 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.093 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.094 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.094 DEBUG:boto:Params: {} 2023-10-17T18:37:43.095 DEBUG:boto:Token: None 2023-10-17T18:37:43.095 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.096 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk= 2023-10-17T18:37:43.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk=', 'Content-Length': '0'} 2023-10-17T18:37:43.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036f3eb2733d7eb95-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.099 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.099 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.100 DEBUG:boto:path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.100 DEBUG:boto:auth_path=/hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.101 DEBUG:boto:Method: GET 2023-10-17T18:37:43.101 DEBUG:boto:Path: /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.101 DEBUG:boto:Data: 2023-10-17T18:37:43.102 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.103 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.103 DEBUG:boto:Params: {} 2023-10-17T18:37:43.104 DEBUG:boto:Token: None 2023-10-17T18:37:43.104 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5?versionId=null 2023-10-17T18:37:43.105 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk= 2023-10-17T18:37:43.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7MQsrYkH1WgfIhNP69xzsXkOxPk=', 'Content-Length': '0'} 2023-10-17T18:37:43.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f778ae5a4a1e85a-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.108 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.109 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.109 DEBUG:boto:path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.109 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.110 DEBUG:boto:Method: GET 2023-10-17T18:37:43.110 DEBUG:boto:Path: /hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.111 DEBUG:boto:Data: 2023-10-17T18:37:43.111 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.112 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.112 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.112 DEBUG:boto:Params: {} 2023-10-17T18:37:43.113 DEBUG:boto:Token: None 2023-10-17T18:37:43.113 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.114 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a37b7e8775320d-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.118 DEBUG:boto:b'hbmuog-47obj51000falseobj52023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.119 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.119 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.119 DEBUG:boto:path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.120 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.120 DEBUG:boto:Method: GET 2023-10-17T18:37:43.121 DEBUG:boto:Path: /hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.121 DEBUG:boto:Data: 2023-10-17T18:37:43.122 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.122 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.123 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.123 DEBUG:boto:Params: {} 2023-10-17T18:37:43.123 DEBUG:boto:Token: None 2023-10-17T18:37:43.124 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d76e166b9b4ea96-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.128 DEBUG:boto:b'hbmuog-47obj51000falseobj52023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.128 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.129 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.129 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.130 DEBUG:boto:Path: /hbmuog-47/obj5 2023-10-17T18:37:43.130 DEBUG:boto:Data: 2023-10-17T18:37:43.131 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.131 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.132 DEBUG:boto:Params: {} 2023-10-17T18:37:43.132 DEBUG:boto:Token: None 2023-10-17T18:37:43.133 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5 2023-10-17T18:37:43.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1F7eeNfzjxDWLUbgHXFOAN2D8fE= 2023-10-17T18:37:43.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1F7eeNfzjxDWLUbgHXFOAN2D8fE=', 'Content-Length': '0'} 2023-10-17T18:37:43.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d29c18564196b506-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.136 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.136 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.137 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.137 DEBUG:boto:Path: /hbmuog-47/obj5 2023-10-17T18:37:43.137 DEBUG:boto:Data: 2023-10-17T18:37:43.138 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.139 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.139 DEBUG:boto:Params: {} 2023-10-17T18:37:43.140 DEBUG:boto:Token: None 2023-10-17T18:37:43.140 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5 2023-10-17T18:37:43.141 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1F7eeNfzjxDWLUbgHXFOAN2D8fE= 2023-10-17T18:37:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1F7eeNfzjxDWLUbgHXFOAN2D8fE=', 'Content-Length': '0'} 2023-10-17T18:37:43.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5aee6bd84e0748-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:37:43.144 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.144 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.145 DEBUG:boto:Method: GET 2023-10-17T18:37:43.145 DEBUG:boto:Path: /hbmuog-47/obj5 2023-10-17T18:37:43.146 DEBUG:boto:Data: 2023-10-17T18:37:43.146 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.147 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.147 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.148 DEBUG:boto:Params: {} 2023-10-17T18:37:43.148 DEBUG:boto:Token: None 2023-10-17T18:37:43.149 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5 2023-10-17T18:37:43.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fWgwZgynwFwIyX1oV9Ena6cphss= 2023-10-17T18:37:43.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fWgwZgynwFwIyX1oV9Ena6cphss=', 'Content-Length': '0'} 2023-10-17T18:37:43.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b438351adb7abca-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.152 DEBUG:boto:path=/hbmuog-47/obj5 2023-10-17T18:37:43.152 DEBUG:boto:auth_path=/hbmuog-47/obj5 2023-10-17T18:37:43.153 DEBUG:boto:Method: GET 2023-10-17T18:37:43.153 DEBUG:boto:Path: /hbmuog-47/obj5 2023-10-17T18:37:43.154 DEBUG:boto:Data: 2023-10-17T18:37:43.154 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.155 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.156 DEBUG:boto:Params: {} 2023-10-17T18:37:43.156 DEBUG:boto:Token: None 2023-10-17T18:37:43.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj5 2023-10-17T18:37:43.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fWgwZgynwFwIyX1oV9Ena6cphss= 2023-10-17T18:37:43.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fWgwZgynwFwIyX1oV9Ena6cphss=', 'Content-Length': '0'} 2023-10-17T18:37:43.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007244e4dfc90a8634-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.202 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.203 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.204 DEBUG:boto:path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.204 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.204 DEBUG:boto:Method: GET 2023-10-17T18:37:43.205 DEBUG:boto:Path: /hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.205 DEBUG:boto:Data: 2023-10-17T18:37:43.206 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.207 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.207 DEBUG:boto:Params: {} 2023-10-17T18:37:43.207 DEBUG:boto:Token: None 2023-10-17T18:37:43.208 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ead9e0cb610b7e0-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.213 DEBUG:boto:b'hbmuog-47obj51000falseobj52023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.213 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.214 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.214 DEBUG:boto:path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.215 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.215 DEBUG:boto:Method: GET 2023-10-17T18:37:43.215 DEBUG:boto:Path: /hbmuog-47/?prefix=obj5 2023-10-17T18:37:43.216 DEBUG:boto:Data: 2023-10-17T18:37:43.216 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.217 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.218 DEBUG:boto:Params: {} 2023-10-17T18:37:43.218 DEBUG:boto:Token: None 2023-10-17T18:37:43.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078223ec6c007bdfe-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.223 DEBUG:boto:b'hbmuog-47obj51000falseobj52023-10-17T18:37:14.678Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:43.224 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.224 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.225 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.225 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.226 DEBUG:boto:Method: GET 2023-10-17T18:37:43.226 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.227 DEBUG:boto:Data: 2023-10-17T18:37:43.227 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.228 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.228 DEBUG:boto:Params: {} 2023-10-17T18:37:43.229 DEBUG:boto:Token: None 2023-10-17T18:37:43.229 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.230 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw= 2023-10-17T18:37:43.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw=', 'Content-Length': '0'} 2023-10-17T18:37:43.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008f409757762274c-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.274 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.275 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.275 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.276 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.276 DEBUG:boto:Method: GET 2023-10-17T18:37:43.277 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.277 DEBUG:boto:Data: 2023-10-17T18:37:43.278 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.279 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.279 DEBUG:boto:Params: {} 2023-10-17T18:37:43.280 DEBUG:boto:Token: None 2023-10-17T18:37:43.280 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.281 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw= 2023-10-17T18:37:43.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw=', 'Content-Length': '0'} 2023-10-17T18:37:43.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a542d716796566b5-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.284 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.284 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.285 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.286 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.287 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.287 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.288 DEBUG:boto:Data: 2023-10-17T18:37:43.288 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.289 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.289 DEBUG:boto:Params: {} 2023-10-17T18:37:43.290 DEBUG:boto:Token: None 2023-10-17T18:37:43.290 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f4vYs3BY+mkO8fXs6GCDe4g8XRw= 2023-10-17T18:37:43.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f4vYs3BY+mkO8fXs6GCDe4g8XRw=', 'Content-Length': '0'} 2023-10-17T18:37:43.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008af01d31885b1edd-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.294 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.294 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.295 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.295 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.295 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.296 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.296 DEBUG:boto:Data: 2023-10-17T18:37:43.297 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.297 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.297 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.298 DEBUG:boto:Params: {} 2023-10-17T18:37:43.298 DEBUG:boto:Token: None 2023-10-17T18:37:43.299 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.299 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f4vYs3BY+mkO8fXs6GCDe4g8XRw= 2023-10-17T18:37:43.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f4vYs3BY+mkO8fXs6GCDe4g8XRw=', 'Content-Length': '0'} 2023-10-17T18:37:43.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a15ffb04c35a41d-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:43.303 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.304 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.304 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.305 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.305 DEBUG:boto:Method: GET 2023-10-17T18:37:43.305 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.306 DEBUG:boto:Data: 2023-10-17T18:37:43.306 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.307 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.307 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.307 DEBUG:boto:Params: {} 2023-10-17T18:37:43.308 DEBUG:boto:Token: None 2023-10-17T18:37:43.308 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.309 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw= 2023-10-17T18:37:43.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw=', 'Content-Length': '0'} 2023-10-17T18:37:43.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa42a85bd36e38c7-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.312 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.313 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.313 DEBUG:boto:path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.314 DEBUG:boto:auth_path=/hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.314 DEBUG:boto:Method: GET 2023-10-17T18:37:43.314 DEBUG:boto:Path: /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.315 DEBUG:boto:Data: 2023-10-17T18:37:43.315 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.316 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.316 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.317 DEBUG:boto:Params: {} 2023-10-17T18:37:43.317 DEBUG:boto:Token: None 2023-10-17T18:37:43.318 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6?versionId=null 2023-10-17T18:37:43.318 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw= 2023-10-17T18:37:43.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JGoeeqfbzs7E9yQW6bPbGHznngw=', 'Content-Length': '0'} 2023-10-17T18:37:43.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050618586238583c7-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.321 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.322 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.322 DEBUG:boto:path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.323 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.323 DEBUG:boto:Method: GET 2023-10-17T18:37:43.323 DEBUG:boto:Path: /hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.324 DEBUG:boto:Data: 2023-10-17T18:37:43.324 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.325 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.326 DEBUG:boto:Params: {} 2023-10-17T18:37:43.326 DEBUG:boto:Token: None 2023-10-17T18:37:43.327 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f96ab881b568c23-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.331 DEBUG:boto:b'hbmuog-47obj61000falseobj62023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.332 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.332 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.333 DEBUG:boto:path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.333 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.333 DEBUG:boto:Method: GET 2023-10-17T18:37:43.334 DEBUG:boto:Path: /hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.334 DEBUG:boto:Data: 2023-10-17T18:37:43.335 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.336 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.336 DEBUG:boto:Params: {} 2023-10-17T18:37:43.337 DEBUG:boto:Token: None 2023-10-17T18:37:43.337 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9761ea48dd6db11-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.341 DEBUG:boto:b'hbmuog-47obj61000falseobj62023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.342 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.342 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.342 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.343 DEBUG:boto:Path: /hbmuog-47/obj6 2023-10-17T18:37:43.343 DEBUG:boto:Data: 2023-10-17T18:37:43.344 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.344 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.345 DEBUG:boto:Params: {} 2023-10-17T18:37:43.345 DEBUG:boto:Token: None 2023-10-17T18:37:43.346 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6 2023-10-17T18:37:43.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SIcEn4ghZRIagwQj2QBY0j7jbVI= 2023-10-17T18:37:43.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SIcEn4ghZRIagwQj2QBY0j7jbVI=', 'Content-Length': '0'} 2023-10-17T18:37:43.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a25a6a6b32ab247-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.349 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.350 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.351 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.351 DEBUG:boto:Path: /hbmuog-47/obj6 2023-10-17T18:37:43.352 DEBUG:boto:Data: 2023-10-17T18:37:43.352 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.353 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.354 DEBUG:boto:Params: {} 2023-10-17T18:37:43.354 DEBUG:boto:Token: None 2023-10-17T18:37:43.355 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6 2023-10-17T18:37:43.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SIcEn4ghZRIagwQj2QBY0j7jbVI= 2023-10-17T18:37:43.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SIcEn4ghZRIagwQj2QBY0j7jbVI=', 'Content-Length': '0'} 2023-10-17T18:37:43.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063a6d9152f522d87-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:37:43.359 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.360 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.360 DEBUG:boto:Method: GET 2023-10-17T18:37:43.360 DEBUG:boto:Path: /hbmuog-47/obj6 2023-10-17T18:37:43.361 DEBUG:boto:Data: 2023-10-17T18:37:43.361 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.362 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.363 DEBUG:boto:Params: {} 2023-10-17T18:37:43.363 DEBUG:boto:Token: None 2023-10-17T18:37:43.364 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6 2023-10-17T18:37:43.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SwP8FuOtNnf6CqKpUzSt/LJdXcc= 2023-10-17T18:37:43.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SwP8FuOtNnf6CqKpUzSt/LJdXcc=', 'Content-Length': '0'} 2023-10-17T18:37:43.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088126741c02ab48c-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.368 DEBUG:boto:path=/hbmuog-47/obj6 2023-10-17T18:37:43.368 DEBUG:boto:auth_path=/hbmuog-47/obj6 2023-10-17T18:37:43.369 DEBUG:boto:Method: GET 2023-10-17T18:37:43.369 DEBUG:boto:Path: /hbmuog-47/obj6 2023-10-17T18:37:43.370 DEBUG:boto:Data: 2023-10-17T18:37:43.371 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.371 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.373 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.373 DEBUG:boto:Params: {} 2023-10-17T18:37:43.374 DEBUG:boto:Token: None 2023-10-17T18:37:43.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj6 2023-10-17T18:37:43.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SwP8FuOtNnf6CqKpUzSt/LJdXcc= 2023-10-17T18:37:43.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SwP8FuOtNnf6CqKpUzSt/LJdXcc=', 'Content-Length': '0'} 2023-10-17T18:37:43.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020587bf92104e573-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.379 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.379 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.380 DEBUG:boto:path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.380 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.380 DEBUG:boto:Method: GET 2023-10-17T18:37:43.381 DEBUG:boto:Path: /hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.381 DEBUG:boto:Data: 2023-10-17T18:37:43.383 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.384 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.384 DEBUG:boto:Params: {} 2023-10-17T18:37:43.385 DEBUG:boto:Token: None 2023-10-17T18:37:43.385 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.386 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080424d6b00ca7108-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.434 DEBUG:boto:b'hbmuog-47obj61000falseobj62023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.435 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.436 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.436 DEBUG:boto:path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.436 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.437 DEBUG:boto:Method: GET 2023-10-17T18:37:43.437 DEBUG:boto:Path: /hbmuog-47/?prefix=obj6 2023-10-17T18:37:43.438 DEBUG:boto:Data: 2023-10-17T18:37:43.438 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.439 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.440 DEBUG:boto:Params: {} 2023-10-17T18:37:43.440 DEBUG:boto:Token: None 2023-10-17T18:37:43.441 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611fba6a4f91cc81-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.446 DEBUG:boto:b'hbmuog-47obj61000falseobj62023-10-17T18:37:14.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:43.447 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.448 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.448 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.449 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.449 DEBUG:boto:Method: GET 2023-10-17T18:37:43.450 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.450 DEBUG:boto:Data: 2023-10-17T18:37:43.451 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.451 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.452 DEBUG:boto:Params: {} 2023-10-17T18:37:43.452 DEBUG:boto:Token: None 2023-10-17T18:37:43.453 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM= 2023-10-17T18:37:43.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM=', 'Content-Length': '0'} 2023-10-17T18:37:43.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046692956aa8e8552-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.456 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.457 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.457 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.458 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.458 DEBUG:boto:Method: GET 2023-10-17T18:37:43.458 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.459 DEBUG:boto:Data: 2023-10-17T18:37:43.459 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.460 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.460 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.461 DEBUG:boto:Params: {} 2023-10-17T18:37:43.461 DEBUG:boto:Token: None 2023-10-17T18:37:43.461 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.462 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM= 2023-10-17T18:37:43.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM=', 'Content-Length': '0'} 2023-10-17T18:37:43.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a1da321970b9835-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.465 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.466 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.466 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.466 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.467 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.467 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.468 DEBUG:boto:Data: 2023-10-17T18:37:43.468 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.469 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.469 DEBUG:boto:Params: {} 2023-10-17T18:37:43.470 DEBUG:boto:Token: None 2023-10-17T18:37:43.470 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bhr9HossBVUQnecKq0jr+oeJvf8= 2023-10-17T18:37:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr9HossBVUQnecKq0jr+oeJvf8=', 'Content-Length': '0'} 2023-10-17T18:37:43.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014d34ddc25ec6c9a-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.474 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.474 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.475 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.475 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.475 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.476 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.476 DEBUG:boto:Data: 2023-10-17T18:37:43.476 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.477 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.478 DEBUG:boto:Params: {} 2023-10-17T18:37:43.478 DEBUG:boto:Token: None 2023-10-17T18:37:43.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bhr9HossBVUQnecKq0jr+oeJvf8= 2023-10-17T18:37:43.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr9HossBVUQnecKq0jr+oeJvf8=', 'Content-Length': '0'} 2023-10-17T18:37:43.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e127321da3fd4e1c-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:43.483 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.483 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.484 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.484 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.484 DEBUG:boto:Method: GET 2023-10-17T18:37:43.485 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.485 DEBUG:boto:Data: 2023-10-17T18:37:43.486 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.487 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.487 DEBUG:boto:Params: {} 2023-10-17T18:37:43.487 DEBUG:boto:Token: None 2023-10-17T18:37:43.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM= 2023-10-17T18:37:43.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM=', 'Content-Length': '0'} 2023-10-17T18:37:43.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e64f99c14ebadfa-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.492 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.492 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.493 DEBUG:boto:path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.493 DEBUG:boto:auth_path=/hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.494 DEBUG:boto:Method: GET 2023-10-17T18:37:43.494 DEBUG:boto:Path: /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.495 DEBUG:boto:Data: 2023-10-17T18:37:43.495 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.496 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.496 DEBUG:boto:Params: {} 2023-10-17T18:37:43.497 DEBUG:boto:Token: None 2023-10-17T18:37:43.497 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7?versionId=null 2023-10-17T18:37:43.498 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM= 2023-10-17T18:37:43.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0nLid4nQyP+356Z2dOMpNAH8fM=', 'Content-Length': '0'} 2023-10-17T18:37:43.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e76f8a99fa2e9e0b-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.501 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.502 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.502 DEBUG:boto:path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.502 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.503 DEBUG:boto:Method: GET 2023-10-17T18:37:43.503 DEBUG:boto:Path: /hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.504 DEBUG:boto:Data: 2023-10-17T18:37:43.504 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.505 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.505 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.505 DEBUG:boto:Params: {} 2023-10-17T18:37:43.506 DEBUG:boto:Token: None 2023-10-17T18:37:43.506 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db196f14d9354c4c-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.510 DEBUG:boto:b'hbmuog-47obj71000falseobj72023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.511 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.512 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.512 DEBUG:boto:path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.512 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.513 DEBUG:boto:Method: GET 2023-10-17T18:37:43.513 DEBUG:boto:Path: /hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.514 DEBUG:boto:Data: 2023-10-17T18:37:43.514 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.515 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.515 DEBUG:boto:Params: {} 2023-10-17T18:37:43.516 DEBUG:boto:Token: None 2023-10-17T18:37:43.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d79664f2473a7f5-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.520 DEBUG:boto:b'hbmuog-47obj71000falseobj72023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.521 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.521 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.522 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.522 DEBUG:boto:Path: /hbmuog-47/obj7 2023-10-17T18:37:43.523 DEBUG:boto:Data: 2023-10-17T18:37:43.523 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.524 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.524 DEBUG:boto:Params: {} 2023-10-17T18:37:43.524 DEBUG:boto:Token: None 2023-10-17T18:37:43.525 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7 2023-10-17T18:37:43.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sFhPNFMIX/A15PeNSz+knEiZM1w= 2023-10-17T18:37:43.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sFhPNFMIX/A15PeNSz+knEiZM1w=', 'Content-Length': '0'} 2023-10-17T18:37:43.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f94b1a987e558b6d-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.528 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.529 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.529 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.530 DEBUG:boto:Path: /hbmuog-47/obj7 2023-10-17T18:37:43.530 DEBUG:boto:Data: 2023-10-17T18:37:43.531 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.531 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.532 DEBUG:boto:Params: {} 2023-10-17T18:37:43.532 DEBUG:boto:Token: None 2023-10-17T18:37:43.533 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7 2023-10-17T18:37:43.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sFhPNFMIX/A15PeNSz+knEiZM1w= 2023-10-17T18:37:43.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sFhPNFMIX/A15PeNSz+knEiZM1w=', 'Content-Length': '0'} 2023-10-17T18:37:43.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e007f610553b088e-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:37:43.537 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.537 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.537 DEBUG:boto:Method: GET 2023-10-17T18:37:43.538 DEBUG:boto:Path: /hbmuog-47/obj7 2023-10-17T18:37:43.538 DEBUG:boto:Data: 2023-10-17T18:37:43.539 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.540 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.540 DEBUG:boto:Params: {} 2023-10-17T18:37:43.541 DEBUG:boto:Token: None 2023-10-17T18:37:43.541 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7 2023-10-17T18:37:43.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d/aFFNUrxaNr/IfKiLEMiO+Uxnc= 2023-10-17T18:37:43.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d/aFFNUrxaNr/IfKiLEMiO+Uxnc=', 'Content-Length': '0'} 2023-10-17T18:37:43.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e6a9c7558e1bb50-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.545 DEBUG:boto:path=/hbmuog-47/obj7 2023-10-17T18:37:43.545 DEBUG:boto:auth_path=/hbmuog-47/obj7 2023-10-17T18:37:43.546 DEBUG:boto:Method: GET 2023-10-17T18:37:43.546 DEBUG:boto:Path: /hbmuog-47/obj7 2023-10-17T18:37:43.546 DEBUG:boto:Data: 2023-10-17T18:37:43.547 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.547 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.548 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.548 DEBUG:boto:Params: {} 2023-10-17T18:37:43.549 DEBUG:boto:Token: None 2023-10-17T18:37:43.549 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj7 2023-10-17T18:37:43.550 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:d/aFFNUrxaNr/IfKiLEMiO+Uxnc= 2023-10-17T18:37:43.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:d/aFFNUrxaNr/IfKiLEMiO+Uxnc=', 'Content-Length': '0'} 2023-10-17T18:37:43.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012f72742f488f778-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.598 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.599 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.599 DEBUG:boto:path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.600 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.600 DEBUG:boto:Method: GET 2023-10-17T18:37:43.600 DEBUG:boto:Path: /hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.601 DEBUG:boto:Data: 2023-10-17T18:37:43.601 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.602 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.602 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.603 DEBUG:boto:Params: {} 2023-10-17T18:37:43.603 DEBUG:boto:Token: None 2023-10-17T18:37:43.604 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.604 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8ecd59c05cc6e8-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.608 DEBUG:boto:b'hbmuog-47obj71000falseobj72023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.608 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.609 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.609 DEBUG:boto:path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.610 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.610 DEBUG:boto:Method: GET 2023-10-17T18:37:43.611 DEBUG:boto:Path: /hbmuog-47/?prefix=obj7 2023-10-17T18:37:43.611 DEBUG:boto:Data: 2023-10-17T18:37:43.611 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.612 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.613 DEBUG:boto:Params: {} 2023-10-17T18:37:43.613 DEBUG:boto:Token: None 2023-10-17T18:37:43.614 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a9fcad436fc1e5b-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.618 DEBUG:boto:b'hbmuog-47obj71000falseobj72023-10-17T18:37:14.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:43.619 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.620 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.620 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.620 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.621 DEBUG:boto:Method: GET 2023-10-17T18:37:43.621 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.622 DEBUG:boto:Data: 2023-10-17T18:37:43.622 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.623 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.623 DEBUG:boto:Params: {} 2023-10-17T18:37:43.624 DEBUG:boto:Token: None 2023-10-17T18:37:43.624 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ= 2023-10-17T18:37:43.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f91b0a41d8148ebb-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.670 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.671 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.671 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.672 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.672 DEBUG:boto:Method: GET 2023-10-17T18:37:43.673 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.673 DEBUG:boto:Data: 2023-10-17T18:37:43.673 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.674 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.675 DEBUG:boto:Params: {} 2023-10-17T18:37:43.675 DEBUG:boto:Token: None 2023-10-17T18:37:43.676 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ= 2023-10-17T18:37:43.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d42d9ad82b6eb490-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.680 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.680 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.680 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.681 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.681 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.682 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.682 DEBUG:boto:Data: 2023-10-17T18:37:43.682 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.683 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.683 DEBUG:boto:Params: {} 2023-10-17T18:37:43.684 DEBUG:boto:Token: None 2023-10-17T18:37:43.684 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pKmt6YpNwev9PL9PcyTXpk8DHcQ= 2023-10-17T18:37:43.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pKmt6YpNwev9PL9PcyTXpk8DHcQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e1fad3ceacfb1c6-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.688 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.688 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.688 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.689 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.689 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.690 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.690 DEBUG:boto:Data: 2023-10-17T18:37:43.691 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.691 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.692 DEBUG:boto:Params: {} 2023-10-17T18:37:43.692 DEBUG:boto:Token: None 2023-10-17T18:37:43.693 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pKmt6YpNwev9PL9PcyTXpk8DHcQ= 2023-10-17T18:37:43.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pKmt6YpNwev9PL9PcyTXpk8DHcQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000595a7c6321d04183-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:43.696 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.697 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.697 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.698 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.698 DEBUG:boto:Method: GET 2023-10-17T18:37:43.698 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.699 DEBUG:boto:Data: 2023-10-17T18:37:43.699 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.700 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.700 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.701 DEBUG:boto:Params: {} 2023-10-17T18:37:43.701 DEBUG:boto:Token: None 2023-10-17T18:37:43.702 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.702 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ= 2023-10-17T18:37:43.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001db3b41f8b1ed15e-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.705 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.706 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.706 DEBUG:boto:path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.706 DEBUG:boto:auth_path=/hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.707 DEBUG:boto:Method: GET 2023-10-17T18:37:43.707 DEBUG:boto:Path: /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.708 DEBUG:boto:Data: 2023-10-17T18:37:43.708 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.708 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.709 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.709 DEBUG:boto:Params: {} 2023-10-17T18:37:43.710 DEBUG:boto:Token: None 2023-10-17T18:37:43.710 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8?versionId=null 2023-10-17T18:37:43.711 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ= 2023-10-17T18:37:43.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8gFjF1OiBuES7Us0Bjj9F0GsQ=', 'Content-Length': '0'} 2023-10-17T18:37:43.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009db23f73cc0120d4-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.714 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.714 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.715 DEBUG:boto:path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.715 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.716 DEBUG:boto:Method: GET 2023-10-17T18:37:43.716 DEBUG:boto:Path: /hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.716 DEBUG:boto:Data: 2023-10-17T18:37:43.717 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.717 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.718 DEBUG:boto:Params: {} 2023-10-17T18:37:43.718 DEBUG:boto:Token: None 2023-10-17T18:37:43.719 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b8de66a93edb9d8-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.723 DEBUG:boto:b'hbmuog-47obj81000falseobj82023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.724 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.724 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.725 DEBUG:boto:path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.725 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.726 DEBUG:boto:Method: GET 2023-10-17T18:37:43.726 DEBUG:boto:Path: /hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.726 DEBUG:boto:Data: 2023-10-17T18:37:43.727 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.727 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.728 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.728 DEBUG:boto:Params: {} 2023-10-17T18:37:43.729 DEBUG:boto:Token: None 2023-10-17T18:37:43.729 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.730 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d66a375a5511254-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.733 DEBUG:boto:b'hbmuog-47obj81000falseobj82023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.734 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.735 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.735 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.735 DEBUG:boto:Path: /hbmuog-47/obj8 2023-10-17T18:37:43.736 DEBUG:boto:Data: 2023-10-17T18:37:43.736 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.737 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.737 DEBUG:boto:Params: {} 2023-10-17T18:37:43.738 DEBUG:boto:Token: None 2023-10-17T18:37:43.738 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8 2023-10-17T18:37:43.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CXmIiit+1Sr28q01POjUyxFAvhs= 2023-10-17T18:37:43.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXmIiit+1Sr28q01POjUyxFAvhs=', 'Content-Length': '0'} 2023-10-17T18:37:43.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c311bae9f5c91e38-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.742 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.742 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.743 DEBUG:boto:Method: HEAD 2023-10-17T18:37:43.743 DEBUG:boto:Path: /hbmuog-47/obj8 2023-10-17T18:37:43.744 DEBUG:boto:Data: 2023-10-17T18:37:43.744 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.745 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.745 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.745 DEBUG:boto:Params: {} 2023-10-17T18:37:43.746 DEBUG:boto:Token: None 2023-10-17T18:37:43.746 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8 2023-10-17T18:37:43.747 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CXmIiit+1Sr28q01POjUyxFAvhs= 2023-10-17T18:37:43.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXmIiit+1Sr28q01POjUyxFAvhs=', 'Content-Length': '0'} 2023-10-17T18:37:43.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c8ecafb5327cb54-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:37:43.751 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.751 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.752 DEBUG:boto:Method: GET 2023-10-17T18:37:43.752 DEBUG:boto:Path: /hbmuog-47/obj8 2023-10-17T18:37:43.753 DEBUG:boto:Data: 2023-10-17T18:37:43.753 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.753 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.754 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.754 DEBUG:boto:Params: {} 2023-10-17T18:37:43.755 DEBUG:boto:Token: None 2023-10-17T18:37:43.755 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8 2023-10-17T18:37:43.756 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f+gc6/2Yct33/UJzBAnFTeTXfgs= 2023-10-17T18:37:43.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f+gc6/2Yct33/UJzBAnFTeTXfgs=', 'Content-Length': '0'} 2023-10-17T18:37:43.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099b91e462a033d1a-00652ed477-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.802 DEBUG:boto:path=/hbmuog-47/obj8 2023-10-17T18:37:43.803 DEBUG:boto:auth_path=/hbmuog-47/obj8 2023-10-17T18:37:43.803 DEBUG:boto:Method: GET 2023-10-17T18:37:43.804 DEBUG:boto:Path: /hbmuog-47/obj8 2023-10-17T18:37:43.804 DEBUG:boto:Data: 2023-10-17T18:37:43.805 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.805 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.806 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.806 DEBUG:boto:Params: {} 2023-10-17T18:37:43.807 DEBUG:boto:Token: None 2023-10-17T18:37:43.807 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/obj8 2023-10-17T18:37:43.808 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f+gc6/2Yct33/UJzBAnFTeTXfgs= 2023-10-17T18:37:43.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f+gc6/2Yct33/UJzBAnFTeTXfgs=', 'Content-Length': '0'} 2023-10-17T18:37:43.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-47:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2f71136c93d35b9-00652ed477-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.811 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.811 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.812 DEBUG:boto:path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.812 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.813 DEBUG:boto:Method: GET 2023-10-17T18:37:43.813 DEBUG:boto:Path: /hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.814 DEBUG:boto:Data: 2023-10-17T18:37:43.814 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.815 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:43.815 DEBUG:boto:Port: 8001 2023-10-17T18:37:43.815 DEBUG:boto:Params: {} 2023-10-17T18:37:43.816 DEBUG:boto:Token: None 2023-10-17T18:37:43.816 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.817 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be26e5aa5ec77305-00652ed477-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.820 DEBUG:boto:b'hbmuog-47obj81000falseobj82023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.821 DEBUG:boto:path=/hbmuog-47/ 2023-10-17T18:37:43.822 DEBUG:boto:auth_path=/hbmuog-47/ 2023-10-17T18:37:43.822 DEBUG:boto:path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.822 DEBUG:boto:auth_path=/hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.823 DEBUG:boto:Method: GET 2023-10-17T18:37:43.823 DEBUG:boto:Path: /hbmuog-47/?prefix=obj8 2023-10-17T18:37:43.824 DEBUG:boto:Data: 2023-10-17T18:37:43.824 DEBUG:boto:Headers: {} 2023-10-17T18:37:43.824 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:43.825 DEBUG:boto:Port: 8000 2023-10-17T18:37:43.825 DEBUG:boto:Params: {} 2023-10-17T18:37:43.826 DEBUG:boto:Token: None 2023-10-17T18:37:43.826 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:37:43 GMT /hbmuog-47/ 2023-10-17T18:37:43.827 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc= 2023-10-17T18:37:43.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s3nneA26OqsLAzRHHAmgQkqFxlc=', 'Content-Length': '0'} 2023-10-17T18:37:43.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e2262038e192430-00652ed477-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:37:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:43.874 DEBUG:boto:b'hbmuog-47obj81000falseobj82023-10-17T18:37:14.788Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:37:43.875 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-47 zones={a2, a1} 2023-10-17T18:37:43.876 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T18:37:43.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:43.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:43.877 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:43.936 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:43.936 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:43.988 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:37:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-10-17T18:37:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:14.485028Z" 2023-10-17T18:37:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-17T18:37:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:28.573589Z" 2023-10-17T18:37:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-17T18:37:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:28.653227Z" 2023-10-17T18:37:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-17T18:37:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:28.497494Z" 2023-10-17T18:37:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:43.992 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:37:43.992 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000006987'} 2023-10-17T18:37:43.993 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:37:43.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:43.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:43.994 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:44.050 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.050 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:44.096 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:37:44.097 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:37:44.097 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:37:44.097 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:37:44.097 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:37:44.098 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:37:44.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.098 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:37:44.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.099 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:37:44.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.099 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000005159", 2023-10-17T18:37:44.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.100 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.100 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:14.485028Z" 2023-10-17T18:37:44.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:37:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-17T18:37:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:28.573589Z" 2023-10-17T18:37:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:37:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-17T18:37:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:28.653227Z" 2023-10-17T18:37:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:37:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-17T18:37:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:28.497494Z" 2023-10-17T18:37:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:37:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:37:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:37:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:37:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:37:44Z" 2023-10-17T18:37:44.109 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:37:44.110 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005159", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:14.485028Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:28.573589Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:28.653227Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006987", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:28.497494Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:37:44Z" } 2023-10-17T18:37:44.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005159', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:14.485028Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:28.573589Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:28.653227Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006987', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:28.497494Z'}}] 2023-10-17T18:37:44.111 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000006987'} 2023-10-17T18:37:44.112 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005159', 1: '00000000000000000000:00000000000000004316', 2: '00000000000000000000:00000000000000002903', 3: '00000000000000000000:00000000000000006987'} 2023-10-17T18:37:44.112 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:37:44.112 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T18:37:44.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.114 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:44.163 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.164 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:44.211 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:37:44.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.212 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002483", 2023-10-17T18:37:44.212 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:33:36.965951Z" 2023-10-17T18:37:44.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.213 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-17T18:37:44.213 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:19.683647Z" 2023-10-17T18:37:44.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.213 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-10-17T18:37:44.214 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:18.618616Z" 2023-10-17T18:37:44.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.214 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-10-17T18:37:44.215 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:18.581236Z" 2023-10-17T18:37:44.215 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.215 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:37:44.216 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006141'} 2023-10-17T18:37:44.216 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:37:44.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.218 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:44.276 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.277 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:44.319 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:37:44.319 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:37:44.319 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:37:44.320 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:37:44.320 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:37:44.320 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:37:44.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.321 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002483", 2023-10-17T18:37:44.322 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.322 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.322 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:33:36.965951Z" 2023-10-17T18:37:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:37:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.324 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.324 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-17T18:37:44.324 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.324 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.325 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:19.683647Z" 2023-10-17T18:37:44.325 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.325 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:37:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-10-17T18:37:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.327 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:18.618616Z" 2023-10-17T18:37:44.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.328 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:37:44.328 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.328 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:37:44.328 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-10-17T18:37:44.328 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.329 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.329 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:18.581236Z" 2023-10-17T18:37:44.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.330 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:37:44.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.330 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:37:44.330 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:37:44.331 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:37:44.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.331 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:37:44Z" 2023-10-17T18:37:44.331 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:37:44.332 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002483", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:33:36.965951Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:19.683647Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:18.618616Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:37:18.581236Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:37:44Z" } 2023-10-17T18:37:44.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002483', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:33:36.965951Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:19.683647Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:18.618616Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:37:18.581236Z'}}] 2023-10-17T18:37:44.333 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006141'} 2023-10-17T18:37:44.333 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002483', 1: '00000000000000000000:00000000000000002620', 2: '00000000000000000000:00000000000000001775', 3: '00000000000000000000:00000000000000006141'} 2023-10-17T18:37:44.334 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:37:44.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-10-17T18:37:44.336 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-48 2023-10-17T18:37:44.336 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:37:44.337 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:37:44.337 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.338 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:37:44.338 DEBUG:boto:Data: 2023-10-17T18:37:44.338 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.339 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.340 DEBUG:boto:Params: {} 2023-10-17T18:37:44.340 DEBUG:boto:Token: None 2023-10-17T18:37:44.341 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/ 2023-10-17T18:37:44.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oIqVjmR8ekB7krJbtoBOhjCoR04= 2023-10-17T18:37:44.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oIqVjmR8ekB7krJbtoBOhjCoR04=', 'Content-Length': '0'} 2023-10-17T18:37:44.356 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055eb5f2cc7bee8bc-00652ed478-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.357 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-49 2023-10-17T18:37:44.357 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:37:44.357 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:37:44.358 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.358 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:37:44.359 DEBUG:boto:Data: 2023-10-17T18:37:44.359 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.359 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.360 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.360 DEBUG:boto:Params: {} 2023-10-17T18:37:44.361 DEBUG:boto:Token: None 2023-10-17T18:37:44.361 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/ 2023-10-17T18:37:44.362 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:odeUVKPoieBVA0/R83DHkV/SPmc= 2023-10-17T18:37:44.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:odeUVKPoieBVA0/R83DHkV/SPmc=', 'Content-Length': '0'} 2023-10-17T18:37:44.384 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000833dc7150cbd9d3b-00652ed478-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.385 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:37:44.385 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:37:44.386 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.386 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:37:44.386 DEBUG:boto:Data: 2023-10-17T18:37:44.387 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.387 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.388 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.388 DEBUG:boto:Params: {} 2023-10-17T18:37:44.388 DEBUG:boto:Token: None 2023-10-17T18:37:44.389 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/ 2023-10-17T18:37:44.389 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs= 2023-10-17T18:37:44.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs=', 'Content-Length': '0'} 2023-10-17T18:37:44.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d987e3476abb575-00652ed478-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.393 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:37:44.393 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:37:44.394 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.394 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:37:44.395 DEBUG:boto:Data: 2023-10-17T18:37:44.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.395 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.396 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.396 DEBUG:boto:Params: {} 2023-10-17T18:37:44.397 DEBUG:boto:Token: None 2023-10-17T18:37:44.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/obj1 2023-10-17T18:37:44.397 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eCxPhz1ZwMxR8lyWXF6VAKvzY7E= 2023-10-17T18:37:44.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eCxPhz1ZwMxR8lyWXF6VAKvzY7E='} 2023-10-17T18:37:44.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000db46038c3610989-00652ed478-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.404 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:37:44.404 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:37:44.405 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.405 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:37:44.405 DEBUG:boto:Data: 2023-10-17T18:37:44.406 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.407 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.407 DEBUG:boto:Params: {} 2023-10-17T18:37:44.408 DEBUG:boto:Token: None 2023-10-17T18:37:44.408 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/ 2023-10-17T18:37:44.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs= 2023-10-17T18:37:44.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs=', 'Content-Length': '0'} 2023-10-17T18:37:44.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b941d1a83b5b991-00652ed478-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.412 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:37:44.412 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:37:44.413 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.413 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:37:44.413 DEBUG:boto:Data: 2023-10-17T18:37:44.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.415 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.415 DEBUG:boto:Params: {} 2023-10-17T18:37:44.415 DEBUG:boto:Token: None 2023-10-17T18:37:44.416 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/obj2 2023-10-17T18:37:44.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jheiLRnWjjsBPTq544HxsI2CH+o= 2023-10-17T18:37:44.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jheiLRnWjjsBPTq544HxsI2CH+o='} 2023-10-17T18:37:44.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9791bb4f63362f0-00652ed478-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.459 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:37:44.460 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:37:44.460 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.460 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:37:44.461 DEBUG:boto:Data: 2023-10-17T18:37:44.461 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.462 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.462 DEBUG:boto:Params: {} 2023-10-17T18:37:44.463 DEBUG:boto:Token: None 2023-10-17T18:37:44.463 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/ 2023-10-17T18:37:44.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs= 2023-10-17T18:37:44.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs=', 'Content-Length': '0'} 2023-10-17T18:37:44.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070847eca533001b1-00652ed478-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.467 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:37:44.468 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:37:44.468 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.469 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:37:44.469 DEBUG:boto:Data: 2023-10-17T18:37:44.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.470 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.470 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.471 DEBUG:boto:Params: {} 2023-10-17T18:37:44.471 DEBUG:boto:Token: None 2023-10-17T18:37:44.472 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/obj3 2023-10-17T18:37:44.472 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/3P0iABo/f7J4H8TZKg4MNogFnU= 2023-10-17T18:37:44.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3P0iABo/f7J4H8TZKg4MNogFnU='} 2023-10-17T18:37:44.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066368de5766325ed-00652ed478-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.478 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:37:44.478 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:37:44.479 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.479 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:37:44.480 DEBUG:boto:Data: 2023-10-17T18:37:44.480 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.481 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.481 DEBUG:boto:Params: {} 2023-10-17T18:37:44.482 DEBUG:boto:Token: None 2023-10-17T18:37:44.482 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/ 2023-10-17T18:37:44.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs= 2023-10-17T18:37:44.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9I2h0ChK+NfwExz1l03UeZbIBqs=', 'Content-Length': '0'} 2023-10-17T18:37:44.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f681c7d7aad28b3-00652ed478-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.486 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:37:44.486 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:37:44.487 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.487 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:37:44.487 DEBUG:boto:Data: 2023-10-17T18:37:44.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.488 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:37:44.489 DEBUG:boto:Port: 8000 2023-10-17T18:37:44.489 DEBUG:boto:Params: {} 2023-10-17T18:37:44.489 DEBUG:boto:Token: None 2023-10-17T18:37:44.490 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-48/obj4 2023-10-17T18:37:44.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UGqP6EXIiIc3hicNVyxopi3MSpQ= 2023-10-17T18:37:44.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UGqP6EXIiIc3hicNVyxopi3MSpQ='} 2023-10-17T18:37:44.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ea5a8f361583471-00652ed478-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.535 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:37:44.536 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:37:44.536 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.536 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:37:44.537 DEBUG:boto:Data: 2023-10-17T18:37:44.537 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.538 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.538 DEBUG:boto:Params: {} 2023-10-17T18:37:44.539 DEBUG:boto:Token: None 2023-10-17T18:37:44.539 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/ 2023-10-17T18:37:44.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A= 2023-10-17T18:37:44.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A=', 'Content-Length': '0'} 2023-10-17T18:37:44.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050293a4444bd3e1f-00652ed478-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.543 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:37:44.544 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:37:44.544 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.545 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:37:44.545 DEBUG:boto:Data: 2023-10-17T18:37:44.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.546 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.546 DEBUG:boto:Params: {} 2023-10-17T18:37:44.547 DEBUG:boto:Token: None 2023-10-17T18:37:44.547 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/obj1 2023-10-17T18:37:44.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fjLogSL1JyL6MYXfl/GQuX9CjBA= 2023-10-17T18:37:44.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fjLogSL1JyL6MYXfl/GQuX9CjBA='} 2023-10-17T18:37:44.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c5851ee23757ba3-00652ed478-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.555 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:37:44.555 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:37:44.555 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.556 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:37:44.556 DEBUG:boto:Data: 2023-10-17T18:37:44.557 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.557 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.558 DEBUG:boto:Params: {} 2023-10-17T18:37:44.558 DEBUG:boto:Token: None 2023-10-17T18:37:44.559 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/ 2023-10-17T18:37:44.559 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A= 2023-10-17T18:37:44.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A=', 'Content-Length': '0'} 2023-10-17T18:37:44.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e0676fc5cc1ea86-00652ed478-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.563 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:37:44.563 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:37:44.564 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.564 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:37:44.564 DEBUG:boto:Data: 2023-10-17T18:37:44.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.565 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.566 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.566 DEBUG:boto:Params: {} 2023-10-17T18:37:44.566 DEBUG:boto:Token: None 2023-10-17T18:37:44.567 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/obj2 2023-10-17T18:37:44.567 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j240QzcSAU4khYcWHN2bR/qzIWE= 2023-10-17T18:37:44.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j240QzcSAU4khYcWHN2bR/qzIWE='} 2023-10-17T18:37:44.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb38d4d5335a9a0a-00652ed478-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.611 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:37:44.612 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:37:44.612 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.612 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:37:44.613 DEBUG:boto:Data: 2023-10-17T18:37:44.613 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.614 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.615 DEBUG:boto:Params: {} 2023-10-17T18:37:44.615 DEBUG:boto:Token: None 2023-10-17T18:37:44.615 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/ 2023-10-17T18:37:44.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A= 2023-10-17T18:37:44.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A=', 'Content-Length': '0'} 2023-10-17T18:37:44.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a68c939e1448e28-00652ed478-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.621 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:37:44.621 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:37:44.622 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.622 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:37:44.623 DEBUG:boto:Data: 2023-10-17T18:37:44.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.624 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.624 DEBUG:boto:Params: {} 2023-10-17T18:37:44.625 DEBUG:boto:Token: None 2023-10-17T18:37:44.625 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/obj3 2023-10-17T18:37:44.626 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1XV20EnvMUmEqVBh/CVP5jX2odk= 2023-10-17T18:37:44.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1XV20EnvMUmEqVBh/CVP5jX2odk='} 2023-10-17T18:37:44.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000036c64e974fc00f-00652ed478-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.632 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:37:44.632 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:37:44.633 DEBUG:boto:Method: HEAD 2023-10-17T18:37:44.633 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:37:44.634 DEBUG:boto:Data: 2023-10-17T18:37:44.634 DEBUG:boto:Headers: {} 2023-10-17T18:37:44.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.635 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.635 DEBUG:boto:Params: {} 2023-10-17T18:37:44.636 DEBUG:boto:Token: None 2023-10-17T18:37:44.636 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/ 2023-10-17T18:37:44.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A= 2023-10-17T18:37:44.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mwvU4d+ofzHb1BoazT5OXOKmU0A=', 'Content-Length': '0'} 2023-10-17T18:37:44.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcc6f3099fe3d1d7-00652ed478-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.640 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:37:44.641 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:37:44.641 DEBUG:boto:Method: PUT 2023-10-17T18:37:44.641 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:37:44.642 DEBUG:boto:Data: 2023-10-17T18:37:44.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:37:44.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:37:44.643 DEBUG:boto:Port: 8001 2023-10-17T18:37:44.644 DEBUG:boto:Params: {} 2023-10-17T18:37:44.644 DEBUG:boto:Token: None 2023-10-17T18:37:44.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:37:44 GMT /hbmuog-49/obj4 2023-10-17T18:37:44.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vSOlVIfxAV2ll3ZNRvHW8/2QifU= 2023-10-17T18:37:44.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:37:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vSOlVIfxAV2ll3ZNRvHW8/2QifU='} 2023-10-17T18:37:44.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069456a77f38de2e9-00652ed478-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:37:44.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.697 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:44.746 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.746 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:44.776 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:37:44.788 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:37:44.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.789 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567864.375672_558.1", 2023-10-17T18:37:44.789 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:37:44.375672Z" 2023-10-17T18:37:44.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.790 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:37:44.790 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:37:44.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.790 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:37:44.791 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:37:44.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.791 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:37:44.792 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:37:44.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.792 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:37:44.793 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697567864.375672_558.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:37:44.794 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:37:44.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:37:44.795 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:37:44.846 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.846 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:44.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:44.876+0000 7f6c2a6d7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:37:44.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:44.877+0000 7f6c2a6d7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:37:44.890 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:37:44.891 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:37:44.891 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:37:44.891 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:37:44.892 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:37:44.892 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:37:44.892 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:37:44.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.892 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:37:44.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.893 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:37:44.893 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.893 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:37:44.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567864.375672_558.1", 2023-10-17T18:37:44.894 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.894 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:37:44.894 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.894 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:44.375672Z", 2023-10-17T18:37:44.895 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:37:44.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.895 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:37:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:37:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:37:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.896 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.897 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.897 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:37:44.897 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:37:44.897 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.898 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:37:44.898 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.898 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:37:44.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:37:44.899 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.899 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.899 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.899 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:37:44.900 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:37:44.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:37:44.901 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:37:44.901 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:37:44.901 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:37:44.901 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:37:44.901 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:37:44.902 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:37:44.902 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:37:44.902 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:37:44.902 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:37:44.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:37:44.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:37:44.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.903 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:37:44.904 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:37:44.904 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:37:44.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:37:44.904 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:37:44Z" 2023-10-17T18:37:44.904 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:37:44.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567864.375672_558.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:37:44.375672Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:37:44Z" } 2023-10-17T18:37:44.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567864.375672_558.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:37:44.375672Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:37:44.906 DEBUG:rgw_multi.tests:log_status={0: '1_1697567864.375672_558.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:37:44.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567864.375672_558.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:37:44.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:37:44.908 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-48 2023-10-17T18:37:44.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:44.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:37:44.909 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:37:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:37:44.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:37:45.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:45.000+0000 7f93434c3780 1 waiting to reach incremental sync.. 2023-10-17T18:37:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:50.001+0000 7f93434c3780 1 waiting to reach incremental sync.. 2023-10-17T18:37:55.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:37:55.002+0000 7f93434c3780 1 waiting to reach incremental sync.. 2023-10-17T18:38:00.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:00.003+0000 7f93434c3780 1 bucket sync caught up with source: 2023-10-17T18:38:00.005 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.23.6, , 00000000001.6.6, , 00000000001.32.6, , , 00000000001.30.6, , , ] 2023-10-17T18:38:00.005 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.23.6, , 00000000001.6.6, , 00000000001.32.6, , , 00000000001.30.6, , , ] 2023-10-17T18:38:00.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:00.003+0000 7f93434c3780 0 bucket checkpoint complete 2023-10-17T18:38:00.022 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-48 2023-10-17T18:38:00.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:38:00.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:38:00.023 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:38:00.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:38:00.088 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:38:00.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:00.126+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:05.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:05.127+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:10.129+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:15.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:15.129+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:20.129+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:25.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:25.129+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:30.132 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:30.130+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:35.132 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:35.131+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:40.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:40.132+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:45.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:45.133+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:50.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:50.134+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:38:55.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:38:55.134+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:00.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:00.134+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:05.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:05.135+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:10.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:10.135+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:15.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:15.136+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:20.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:20.138+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:25.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:25.138+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:30.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:30.139+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:35.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:35.138+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:40.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:40.139+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:45.140+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:50.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:50.140+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:39:55.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:39:55.141+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:00.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:00.143+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:05.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:05.143+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:10.143+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:15.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:15.143+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:20.144+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:25.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:25.145+0000 7fbfa0d5e780 1 waiting to reach incremental sync.. 2023-10-17T18:40:30.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:30.147+0000 7fbfa0d5e780 1 bucket sync caught up with source: 2023-10-17T18:40:30.148 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.198.6, , 00000000001.11.6, , 00000000001.191.6, , , 00000000001.264.6, , , ] 2023-10-17T18:40:30.149 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.198.6, , 00000000001.11.6, , 00000000001.191.6, , , 00000000001.264.6, , , ] 2023-10-17T18:40:30.149 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:30.147+0000 7fbfa0d5e780 0 bucket checkpoint complete 2023-10-17T18:40:30.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-48 zones={a2, a1} 2023-10-17T18:40:30.165 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.166 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.166 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.167 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:30.167 DEBUG:boto:Data: 2023-10-17T18:40:30.167 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.168 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.168 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.169 DEBUG:boto:Params: {} 2023-10-17T18:40:30.169 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:40:30.170 DEBUG:boto:Token: None 2023-10-17T18:40:30.171 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SIqEGb/tSNLrWjbQg+Auq1Akfl0= 2023-10-17T18:40:30.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SIqEGb/tSNLrWjbQg+Auq1Akfl0=', 'Content-Length': '0'} 2023-10-17T18:40:30.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003feccc83a71b6f06-00652ed51e-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.177 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.177 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.178 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.178 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:30.178 DEBUG:boto:Data: 2023-10-17T18:40:30.179 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.179 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.180 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.180 DEBUG:boto:Params: {} 2023-10-17T18:40:30.181 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:40:30.181 DEBUG:boto:Token: None 2023-10-17T18:40:30.182 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.182 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SIqEGb/tSNLrWjbQg+Auq1Akfl0= 2023-10-17T18:40:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SIqEGb/tSNLrWjbQg+Auq1Akfl0=', 'Content-Length': '0'} 2023-10-17T18:40:30.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007730c68269afebb9-00652ed51e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:40:30.188 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.188 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.189 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.189 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.189 DEBUG:boto:Method: GET 2023-10-17T18:40:30.190 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.190 DEBUG:boto:Data: 2023-10-17T18:40:30.191 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.191 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.192 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.192 DEBUG:boto:Params: {} 2023-10-17T18:40:30.193 DEBUG:boto:Token: None 2023-10-17T18:40:30.193 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/?versions 2023-10-17T18:40:30.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png= 2023-10-17T18:40:30.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png=', 'Content-Length': '0'} 2023-10-17T18:40:30.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274df61fc17bef37-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.197 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:40:30.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:40:30.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:40:30.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:40:30.200 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:40:30.201 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.201 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.201 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.202 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.202 DEBUG:boto:Method: GET 2023-10-17T18:40:30.203 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.203 DEBUG:boto:Data: 2023-10-17T18:40:30.204 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.204 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.204 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.205 DEBUG:boto:Params: {} 2023-10-17T18:40:30.205 DEBUG:boto:Token: None 2023-10-17T18:40:30.206 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/?versions 2023-10-17T18:40:30.206 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png= 2023-10-17T18:40:30.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png=', 'Content-Length': '0'} 2023-10-17T18:40:30.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea19a042486ddb56-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.210 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:40:30.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:40:30.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:40:30.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:40:30.212 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.213 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.213 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.214 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.214 DEBUG:boto:Method: GET 2023-10-17T18:40:30.215 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.215 DEBUG:boto:Data: 2023-10-17T18:40:30.215 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.216 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.216 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.217 DEBUG:boto:Params: {} 2023-10-17T18:40:30.217 DEBUG:boto:Token: None 2023-10-17T18:40:30.218 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/?versions 2023-10-17T18:40:30.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png= 2023-10-17T18:40:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png=', 'Content-Length': '0'} 2023-10-17T18:40:30.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f6ff3776013bbf-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.221 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.222 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.223 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.223 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.224 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.224 DEBUG:boto:Method: GET 2023-10-17T18:40:30.225 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:40:30.225 DEBUG:boto:Data: 2023-10-17T18:40:30.225 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.226 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.227 DEBUG:boto:Params: {} 2023-10-17T18:40:30.227 DEBUG:boto:Token: None 2023-10-17T18:40:30.228 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/?versions 2023-10-17T18:40:30.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png= 2023-10-17T18:40:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hh/ctOjIMVi3q8Vvn9u09lU2png=', 'Content-Length': '0'} 2023-10-17T18:40:30.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094321dc17bb901fc-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.231 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:30.233 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.233 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.234 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.234 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.234 DEBUG:boto:Method: GET 2023-10-17T18:40:30.235 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.235 DEBUG:boto:Data: 2023-10-17T18:40:30.236 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.236 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.237 DEBUG:boto:Params: {} 2023-10-17T18:40:30.237 DEBUG:boto:Token: None 2023-10-17T18:40:30.238 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8= 2023-10-17T18:40:30.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8=', 'Content-Length': '0'} 2023-10-17T18:40:30.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099b36861bc48bb87-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.241 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.242 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.242 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.243 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.243 DEBUG:boto:Method: GET 2023-10-17T18:40:30.244 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.244 DEBUG:boto:Data: 2023-10-17T18:40:30.245 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.245 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.246 DEBUG:boto:Params: {} 2023-10-17T18:40:30.246 DEBUG:boto:Token: None 2023-10-17T18:40:30.247 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8= 2023-10-17T18:40:30.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8=', 'Content-Length': '0'} 2023-10-17T18:40:30.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfc597fe0be7fcf7-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.250 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.251 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.251 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.251 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.252 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.252 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.252 DEBUG:boto:Data: 2023-10-17T18:40:30.253 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.253 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.254 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.254 DEBUG:boto:Params: {} 2023-10-17T18:40:30.255 DEBUG:boto:Token: None 2023-10-17T18:40:30.255 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.256 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUeUYyUb7fPK8+Uxmfk4i/+obPo= 2023-10-17T18:40:30.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUeUYyUb7fPK8+Uxmfk4i/+obPo=', 'Content-Length': '0'} 2023-10-17T18:40:30.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011302e0d1f7e4a09-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.259 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.259 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.259 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.260 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.260 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.261 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.261 DEBUG:boto:Data: 2023-10-17T18:40:30.261 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.262 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.262 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.263 DEBUG:boto:Params: {} 2023-10-17T18:40:30.263 DEBUG:boto:Token: None 2023-10-17T18:40:30.263 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.264 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUeUYyUb7fPK8+Uxmfk4i/+obPo= 2023-10-17T18:40:30.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUeUYyUb7fPK8+Uxmfk4i/+obPo=', 'Content-Length': '0'} 2023-10-17T18:40:30.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000094b2c6fa77831e9-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:30.267 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.268 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.268 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.269 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.269 DEBUG:boto:Method: GET 2023-10-17T18:40:30.270 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.270 DEBUG:boto:Data: 2023-10-17T18:40:30.271 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.271 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.271 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.272 DEBUG:boto:Params: {} 2023-10-17T18:40:30.272 DEBUG:boto:Token: None 2023-10-17T18:40:30.273 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8= 2023-10-17T18:40:30.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8=', 'Content-Length': '0'} 2023-10-17T18:40:30.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032aae88c1a6793bc-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.276 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.277 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.277 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.278 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.278 DEBUG:boto:Method: GET 2023-10-17T18:40:30.279 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.279 DEBUG:boto:Data: 2023-10-17T18:40:30.280 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.280 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.280 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.281 DEBUG:boto:Params: {} 2023-10-17T18:40:30.281 DEBUG:boto:Token: None 2023-10-17T18:40:30.282 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:40:30.282 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8= 2023-10-17T18:40:30.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H7F8P3aLTG9YzQ9Tvr0vVzYZ6W8=', 'Content-Length': '0'} 2023-10-17T18:40:30.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc7fbe351b590a4-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.326 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.327 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.327 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.328 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.328 DEBUG:boto:Method: GET 2023-10-17T18:40:30.329 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.329 DEBUG:boto:Data: 2023-10-17T18:40:30.330 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.330 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.331 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.331 DEBUG:boto:Params: {} 2023-10-17T18:40:30.331 DEBUG:boto:Token: None 2023-10-17T18:40:30.332 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.332 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab55d050f06a6759-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.336 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.337 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.337 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.338 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.338 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.338 DEBUG:boto:Method: GET 2023-10-17T18:40:30.339 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.339 DEBUG:boto:Data: 2023-10-17T18:40:30.340 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.340 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.340 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.341 DEBUG:boto:Params: {} 2023-10-17T18:40:30.341 DEBUG:boto:Token: None 2023-10-17T18:40:30.342 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.342 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c98d26477fe1b6-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.346 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.347 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.347 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.348 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.348 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:40:30.348 DEBUG:boto:Data: 2023-10-17T18:40:30.349 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.349 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.350 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.350 DEBUG:boto:Params: {} 2023-10-17T18:40:30.351 DEBUG:boto:Token: None 2023-10-17T18:40:30.351 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1 2023-10-17T18:40:30.352 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiXdGgeRFseCSy1ct8o/t+ubHV4= 2023-10-17T18:40:30.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiXdGgeRFseCSy1ct8o/t+ubHV4=', 'Content-Length': '0'} 2023-10-17T18:40:30.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea12a90235e4a2cf-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.354 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.355 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.355 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.356 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:40:30.356 DEBUG:boto:Data: 2023-10-17T18:40:30.356 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.357 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.358 DEBUG:boto:Params: {} 2023-10-17T18:40:30.358 DEBUG:boto:Token: None 2023-10-17T18:40:30.358 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1 2023-10-17T18:40:30.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oiXdGgeRFseCSy1ct8o/t+ubHV4= 2023-10-17T18:40:30.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oiXdGgeRFseCSy1ct8o/t+ubHV4=', 'Content-Length': '0'} 2023-10-17T18:40:30.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082628ae09d8f57ad-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:30.362 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.363 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.363 DEBUG:boto:Method: GET 2023-10-17T18:40:30.364 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:40:30.364 DEBUG:boto:Data: 2023-10-17T18:40:30.364 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.365 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.366 DEBUG:boto:Params: {} 2023-10-17T18:40:30.366 DEBUG:boto:Token: None 2023-10-17T18:40:30.367 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1 2023-10-17T18:40:30.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Up29Bxt7Ei+Faw3QhfJ9G5lAzBU= 2023-10-17T18:40:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Up29Bxt7Ei+Faw3QhfJ9G5lAzBU=', 'Content-Length': '0'} 2023-10-17T18:40:30.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033f7f899e9e5631e-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.370 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:40:30.371 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:40:30.371 DEBUG:boto:Method: GET 2023-10-17T18:40:30.372 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:40:30.372 DEBUG:boto:Data: 2023-10-17T18:40:30.372 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.373 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.373 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.374 DEBUG:boto:Params: {} 2023-10-17T18:40:30.374 DEBUG:boto:Token: None 2023-10-17T18:40:30.375 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj1 2023-10-17T18:40:30.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Up29Bxt7Ei+Faw3QhfJ9G5lAzBU= 2023-10-17T18:40:30.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Up29Bxt7Ei+Faw3QhfJ9G5lAzBU=', 'Content-Length': '0'} 2023-10-17T18:40:30.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eafc90fcab36e1a8-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.378 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.379 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.379 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.380 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.380 DEBUG:boto:Method: GET 2023-10-17T18:40:30.381 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.381 DEBUG:boto:Data: 2023-10-17T18:40:30.381 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.382 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.382 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.383 DEBUG:boto:Params: {} 2023-10-17T18:40:30.383 DEBUG:boto:Token: None 2023-10-17T18:40:30.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.384 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003aff4ac1b12ceb-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.387 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.388 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.389 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.389 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.389 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.390 DEBUG:boto:Method: GET 2023-10-17T18:40:30.390 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:40:30.391 DEBUG:boto:Data: 2023-10-17T18:40:30.391 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.392 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.392 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.393 DEBUG:boto:Params: {} 2023-10-17T18:40:30.393 DEBUG:boto:Token: None 2023-10-17T18:40:30.394 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.394 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e69fed8e80d0629-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.398 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:30.399 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.400 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.400 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.400 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.401 DEBUG:boto:Method: GET 2023-10-17T18:40:30.401 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.402 DEBUG:boto:Data: 2023-10-17T18:40:30.402 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.403 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.403 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.404 DEBUG:boto:Params: {} 2023-10-17T18:40:30.404 DEBUG:boto:Token: None 2023-10-17T18:40:30.405 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA= 2023-10-17T18:40:30.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA=', 'Content-Length': '0'} 2023-10-17T18:40:30.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049c4b851fd308ff7-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.408 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.409 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.409 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.410 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.410 DEBUG:boto:Method: GET 2023-10-17T18:40:30.411 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.411 DEBUG:boto:Data: 2023-10-17T18:40:30.411 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.413 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.414 DEBUG:boto:Params: {} 2023-10-17T18:40:30.414 DEBUG:boto:Token: None 2023-10-17T18:40:30.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA= 2023-10-17T18:40:30.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA=', 'Content-Length': '0'} 2023-10-17T18:40:30.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009340b054aaf0988-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.418 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.419 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.419 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.420 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.420 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.420 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.421 DEBUG:boto:Data: 2023-10-17T18:40:30.421 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.422 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.423 DEBUG:boto:Params: {} 2023-10-17T18:40:30.423 DEBUG:boto:Token: None 2023-10-17T18:40:30.423 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.424 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SisnSGlR87VFTkmpgaVIY+tHLl8= 2023-10-17T18:40:30.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SisnSGlR87VFTkmpgaVIY+tHLl8=', 'Content-Length': '0'} 2023-10-17T18:40:30.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cea86f39edba6e8-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.427 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.427 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.428 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.428 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.429 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.429 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.429 DEBUG:boto:Data: 2023-10-17T18:40:30.430 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.431 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.431 DEBUG:boto:Params: {} 2023-10-17T18:40:30.432 DEBUG:boto:Token: None 2023-10-17T18:40:30.432 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.433 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SisnSGlR87VFTkmpgaVIY+tHLl8= 2023-10-17T18:40:30.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SisnSGlR87VFTkmpgaVIY+tHLl8=', 'Content-Length': '0'} 2023-10-17T18:40:30.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003676f5531ded1e27-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:30.436 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.437 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.437 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.438 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.438 DEBUG:boto:Method: GET 2023-10-17T18:40:30.439 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.439 DEBUG:boto:Data: 2023-10-17T18:40:30.439 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.440 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.440 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.441 DEBUG:boto:Params: {} 2023-10-17T18:40:30.441 DEBUG:boto:Token: None 2023-10-17T18:40:30.442 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA= 2023-10-17T18:40:30.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA=', 'Content-Length': '0'} 2023-10-17T18:40:30.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000379699522200948-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.486 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.487 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.488 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.488 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.488 DEBUG:boto:Method: GET 2023-10-17T18:40:30.489 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.489 DEBUG:boto:Data: 2023-10-17T18:40:30.490 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.491 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.491 DEBUG:boto:Params: {} 2023-10-17T18:40:30.492 DEBUG:boto:Token: None 2023-10-17T18:40:30.492 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:40:30.493 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA= 2023-10-17T18:40:30.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zIwXV1wwbu5qb8r3QcmN3W+1zgA=', 'Content-Length': '0'} 2023-10-17T18:40:30.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e325218f28eb9b76-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.496 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.497 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.497 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.498 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.498 DEBUG:boto:Method: GET 2023-10-17T18:40:30.499 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.499 DEBUG:boto:Data: 2023-10-17T18:40:30.500 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.500 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.501 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.501 DEBUG:boto:Params: {} 2023-10-17T18:40:30.502 DEBUG:boto:Token: None 2023-10-17T18:40:30.502 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8de4026ab1651af-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.506 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.507 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.508 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.508 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.509 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.509 DEBUG:boto:Method: GET 2023-10-17T18:40:30.509 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.510 DEBUG:boto:Data: 2023-10-17T18:40:30.510 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.511 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.511 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.512 DEBUG:boto:Params: {} 2023-10-17T18:40:30.512 DEBUG:boto:Token: None 2023-10-17T18:40:30.513 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.513 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051af4c44d0b4b554-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.517 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.518 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.518 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.519 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.519 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:40:30.519 DEBUG:boto:Data: 2023-10-17T18:40:30.520 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.521 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.521 DEBUG:boto:Params: {} 2023-10-17T18:40:30.522 DEBUG:boto:Token: None 2023-10-17T18:40:30.522 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2 2023-10-17T18:40:30.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qN+lcz3QwqGT9w8XD52EUQnuz9Y= 2023-10-17T18:40:30.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qN+lcz3QwqGT9w8XD52EUQnuz9Y=', 'Content-Length': '0'} 2023-10-17T18:40:30.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d75a39b56bb95f8-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.526 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.526 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.527 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.527 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:40:30.527 DEBUG:boto:Data: 2023-10-17T18:40:30.528 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.529 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.529 DEBUG:boto:Params: {} 2023-10-17T18:40:30.529 DEBUG:boto:Token: None 2023-10-17T18:40:30.530 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2 2023-10-17T18:40:30.530 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qN+lcz3QwqGT9w8XD52EUQnuz9Y= 2023-10-17T18:40:30.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qN+lcz3QwqGT9w8XD52EUQnuz9Y=', 'Content-Length': '0'} 2023-10-17T18:40:30.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c3793dfb8bccdc1-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:30.533 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.534 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.534 DEBUG:boto:Method: GET 2023-10-17T18:40:30.535 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:40:30.535 DEBUG:boto:Data: 2023-10-17T18:40:30.536 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.536 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.537 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.537 DEBUG:boto:Params: {} 2023-10-17T18:40:30.538 DEBUG:boto:Token: None 2023-10-17T18:40:30.538 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2 2023-10-17T18:40:30.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ynuViX68TLZKvQSsXMgNGuDMmM= 2023-10-17T18:40:30.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ynuViX68TLZKvQSsXMgNGuDMmM=', 'Content-Length': '0'} 2023-10-17T18:40:30.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006444c8432e5ddb75-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.542 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:40:30.542 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:40:30.543 DEBUG:boto:Method: GET 2023-10-17T18:40:30.543 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:40:30.544 DEBUG:boto:Data: 2023-10-17T18:40:30.544 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.545 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.545 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.545 DEBUG:boto:Params: {} 2023-10-17T18:40:30.546 DEBUG:boto:Token: None 2023-10-17T18:40:30.546 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj2 2023-10-17T18:40:30.547 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ynuViX68TLZKvQSsXMgNGuDMmM= 2023-10-17T18:40:30.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ynuViX68TLZKvQSsXMgNGuDMmM=', 'Content-Length': '0'} 2023-10-17T18:40:30.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cbb38021bc7d758-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.550 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.550 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.551 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.551 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.552 DEBUG:boto:Method: GET 2023-10-17T18:40:30.552 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.553 DEBUG:boto:Data: 2023-10-17T18:40:30.553 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.554 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.554 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.554 DEBUG:boto:Params: {} 2023-10-17T18:40:30.555 DEBUG:boto:Token: None 2023-10-17T18:40:30.555 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a36aa18acec63c-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.560 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.560 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.561 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.561 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.562 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.562 DEBUG:boto:Method: GET 2023-10-17T18:40:30.562 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:40:30.563 DEBUG:boto:Data: 2023-10-17T18:40:30.563 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.564 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.564 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.565 DEBUG:boto:Params: {} 2023-10-17T18:40:30.565 DEBUG:boto:Token: None 2023-10-17T18:40:30.566 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.566 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c495a46169295a-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.570 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:30.571 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.572 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.572 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.573 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.573 DEBUG:boto:Method: GET 2023-10-17T18:40:30.573 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.574 DEBUG:boto:Data: 2023-10-17T18:40:30.574 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.575 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.575 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.576 DEBUG:boto:Params: {} 2023-10-17T18:40:30.576 DEBUG:boto:Token: None 2023-10-17T18:40:30.576 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.577 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo= 2023-10-17T18:40:30.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo=', 'Content-Length': '0'} 2023-10-17T18:40:30.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082348d320773b78d-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.580 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.581 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.581 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.581 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.582 DEBUG:boto:Method: GET 2023-10-17T18:40:30.582 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.583 DEBUG:boto:Data: 2023-10-17T18:40:30.583 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.584 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.584 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.585 DEBUG:boto:Params: {} 2023-10-17T18:40:30.585 DEBUG:boto:Token: None 2023-10-17T18:40:30.585 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.586 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo= 2023-10-17T18:40:30.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo=', 'Content-Length': '0'} 2023-10-17T18:40:30.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099716a1ce417ec51-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.589 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.590 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.590 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.591 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.591 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.591 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.592 DEBUG:boto:Data: 2023-10-17T18:40:30.592 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.593 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.593 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.594 DEBUG:boto:Params: {} 2023-10-17T18:40:30.594 DEBUG:boto:Token: None 2023-10-17T18:40:30.595 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.595 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9dRS9R0BtDxjXaBL6DDyp8+xwvo= 2023-10-17T18:40:30.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9dRS9R0BtDxjXaBL6DDyp8+xwvo=', 'Content-Length': '0'} 2023-10-17T18:40:30.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006689b5bf9bd13946-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.598 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.598 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.599 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.599 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.600 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.600 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.600 DEBUG:boto:Data: 2023-10-17T18:40:30.601 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.601 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.602 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.602 DEBUG:boto:Params: {} 2023-10-17T18:40:30.603 DEBUG:boto:Token: None 2023-10-17T18:40:30.603 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.604 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9dRS9R0BtDxjXaBL6DDyp8+xwvo= 2023-10-17T18:40:30.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9dRS9R0BtDxjXaBL6DDyp8+xwvo=', 'Content-Length': '0'} 2023-10-17T18:40:30.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac0c5d798f6aef49-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:30.607 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.608 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.608 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.609 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.609 DEBUG:boto:Method: GET 2023-10-17T18:40:30.609 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.610 DEBUG:boto:Data: 2023-10-17T18:40:30.610 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.611 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.611 DEBUG:boto:Params: {} 2023-10-17T18:40:30.612 DEBUG:boto:Token: None 2023-10-17T18:40:30.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo= 2023-10-17T18:40:30.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo=', 'Content-Length': '0'} 2023-10-17T18:40:30.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb70c4788325d57f-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.616 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.616 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.617 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.617 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.618 DEBUG:boto:Method: GET 2023-10-17T18:40:30.618 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.619 DEBUG:boto:Data: 2023-10-17T18:40:30.619 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.620 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.620 DEBUG:boto:Params: {} 2023-10-17T18:40:30.621 DEBUG:boto:Token: None 2023-10-17T18:40:30.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:40:30.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo= 2023-10-17T18:40:30.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hn3bFYX+u9CWwOVc2dEE21Tbvwo=', 'Content-Length': '0'} 2023-10-17T18:40:30.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0cf9a79f823585-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.670 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.671 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.671 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.672 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.672 DEBUG:boto:Method: GET 2023-10-17T18:40:30.673 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.673 DEBUG:boto:Data: 2023-10-17T18:40:30.674 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.674 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.675 DEBUG:boto:Params: {} 2023-10-17T18:40:30.675 DEBUG:boto:Token: None 2023-10-17T18:40:30.676 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd3c87246caa3178-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.680 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.681 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.681 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.682 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.682 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.683 DEBUG:boto:Method: GET 2023-10-17T18:40:30.683 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.684 DEBUG:boto:Data: 2023-10-17T18:40:30.684 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.684 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.685 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.685 DEBUG:boto:Params: {} 2023-10-17T18:40:30.686 DEBUG:boto:Token: None 2023-10-17T18:40:30.686 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.687 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090aecfc646dd02d4-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.690 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.691 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.691 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.692 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.692 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:40:30.693 DEBUG:boto:Data: 2023-10-17T18:40:30.693 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.694 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.694 DEBUG:boto:Params: {} 2023-10-17T18:40:30.695 DEBUG:boto:Token: None 2023-10-17T18:40:30.695 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3 2023-10-17T18:40:30.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PFkun8Rn8zgCj4cG/ulpyGbkU0E= 2023-10-17T18:40:30.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PFkun8Rn8zgCj4cG/ulpyGbkU0E=', 'Content-Length': '0'} 2023-10-17T18:40:30.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdd9756647f2851e-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.699 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.699 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.700 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.700 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:40:30.701 DEBUG:boto:Data: 2023-10-17T18:40:30.701 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.702 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.702 DEBUG:boto:Params: {} 2023-10-17T18:40:30.703 DEBUG:boto:Token: None 2023-10-17T18:40:30.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3 2023-10-17T18:40:30.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PFkun8Rn8zgCj4cG/ulpyGbkU0E= 2023-10-17T18:40:30.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PFkun8Rn8zgCj4cG/ulpyGbkU0E=', 'Content-Length': '0'} 2023-10-17T18:40:30.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e83521e9cfba0a50-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:30.707 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.708 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.708 DEBUG:boto:Method: GET 2023-10-17T18:40:30.709 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:40:30.709 DEBUG:boto:Data: 2023-10-17T18:40:30.710 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.710 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.711 DEBUG:boto:Params: {} 2023-10-17T18:40:30.711 DEBUG:boto:Token: None 2023-10-17T18:40:30.712 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3 2023-10-17T18:40:30.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9ose7C7VvolPWOdm+Qa8QI7R028= 2023-10-17T18:40:30.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9ose7C7VvolPWOdm+Qa8QI7R028=', 'Content-Length': '0'} 2023-10-17T18:40:30.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000513db2cb0e54e99b-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.762 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:40:30.763 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:40:30.763 DEBUG:boto:Method: GET 2023-10-17T18:40:30.764 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:40:30.764 DEBUG:boto:Data: 2023-10-17T18:40:30.765 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.765 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.766 DEBUG:boto:Params: {} 2023-10-17T18:40:30.766 DEBUG:boto:Token: None 2023-10-17T18:40:30.767 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj3 2023-10-17T18:40:30.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9ose7C7VvolPWOdm+Qa8QI7R028= 2023-10-17T18:40:30.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9ose7C7VvolPWOdm+Qa8QI7R028=', 'Content-Length': '0'} 2023-10-17T18:40:30.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008631e93dd4ca3977-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.771 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.771 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.772 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.772 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.772 DEBUG:boto:Method: GET 2023-10-17T18:40:30.773 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.773 DEBUG:boto:Data: 2023-10-17T18:40:30.774 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.775 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.775 DEBUG:boto:Params: {} 2023-10-17T18:40:30.776 DEBUG:boto:Token: None 2023-10-17T18:40:30.776 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b5b2d5e1099c30-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.780 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.781 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.782 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.782 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.783 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.783 DEBUG:boto:Method: GET 2023-10-17T18:40:30.783 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:40:30.784 DEBUG:boto:Data: 2023-10-17T18:40:30.784 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.785 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.786 DEBUG:boto:Params: {} 2023-10-17T18:40:30.786 DEBUG:boto:Token: None 2023-10-17T18:40:30.787 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495db13226c3204d-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.791 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:30.792 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.792 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.793 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.793 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.794 DEBUG:boto:Method: GET 2023-10-17T18:40:30.794 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.795 DEBUG:boto:Data: 2023-10-17T18:40:30.795 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.795 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.796 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.796 DEBUG:boto:Params: {} 2023-10-17T18:40:30.797 DEBUG:boto:Token: None 2023-10-17T18:40:30.797 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk= 2023-10-17T18:40:30.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk=', 'Content-Length': '0'} 2023-10-17T18:40:30.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000154456559def0cad-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.801 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.801 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.802 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.802 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.803 DEBUG:boto:Method: GET 2023-10-17T18:40:30.803 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.804 DEBUG:boto:Data: 2023-10-17T18:40:30.804 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.805 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.805 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.806 DEBUG:boto:Params: {} 2023-10-17T18:40:30.806 DEBUG:boto:Token: None 2023-10-17T18:40:30.807 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.807 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk= 2023-10-17T18:40:30.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk=', 'Content-Length': '0'} 2023-10-17T18:40:30.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a6346070d8881cc-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.811 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.811 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.811 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.812 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.812 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.813 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.813 DEBUG:boto:Data: 2023-10-17T18:40:30.813 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.814 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.814 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.815 DEBUG:boto:Params: {} 2023-10-17T18:40:30.815 DEBUG:boto:Token: None 2023-10-17T18:40:30.816 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QlHOw5u77jR1SdktrntXiBQhUHs= 2023-10-17T18:40:30.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QlHOw5u77jR1SdktrntXiBQhUHs=', 'Content-Length': '0'} 2023-10-17T18:40:30.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb802d6bca4e32bc-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.819 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.820 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.820 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.821 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.821 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.821 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.822 DEBUG:boto:Data: 2023-10-17T18:40:30.822 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.823 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.823 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.823 DEBUG:boto:Params: {} 2023-10-17T18:40:30.824 DEBUG:boto:Token: None 2023-10-17T18:40:30.825 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.826 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QlHOw5u77jR1SdktrntXiBQhUHs= 2023-10-17T18:40:30.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QlHOw5u77jR1SdktrntXiBQhUHs=', 'Content-Length': '0'} 2023-10-17T18:40:30.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c2ee3a8e40e0ae1-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:30.829 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.830 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.830 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.830 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.831 DEBUG:boto:Method: GET 2023-10-17T18:40:30.831 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.832 DEBUG:boto:Data: 2023-10-17T18:40:30.832 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.833 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.834 DEBUG:boto:Params: {} 2023-10-17T18:40:30.834 DEBUG:boto:Token: None 2023-10-17T18:40:30.835 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk= 2023-10-17T18:40:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk=', 'Content-Length': '0'} 2023-10-17T18:40:30.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c59c8e483c1353ff-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.878 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.879 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.879 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.880 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.880 DEBUG:boto:Method: GET 2023-10-17T18:40:30.881 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.881 DEBUG:boto:Data: 2023-10-17T18:40:30.882 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.882 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.883 DEBUG:boto:Params: {} 2023-10-17T18:40:30.883 DEBUG:boto:Token: None 2023-10-17T18:40:30.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:40:30.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk= 2023-10-17T18:40:30.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6KadgOU2tXIBnKv7ZwYZWzaG/Zk=', 'Content-Length': '0'} 2023-10-17T18:40:30.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbf2a39b3ddf4f8c-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.887 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.888 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.888 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.889 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.889 DEBUG:boto:Method: GET 2023-10-17T18:40:30.890 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.890 DEBUG:boto:Data: 2023-10-17T18:40:30.891 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.892 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.892 DEBUG:boto:Params: {} 2023-10-17T18:40:30.893 DEBUG:boto:Token: None 2023-10-17T18:40:30.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.894 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008d019e509693e2-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.897 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.898 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.899 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.899 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.900 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.900 DEBUG:boto:Method: GET 2023-10-17T18:40:30.900 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.901 DEBUG:boto:Data: 2023-10-17T18:40:30.901 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.902 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.903 DEBUG:boto:Params: {} 2023-10-17T18:40:30.903 DEBUG:boto:Token: None 2023-10-17T18:40:30.903 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d93e28ac1acb350a-00652ed51e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.950 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.951 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.951 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.952 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.952 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:40:30.953 DEBUG:boto:Data: 2023-10-17T18:40:30.953 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.953 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.954 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.954 DEBUG:boto:Params: {} 2023-10-17T18:40:30.955 DEBUG:boto:Token: None 2023-10-17T18:40:30.955 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4 2023-10-17T18:40:30.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oGKLdnEO3Pz6AbF+kWxdeQqtGao= 2023-10-17T18:40:30.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oGKLdnEO3Pz6AbF+kWxdeQqtGao=', 'Content-Length': '0'} 2023-10-17T18:40:30.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053d70b3bc6ea4ff0-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.959 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.959 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.959 DEBUG:boto:Method: HEAD 2023-10-17T18:40:30.960 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:40:30.960 DEBUG:boto:Data: 2023-10-17T18:40:30.960 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.961 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.961 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.962 DEBUG:boto:Params: {} 2023-10-17T18:40:30.962 DEBUG:boto:Token: None 2023-10-17T18:40:30.963 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4 2023-10-17T18:40:30.963 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oGKLdnEO3Pz6AbF+kWxdeQqtGao= 2023-10-17T18:40:30.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oGKLdnEO3Pz6AbF+kWxdeQqtGao=', 'Content-Length': '0'} 2023-10-17T18:40:30.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde37657e952a11d-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:30.966 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.967 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.967 DEBUG:boto:Method: GET 2023-10-17T18:40:30.968 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:40:30.968 DEBUG:boto:Data: 2023-10-17T18:40:30.969 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.970 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.970 DEBUG:boto:Params: {} 2023-10-17T18:40:30.971 DEBUG:boto:Token: None 2023-10-17T18:40:30.971 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4 2023-10-17T18:40:30.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FYW61umBQsXlU9ZONfa1gZk5LH8= 2023-10-17T18:40:30.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FYW61umBQsXlU9ZONfa1gZk5LH8=', 'Content-Length': '0'} 2023-10-17T18:40:30.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006957954f44ba4a21-00652ed51e-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.975 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:40:30.975 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:40:30.976 DEBUG:boto:Method: GET 2023-10-17T18:40:30.976 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:40:30.977 DEBUG:boto:Data: 2023-10-17T18:40:30.977 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.978 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.978 DEBUG:boto:Params: {} 2023-10-17T18:40:30.979 DEBUG:boto:Token: None 2023-10-17T18:40:30.979 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/obj4 2023-10-17T18:40:30.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FYW61umBQsXlU9ZONfa1gZk5LH8= 2023-10-17T18:40:30.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FYW61umBQsXlU9ZONfa1gZk5LH8=', 'Content-Length': '0'} 2023-10-17T18:40:30.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021841260c5955711-00652ed51e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.983 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.984 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.984 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.985 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.985 DEBUG:boto:Method: GET 2023-10-17T18:40:30.986 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.986 DEBUG:boto:Data: 2023-10-17T18:40:30.986 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.987 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:30.987 DEBUG:boto:Port: 8001 2023-10-17T18:40:30.988 DEBUG:boto:Params: {} 2023-10-17T18:40:30.988 DEBUG:boto:Token: None 2023-10-17T18:40:30.989 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:30.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:30.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:30.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b3f7ea7de2bf8c1-00652ed51e-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:30.993 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:30.994 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:30.994 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:30.995 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.995 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.996 DEBUG:boto:Method: GET 2023-10-17T18:40:30.996 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:40:30.996 DEBUG:boto:Data: 2023-10-17T18:40:30.997 DEBUG:boto:Headers: {} 2023-10-17T18:40:30.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:30.998 DEBUG:boto:Port: 8000 2023-10-17T18:40:30.998 DEBUG:boto:Params: {} 2023-10-17T18:40:30.999 DEBUG:boto:Token: None 2023-10-17T18:40:30.999 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:30 GMT /hbmuog-48/ 2023-10-17T18:40:31.000 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ= 2023-10-17T18:40:31.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFcg9ZEqvLajv/c4aNbocSqYKBQ=', 'Content-Length': '0'} 2023-10-17T18:40:31.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e428c482505b0fbe-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.003 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-48 zones={a2, a1} 2023-10-17T18:40:31.004 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-49 2023-10-17T18:40:31.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:31.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:31.006 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:40:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:31.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:31.111+0000 7fd73b65b780 1 bucket sync caught up with source: 2023-10-17T18:40:31.113 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.17.6, , 00000000001.9.6, , 00000000001.36.6, , , 00000000001.29.6, , , ] 2023-10-17T18:40:31.113 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.17.6, , 00000000001.9.6, , 00000000001.36.6, , , 00000000001.29.6, , , ] 2023-10-17T18:40:31.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:31.111+0000 7fd73b65b780 0 bucket checkpoint complete 2023-10-17T18:40:31.126 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-49 2023-10-17T18:40:31.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:31.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:31.128 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:40:31.187 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:31.187 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:31.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:31.223+0000 7f6f48934780 1 bucket sync caught up with source: 2023-10-17T18:40:31.225 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.87.6, , 00000000001.128.6, , 00000000001.244.6, , , 00000000001.87.6, , , ] 2023-10-17T18:40:31.225 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.87.6, , 00000000001.128.6, , 00000000001.244.6, , , 00000000001.87.6, , , ] 2023-10-17T18:40:31.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:31.223+0000 7f6f48934780 0 bucket checkpoint complete 2023-10-17T18:40:31.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-49 zones={a2, a1} 2023-10-17T18:40:31.238 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.239 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.239 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.240 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:31.240 DEBUG:boto:Data: 2023-10-17T18:40:31.240 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.241 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.242 DEBUG:boto:Params: {} 2023-10-17T18:40:31.242 DEBUG:boto:Token: None 2023-10-17T18:40:31.243 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mphndd46Qha2+YfTyIeWAoqpyqM= 2023-10-17T18:40:31.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mphndd46Qha2+YfTyIeWAoqpyqM=', 'Content-Length': '0'} 2023-10-17T18:40:31.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dc091470402a9df-00652ed51f-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.247 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.247 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.248 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.248 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:31.249 DEBUG:boto:Data: 2023-10-17T18:40:31.249 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.250 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.250 DEBUG:boto:Params: {} 2023-10-17T18:40:31.251 DEBUG:boto:Token: None 2023-10-17T18:40:31.251 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mphndd46Qha2+YfTyIeWAoqpyqM= 2023-10-17T18:40:31.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mphndd46Qha2+YfTyIeWAoqpyqM=', 'Content-Length': '0'} 2023-10-17T18:40:31.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000768bb68442ca51a4-00652ed51f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.255 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:40:31.255 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.256 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.256 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.257 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.257 DEBUG:boto:Method: GET 2023-10-17T18:40:31.257 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.258 DEBUG:boto:Data: 2023-10-17T18:40:31.258 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.259 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.259 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.260 DEBUG:boto:Params: {} 2023-10-17T18:40:31.260 DEBUG:boto:Token: None 2023-10-17T18:40:31.260 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/?versions 2023-10-17T18:40:31.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY= 2023-10-17T18:40:31.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY=', 'Content-Length': '0'} 2023-10-17T18:40:31.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998e3cdb2c445300-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.264 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:40:31.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:40:31.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:40:31.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:40:31.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:40:31.267 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.267 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.268 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.268 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.268 DEBUG:boto:Method: GET 2023-10-17T18:40:31.269 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.269 DEBUG:boto:Data: 2023-10-17T18:40:31.269 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.270 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.271 DEBUG:boto:Params: {} 2023-10-17T18:40:31.271 DEBUG:boto:Token: None 2023-10-17T18:40:31.272 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/?versions 2023-10-17T18:40:31.272 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY= 2023-10-17T18:40:31.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY=', 'Content-Length': '0'} 2023-10-17T18:40:31.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a21be021231851-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.276 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:40:31.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:40:31.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:40:31.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:40:31.278 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.278 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.279 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.279 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.279 DEBUG:boto:Method: GET 2023-10-17T18:40:31.280 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.280 DEBUG:boto:Data: 2023-10-17T18:40:31.281 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.281 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.282 DEBUG:boto:Params: {} 2023-10-17T18:40:31.282 DEBUG:boto:Token: None 2023-10-17T18:40:31.283 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/?versions 2023-10-17T18:40:31.283 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY= 2023-10-17T18:40:31.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY=', 'Content-Length': '0'} 2023-10-17T18:40:31.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9a7c02fc1eb810-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.287 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.287 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.288 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.288 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.289 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.289 DEBUG:boto:Method: GET 2023-10-17T18:40:31.289 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:40:31.290 DEBUG:boto:Data: 2023-10-17T18:40:31.290 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.291 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.291 DEBUG:boto:Params: {} 2023-10-17T18:40:31.292 DEBUG:boto:Token: None 2023-10-17T18:40:31.292 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/?versions 2023-10-17T18:40:31.293 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY= 2023-10-17T18:40:31.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:134A7xeRxBfiHSwYD0/vpHg40TY=', 'Content-Length': '0'} 2023-10-17T18:40:31.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000429af1528a484443-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.296 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:31.297 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.298 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.298 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.299 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.299 DEBUG:boto:Method: GET 2023-10-17T18:40:31.299 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.300 DEBUG:boto:Data: 2023-10-17T18:40:31.300 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.301 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.301 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.302 DEBUG:boto:Params: {} 2023-10-17T18:40:31.302 DEBUG:boto:Token: None 2023-10-17T18:40:31.303 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.303 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI= 2023-10-17T18:40:31.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI=', 'Content-Length': '0'} 2023-10-17T18:40:31.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000880be55769ce68e2-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.350 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.351 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.351 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.352 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.352 DEBUG:boto:Method: GET 2023-10-17T18:40:31.353 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.353 DEBUG:boto:Data: 2023-10-17T18:40:31.354 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.354 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.355 DEBUG:boto:Params: {} 2023-10-17T18:40:31.355 DEBUG:boto:Token: None 2023-10-17T18:40:31.356 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI= 2023-10-17T18:40:31.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI=', 'Content-Length': '0'} 2023-10-17T18:40:31.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f424f57b7c6c237-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.415 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.415 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.415 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.416 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.416 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.417 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.417 DEBUG:boto:Data: 2023-10-17T18:40:31.417 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.418 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.419 DEBUG:boto:Params: {} 2023-10-17T18:40:31.419 DEBUG:boto:Token: None 2023-10-17T18:40:31.419 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4o19exbfJOeGXdCKwIlz4RQt8k= 2023-10-17T18:40:31.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4o19exbfJOeGXdCKwIlz4RQt8k=', 'Content-Length': '0'} 2023-10-17T18:40:31.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b28d0cdf4a5edb71-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.423 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.423 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.423 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.424 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.424 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.425 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.425 DEBUG:boto:Data: 2023-10-17T18:40:31.425 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.426 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.427 DEBUG:boto:Params: {} 2023-10-17T18:40:31.427 DEBUG:boto:Token: None 2023-10-17T18:40:31.428 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.428 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4o19exbfJOeGXdCKwIlz4RQt8k= 2023-10-17T18:40:31.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4o19exbfJOeGXdCKwIlz4RQt8k=', 'Content-Length': '0'} 2023-10-17T18:40:31.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080708fe63a2e99c5-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:31.431 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.431 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.432 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.432 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.433 DEBUG:boto:Method: GET 2023-10-17T18:40:31.433 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.433 DEBUG:boto:Data: 2023-10-17T18:40:31.434 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.435 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.435 DEBUG:boto:Params: {} 2023-10-17T18:40:31.436 DEBUG:boto:Token: None 2023-10-17T18:40:31.436 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI= 2023-10-17T18:40:31.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI=', 'Content-Length': '0'} 2023-10-17T18:40:31.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005189489451d452a8-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.439 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.441 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.441 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.442 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.442 DEBUG:boto:Method: GET 2023-10-17T18:40:31.443 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.443 DEBUG:boto:Data: 2023-10-17T18:40:31.444 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.444 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.445 DEBUG:boto:Params: {} 2023-10-17T18:40:31.445 DEBUG:boto:Token: None 2023-10-17T18:40:31.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:40:31.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI= 2023-10-17T18:40:31.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k+xXHhYT37Uf6pabsHKjr+R9QtI=', 'Content-Length': '0'} 2023-10-17T18:40:31.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c842ba4f08a3a5bd-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.449 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.450 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.450 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.451 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.451 DEBUG:boto:Method: GET 2023-10-17T18:40:31.451 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.452 DEBUG:boto:Data: 2023-10-17T18:40:31.452 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.453 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.454 DEBUG:boto:Params: {} 2023-10-17T18:40:31.454 DEBUG:boto:Token: None 2023-10-17T18:40:31.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.455 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe73c24f49e8b09e-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.458 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.459 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.460 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.460 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.460 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.461 DEBUG:boto:Method: GET 2023-10-17T18:40:31.461 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.462 DEBUG:boto:Data: 2023-10-17T18:40:31.462 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.463 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.463 DEBUG:boto:Params: {} 2023-10-17T18:40:31.464 DEBUG:boto:Token: None 2023-10-17T18:40:31.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097490e051d4d032a-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.468 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.468 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.469 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.469 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.470 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:40:31.470 DEBUG:boto:Data: 2023-10-17T18:40:31.470 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.471 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.472 DEBUG:boto:Params: {} 2023-10-17T18:40:31.472 DEBUG:boto:Token: None 2023-10-17T18:40:31.473 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1 2023-10-17T18:40:31.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lno8F41Hi4d/KkKCQSM1zFDw24Y= 2023-10-17T18:40:31.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lno8F41Hi4d/KkKCQSM1zFDw24Y=', 'Content-Length': '0'} 2023-10-17T18:40:31.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000225c6cec3a74e6c2-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.476 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.476 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.476 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.477 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:40:31.477 DEBUG:boto:Data: 2023-10-17T18:40:31.478 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.478 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.479 DEBUG:boto:Params: {} 2023-10-17T18:40:31.479 DEBUG:boto:Token: None 2023-10-17T18:40:31.480 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1 2023-10-17T18:40:31.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lno8F41Hi4d/KkKCQSM1zFDw24Y= 2023-10-17T18:40:31.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lno8F41Hi4d/KkKCQSM1zFDw24Y=', 'Content-Length': '0'} 2023-10-17T18:40:31.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccbd74b62180411b-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:40:31.483 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.484 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.484 DEBUG:boto:Method: GET 2023-10-17T18:40:31.485 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:40:31.485 DEBUG:boto:Data: 2023-10-17T18:40:31.485 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.486 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.487 DEBUG:boto:Params: {} 2023-10-17T18:40:31.487 DEBUG:boto:Token: None 2023-10-17T18:40:31.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1 2023-10-17T18:40:31.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IRMH0eX8WhCRlnfMrdAmP9QZKUM= 2023-10-17T18:40:31.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IRMH0eX8WhCRlnfMrdAmP9QZKUM=', 'Content-Length': '0'} 2023-10-17T18:40:31.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfb01a4a465f2ef8-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.491 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:40:31.491 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:40:31.492 DEBUG:boto:Method: GET 2023-10-17T18:40:31.492 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:40:31.493 DEBUG:boto:Data: 2023-10-17T18:40:31.493 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.494 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.494 DEBUG:boto:Params: {} 2023-10-17T18:40:31.495 DEBUG:boto:Token: None 2023-10-17T18:40:31.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj1 2023-10-17T18:40:31.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IRMH0eX8WhCRlnfMrdAmP9QZKUM= 2023-10-17T18:40:31.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IRMH0eX8WhCRlnfMrdAmP9QZKUM=', 'Content-Length': '0'} 2023-10-17T18:40:31.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bacef9ace99a7fd0-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.499 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.499 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.500 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.500 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.501 DEBUG:boto:Method: GET 2023-10-17T18:40:31.501 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.501 DEBUG:boto:Data: 2023-10-17T18:40:31.502 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.503 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.503 DEBUG:boto:Params: {} 2023-10-17T18:40:31.504 DEBUG:boto:Token: None 2023-10-17T18:40:31.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000166df886c1f744ae-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.508 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.509 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.510 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.510 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.511 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.511 DEBUG:boto:Method: GET 2023-10-17T18:40:31.511 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:40:31.512 DEBUG:boto:Data: 2023-10-17T18:40:31.512 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.513 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.513 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.514 DEBUG:boto:Params: {} 2023-10-17T18:40:31.514 DEBUG:boto:Token: None 2023-10-17T18:40:31.515 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ae9aee9bc18177-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.562 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:31.564 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.564 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.565 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.565 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.566 DEBUG:boto:Method: GET 2023-10-17T18:40:31.566 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.566 DEBUG:boto:Data: 2023-10-17T18:40:31.567 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.568 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.568 DEBUG:boto:Params: {} 2023-10-17T18:40:31.569 DEBUG:boto:Token: None 2023-10-17T18:40:31.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w= 2023-10-17T18:40:31.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w=', 'Content-Length': '0'} 2023-10-17T18:40:31.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034066baad2f45917-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.573 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.573 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.574 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.574 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.575 DEBUG:boto:Method: GET 2023-10-17T18:40:31.575 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.575 DEBUG:boto:Data: 2023-10-17T18:40:31.576 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.577 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.577 DEBUG:boto:Params: {} 2023-10-17T18:40:31.578 DEBUG:boto:Token: None 2023-10-17T18:40:31.578 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w= 2023-10-17T18:40:31.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w=', 'Content-Length': '0'} 2023-10-17T18:40:31.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a00f7108ab50b753-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.582 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.582 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.582 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.583 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.583 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.584 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.584 DEBUG:boto:Data: 2023-10-17T18:40:31.585 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.585 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.586 DEBUG:boto:Params: {} 2023-10-17T18:40:31.586 DEBUG:boto:Token: None 2023-10-17T18:40:31.587 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GlKftWYn1WAwJYpU8pQf/sJ4F9s= 2023-10-17T18:40:31.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GlKftWYn1WAwJYpU8pQf/sJ4F9s=', 'Content-Length': '0'} 2023-10-17T18:40:31.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa1dd7f91228abaa-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.590 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.590 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.591 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.591 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.592 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.592 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.592 DEBUG:boto:Data: 2023-10-17T18:40:31.593 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.593 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.594 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.594 DEBUG:boto:Params: {} 2023-10-17T18:40:31.595 DEBUG:boto:Token: None 2023-10-17T18:40:31.595 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.595 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GlKftWYn1WAwJYpU8pQf/sJ4F9s= 2023-10-17T18:40:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GlKftWYn1WAwJYpU8pQf/sJ4F9s=', 'Content-Length': '0'} 2023-10-17T18:40:31.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e573ea2ea45b3a6f-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:31.598 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.599 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.599 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.600 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.600 DEBUG:boto:Method: GET 2023-10-17T18:40:31.601 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.601 DEBUG:boto:Data: 2023-10-17T18:40:31.602 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.602 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.602 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.603 DEBUG:boto:Params: {} 2023-10-17T18:40:31.603 DEBUG:boto:Token: None 2023-10-17T18:40:31.604 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.604 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w= 2023-10-17T18:40:31.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w=', 'Content-Length': '0'} 2023-10-17T18:40:31.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371f52cef66e98ac-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.607 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.608 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.608 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.609 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.609 DEBUG:boto:Method: GET 2023-10-17T18:40:31.609 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.610 DEBUG:boto:Data: 2023-10-17T18:40:31.610 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.611 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.611 DEBUG:boto:Params: {} 2023-10-17T18:40:31.612 DEBUG:boto:Token: None 2023-10-17T18:40:31.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:40:31.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w= 2023-10-17T18:40:31.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S6/DcZpRyJd5Q4HbtIov1qP6b3w=', 'Content-Length': '0'} 2023-10-17T18:40:31.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005af06c87d10a5b30-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.616 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.616 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.617 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.617 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.617 DEBUG:boto:Method: GET 2023-10-17T18:40:31.618 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.618 DEBUG:boto:Data: 2023-10-17T18:40:31.619 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.620 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.620 DEBUG:boto:Params: {} 2023-10-17T18:40:31.620 DEBUG:boto:Token: None 2023-10-17T18:40:31.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a5ec15ebc7f2bc-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.666 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.667 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.668 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.668 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.669 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.669 DEBUG:boto:Method: GET 2023-10-17T18:40:31.669 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.670 DEBUG:boto:Data: 2023-10-17T18:40:31.670 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.671 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.671 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.672 DEBUG:boto:Params: {} 2023-10-17T18:40:31.672 DEBUG:boto:Token: None 2023-10-17T18:40:31.673 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5b3f7a4ddec7eb-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.677 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.677 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.678 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.678 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.679 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:40:31.679 DEBUG:boto:Data: 2023-10-17T18:40:31.680 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.680 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.681 DEBUG:boto:Params: {} 2023-10-17T18:40:31.681 DEBUG:boto:Token: None 2023-10-17T18:40:31.682 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2 2023-10-17T18:40:31.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P6DWi78jbgY+n7360KjLtTYLvvE= 2023-10-17T18:40:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P6DWi78jbgY+n7360KjLtTYLvvE=', 'Content-Length': '0'} 2023-10-17T18:40:31.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de130ef2f22c1f85-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.685 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.686 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.686 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.687 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:40:31.687 DEBUG:boto:Data: 2023-10-17T18:40:31.687 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.688 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.689 DEBUG:boto:Params: {} 2023-10-17T18:40:31.689 DEBUG:boto:Token: None 2023-10-17T18:40:31.690 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2 2023-10-17T18:40:31.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P6DWi78jbgY+n7360KjLtTYLvvE= 2023-10-17T18:40:31.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P6DWi78jbgY+n7360KjLtTYLvvE=', 'Content-Length': '0'} 2023-10-17T18:40:31.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014d2477a1f5c6e45-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:40:31.693 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.694 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.694 DEBUG:boto:Method: GET 2023-10-17T18:40:31.695 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:40:31.695 DEBUG:boto:Data: 2023-10-17T18:40:31.696 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.696 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.697 DEBUG:boto:Params: {} 2023-10-17T18:40:31.697 DEBUG:boto:Token: None 2023-10-17T18:40:31.698 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2 2023-10-17T18:40:31.698 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VyXCOgLp57o0vvxewC21t3UmD+s= 2023-10-17T18:40:31.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VyXCOgLp57o0vvxewC21t3UmD+s=', 'Content-Length': '0'} 2023-10-17T18:40:31.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057607e8122a23628-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.701 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:40:31.701 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:40:31.702 DEBUG:boto:Method: GET 2023-10-17T18:40:31.702 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:40:31.703 DEBUG:boto:Data: 2023-10-17T18:40:31.703 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.703 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.704 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.704 DEBUG:boto:Params: {} 2023-10-17T18:40:31.705 DEBUG:boto:Token: None 2023-10-17T18:40:31.705 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj2 2023-10-17T18:40:31.706 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VyXCOgLp57o0vvxewC21t3UmD+s= 2023-10-17T18:40:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VyXCOgLp57o0vvxewC21t3UmD+s=', 'Content-Length': '0'} 2023-10-17T18:40:31.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044507d11ef45a9df-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.709 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.709 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.710 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.710 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.711 DEBUG:boto:Method: GET 2023-10-17T18:40:31.711 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.711 DEBUG:boto:Data: 2023-10-17T18:40:31.712 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.712 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.713 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.713 DEBUG:boto:Params: {} 2023-10-17T18:40:31.714 DEBUG:boto:Token: None 2023-10-17T18:40:31.714 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.714 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b138b896c0dd639f-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.719 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.720 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.720 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.720 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.721 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.721 DEBUG:boto:Method: GET 2023-10-17T18:40:31.722 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:40:31.722 DEBUG:boto:Data: 2023-10-17T18:40:31.723 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.723 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.724 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.724 DEBUG:boto:Params: {} 2023-10-17T18:40:31.724 DEBUG:boto:Token: None 2023-10-17T18:40:31.725 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008903b3c180105284-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.728 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:31.730 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.730 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.730 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.731 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.731 DEBUG:boto:Method: GET 2023-10-17T18:40:31.732 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.732 DEBUG:boto:Data: 2023-10-17T18:40:31.733 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.733 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.734 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.734 DEBUG:boto:Params: {} 2023-10-17T18:40:31.734 DEBUG:boto:Token: None 2023-10-17T18:40:31.735 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.735 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A= 2023-10-17T18:40:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A=', 'Content-Length': '0'} 2023-10-17T18:40:31.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000003c741c00e425fc-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.738 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.739 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.739 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.740 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.740 DEBUG:boto:Method: GET 2023-10-17T18:40:31.740 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.741 DEBUG:boto:Data: 2023-10-17T18:40:31.741 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.742 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.742 DEBUG:boto:Params: {} 2023-10-17T18:40:31.743 DEBUG:boto:Token: None 2023-10-17T18:40:31.743 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A= 2023-10-17T18:40:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A=', 'Content-Length': '0'} 2023-10-17T18:40:31.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024af250f3a980c42-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.747 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.747 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.748 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.748 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.748 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.749 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.749 DEBUG:boto:Data: 2023-10-17T18:40:31.750 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.750 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.751 DEBUG:boto:Params: {} 2023-10-17T18:40:31.751 DEBUG:boto:Token: None 2023-10-17T18:40:31.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7TGEOzl4K55igHwHD2x97F29opw= 2023-10-17T18:40:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7TGEOzl4K55igHwHD2x97F29opw=', 'Content-Length': '0'} 2023-10-17T18:40:31.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b985812571b050d3-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.755 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.755 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.756 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.756 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.756 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.757 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.757 DEBUG:boto:Data: 2023-10-17T18:40:31.758 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.758 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.759 DEBUG:boto:Params: {} 2023-10-17T18:40:31.759 DEBUG:boto:Token: None 2023-10-17T18:40:31.760 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7TGEOzl4K55igHwHD2x97F29opw= 2023-10-17T18:40:31.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7TGEOzl4K55igHwHD2x97F29opw=', 'Content-Length': '0'} 2023-10-17T18:40:31.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6e4f9bfea2a79d-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:31.763 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.764 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.764 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.765 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.765 DEBUG:boto:Method: GET 2023-10-17T18:40:31.765 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.766 DEBUG:boto:Data: 2023-10-17T18:40:31.766 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.767 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.767 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.768 DEBUG:boto:Params: {} 2023-10-17T18:40:31.768 DEBUG:boto:Token: None 2023-10-17T18:40:31.769 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A= 2023-10-17T18:40:31.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A=', 'Content-Length': '0'} 2023-10-17T18:40:31.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007910850872db9261-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.772 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.772 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.772 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.773 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.773 DEBUG:boto:Method: GET 2023-10-17T18:40:31.774 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.774 DEBUG:boto:Data: 2023-10-17T18:40:31.774 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.775 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.776 DEBUG:boto:Params: {} 2023-10-17T18:40:31.776 DEBUG:boto:Token: None 2023-10-17T18:40:31.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:40:31.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A= 2023-10-17T18:40:31.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+0mYE/gDFRBOl/EcW/8KTekIZ6A=', 'Content-Length': '0'} 2023-10-17T18:40:31.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fbcca9b6841de4e-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.822 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.823 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.823 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.824 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.824 DEBUG:boto:Method: GET 2023-10-17T18:40:31.825 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.825 DEBUG:boto:Data: 2023-10-17T18:40:31.825 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.826 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.827 DEBUG:boto:Params: {} 2023-10-17T18:40:31.827 DEBUG:boto:Token: None 2023-10-17T18:40:31.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7aa3a2a9328296-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.831 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.832 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.833 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.833 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.834 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.834 DEBUG:boto:Method: GET 2023-10-17T18:40:31.835 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.835 DEBUG:boto:Data: 2023-10-17T18:40:31.835 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.836 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.837 DEBUG:boto:Params: {} 2023-10-17T18:40:31.837 DEBUG:boto:Token: None 2023-10-17T18:40:31.838 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c106af843806d5-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.841 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.842 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.843 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.843 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.843 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:40:31.844 DEBUG:boto:Data: 2023-10-17T18:40:31.844 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.845 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.846 DEBUG:boto:Params: {} 2023-10-17T18:40:31.846 DEBUG:boto:Token: None 2023-10-17T18:40:31.847 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3 2023-10-17T18:40:31.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9wpNMzDtDmOTcc39iJhkRh01rqw= 2023-10-17T18:40:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9wpNMzDtDmOTcc39iJhkRh01rqw=', 'Content-Length': '0'} 2023-10-17T18:40:31.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000360cf32dc02f9009-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.850 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.850 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.851 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.851 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:40:31.852 DEBUG:boto:Data: 2023-10-17T18:40:31.852 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.852 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.853 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.853 DEBUG:boto:Params: {} 2023-10-17T18:40:31.854 DEBUG:boto:Token: None 2023-10-17T18:40:31.854 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3 2023-10-17T18:40:31.855 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9wpNMzDtDmOTcc39iJhkRh01rqw= 2023-10-17T18:40:31.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9wpNMzDtDmOTcc39iJhkRh01rqw=', 'Content-Length': '0'} 2023-10-17T18:40:31.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca3514531cc1b8e4-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:40:31.858 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.858 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.859 DEBUG:boto:Method: GET 2023-10-17T18:40:31.859 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:40:31.860 DEBUG:boto:Data: 2023-10-17T18:40:31.860 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.860 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.861 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.861 DEBUG:boto:Params: {} 2023-10-17T18:40:31.862 DEBUG:boto:Token: None 2023-10-17T18:40:31.862 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3 2023-10-17T18:40:31.863 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZL2rKzuBxLcaXkneocXK/INNxc0= 2023-10-17T18:40:31.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZL2rKzuBxLcaXkneocXK/INNxc0=', 'Content-Length': '0'} 2023-10-17T18:40:31.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc5302e76a80ac4-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.910 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:40:31.911 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:40:31.912 DEBUG:boto:Method: GET 2023-10-17T18:40:31.912 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:40:31.912 DEBUG:boto:Data: 2023-10-17T18:40:31.913 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.914 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.914 DEBUG:boto:Params: {} 2023-10-17T18:40:31.915 DEBUG:boto:Token: None 2023-10-17T18:40:31.915 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj3 2023-10-17T18:40:31.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZL2rKzuBxLcaXkneocXK/INNxc0= 2023-10-17T18:40:31.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZL2rKzuBxLcaXkneocXK/INNxc0=', 'Content-Length': '0'} 2023-10-17T18:40:31.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f35aaf8bc02a1b4-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.919 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.919 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.919 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.920 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.920 DEBUG:boto:Method: GET 2023-10-17T18:40:31.921 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.921 DEBUG:boto:Data: 2023-10-17T18:40:31.921 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.922 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.922 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.923 DEBUG:boto:Params: {} 2023-10-17T18:40:31.923 DEBUG:boto:Token: None 2023-10-17T18:40:31.924 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.924 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000de0db291325a4bc-00652ed51f-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.928 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.928 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:31.929 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:31.929 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.930 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.930 DEBUG:boto:Method: GET 2023-10-17T18:40:31.930 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:40:31.931 DEBUG:boto:Data: 2023-10-17T18:40:31.931 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.932 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.932 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.932 DEBUG:boto:Params: {} 2023-10-17T18:40:31.933 DEBUG:boto:Token: None 2023-10-17T18:40:31.933 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/ 2023-10-17T18:40:31.934 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k= 2023-10-17T18:40:31.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ElLc4Rm65fvYYXt6hGObTJOvY2k=', 'Content-Length': '0'} 2023-10-17T18:40:31.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a79e05ea50be5ae-00652ed51f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.937 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:31.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:31.938 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:31.939 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:31.939 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.940 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.940 DEBUG:boto:Method: GET 2023-10-17T18:40:31.941 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.941 DEBUG:boto:Data: 2023-10-17T18:40:31.941 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.942 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.943 DEBUG:boto:Params: {} 2023-10-17T18:40:31.943 DEBUG:boto:Token: None 2023-10-17T18:40:31.944 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8= 2023-10-17T18:40:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8=', 'Content-Length': '0'} 2023-10-17T18:40:31.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0dc72be0e73bb3c-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.947 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:31.947 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:31.948 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.948 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.949 DEBUG:boto:Method: GET 2023-10-17T18:40:31.949 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.949 DEBUG:boto:Data: 2023-10-17T18:40:31.950 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.950 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.951 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.951 DEBUG:boto:Params: {} 2023-10-17T18:40:31.952 DEBUG:boto:Token: None 2023-10-17T18:40:31.952 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8= 2023-10-17T18:40:31.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8=', 'Content-Length': '0'} 2023-10-17T18:40:31.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d45786562b213b2-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.955 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:31.956 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:31.956 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.957 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.957 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.957 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.958 DEBUG:boto:Data: 2023-10-17T18:40:31.958 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.959 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.959 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.959 DEBUG:boto:Params: {} 2023-10-17T18:40:31.960 DEBUG:boto:Token: None 2023-10-17T18:40:31.960 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.961 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EqcZzR0NtTUX9u5qNzG2TuKfpow= 2023-10-17T18:40:31.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EqcZzR0NtTUX9u5qNzG2TuKfpow=', 'Content-Length': '0'} 2023-10-17T18:40:31.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c52b32f619b75b5-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.963 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:31.964 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:31.964 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.965 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.965 DEBUG:boto:Method: HEAD 2023-10-17T18:40:31.965 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.966 DEBUG:boto:Data: 2023-10-17T18:40:31.966 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:31.967 DEBUG:boto:Port: 8000 2023-10-17T18:40:31.967 DEBUG:boto:Params: {} 2023-10-17T18:40:31.968 DEBUG:boto:Token: None 2023-10-17T18:40:31.968 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.969 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EqcZzR0NtTUX9u5qNzG2TuKfpow= 2023-10-17T18:40:31.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EqcZzR0NtTUX9u5qNzG2TuKfpow=', 'Content-Length': '0'} 2023-10-17T18:40:31.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcde6ccee9fd3659-00652ed51f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:31.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:31.972 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:31.972 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:31.973 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.973 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.973 DEBUG:boto:Method: GET 2023-10-17T18:40:31.974 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.974 DEBUG:boto:Data: 2023-10-17T18:40:31.975 DEBUG:boto:Headers: {} 2023-10-17T18:40:31.975 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:31.976 DEBUG:boto:Port: 8001 2023-10-17T18:40:31.976 DEBUG:boto:Params: {} 2023-10-17T18:40:31.977 DEBUG:boto:Token: None 2023-10-17T18:40:31.977 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:31 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:31.977 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8= 2023-10-17T18:40:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Wee0oBBL3S0NmF/4l4sXsBbsRA8=', 'Content-Length': '0'} 2023-10-17T18:40:31.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb859dea74726901-00652ed51f-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.026 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:32.027 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:32.027 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:32.027 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:40:32.028 DEBUG:boto:Method: GET 2023-10-17T18:40:32.028 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:32.029 DEBUG:boto:Data: 2023-10-17T18:40:32.029 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.029 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:32.030 DEBUG:boto:Port: 8000 2023-10-17T18:40:32.030 DEBUG:boto:Params: {} 2023-10-17T18:40:32.031 DEBUG:boto:Token: None 2023-10-17T18:40:32.031 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:40:32.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wAknT0eVMGIW6NI+FlAN9W2dhWw= 2023-10-17T18:40:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wAknT0eVMGIW6NI+FlAN9W2dhWw=', 'Content-Length': '0'} 2023-10-17T18:40:32.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008967137176c10bca-00652ed520-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.035 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:32.035 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:32.035 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.036 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.036 DEBUG:boto:Method: GET 2023-10-17T18:40:32.037 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.037 DEBUG:boto:Data: 2023-10-17T18:40:32.037 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:32.038 DEBUG:boto:Port: 8001 2023-10-17T18:40:32.039 DEBUG:boto:Params: {} 2023-10-17T18:40:32.039 DEBUG:boto:Token: None 2023-10-17T18:40:32.039 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/ 2023-10-17T18:40:32.040 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28= 2023-10-17T18:40:32.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28=', 'Content-Length': '0'} 2023-10-17T18:40:32.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d17ee690feae404a-00652ed520-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.043 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:32.044 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:32.044 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:32.045 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.045 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.045 DEBUG:boto:Method: GET 2023-10-17T18:40:32.046 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.046 DEBUG:boto:Data: 2023-10-17T18:40:32.047 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:32.048 DEBUG:boto:Port: 8000 2023-10-17T18:40:32.048 DEBUG:boto:Params: {} 2023-10-17T18:40:32.048 DEBUG:boto:Token: None 2023-10-17T18:40:32.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/ 2023-10-17T18:40:32.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28= 2023-10-17T18:40:32.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28=', 'Content-Length': '0'} 2023-10-17T18:40:32.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063c8ea8ca81be40-00652ed520-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.054 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:32.054 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:32.055 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:32.055 DEBUG:boto:Method: HEAD 2023-10-17T18:40:32.056 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:40:32.056 DEBUG:boto:Data: 2023-10-17T18:40:32.056 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:32.057 DEBUG:boto:Port: 8001 2023-10-17T18:40:32.058 DEBUG:boto:Params: {} 2023-10-17T18:40:32.058 DEBUG:boto:Token: None 2023-10-17T18:40:32.059 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/obj4 2023-10-17T18:40:32.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RAK4j0ZEUcO+nC5O3K6FtaEeIVU= 2023-10-17T18:40:32.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RAK4j0ZEUcO+nC5O3K6FtaEeIVU=', 'Content-Length': '0'} 2023-10-17T18:40:32.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030497dd590a3550a-00652ed520-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.062 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:32.063 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:32.063 DEBUG:boto:Method: HEAD 2023-10-17T18:40:32.064 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:40:32.064 DEBUG:boto:Data: 2023-10-17T18:40:32.064 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:32.065 DEBUG:boto:Port: 8000 2023-10-17T18:40:32.066 DEBUG:boto:Params: {} 2023-10-17T18:40:32.066 DEBUG:boto:Token: None 2023-10-17T18:40:32.067 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/obj4 2023-10-17T18:40:32.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RAK4j0ZEUcO+nC5O3K6FtaEeIVU= 2023-10-17T18:40:32.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RAK4j0ZEUcO+nC5O3K6FtaEeIVU=', 'Content-Length': '0'} 2023-10-17T18:40:32.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065c14f01d5504b8c-00652ed520-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:40:32.070 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:32.071 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:32.071 DEBUG:boto:Method: GET 2023-10-17T18:40:32.072 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:40:32.072 DEBUG:boto:Data: 2023-10-17T18:40:32.073 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:32.074 DEBUG:boto:Port: 8001 2023-10-17T18:40:32.074 DEBUG:boto:Params: {} 2023-10-17T18:40:32.075 DEBUG:boto:Token: None 2023-10-17T18:40:32.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/obj4 2023-10-17T18:40:32.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yolMYNOytiscSU6HftYyaNEAJWo= 2023-10-17T18:40:32.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yolMYNOytiscSU6HftYyaNEAJWo=', 'Content-Length': '0'} 2023-10-17T18:40:32.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003312570a25d7fe92-00652ed520-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.079 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:40:32.079 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:40:32.080 DEBUG:boto:Method: GET 2023-10-17T18:40:32.080 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:40:32.081 DEBUG:boto:Data: 2023-10-17T18:40:32.081 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:32.082 DEBUG:boto:Port: 8000 2023-10-17T18:40:32.083 DEBUG:boto:Params: {} 2023-10-17T18:40:32.083 DEBUG:boto:Token: None 2023-10-17T18:40:32.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/obj4 2023-10-17T18:40:32.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yolMYNOytiscSU6HftYyaNEAJWo= 2023-10-17T18:40:32.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yolMYNOytiscSU6HftYyaNEAJWo=', 'Content-Length': '0'} 2023-10-17T18:40:32.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062e8bee230f68106-00652ed520-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.087 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:32.088 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:32.088 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.088 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.089 DEBUG:boto:Method: GET 2023-10-17T18:40:32.089 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.090 DEBUG:boto:Data: 2023-10-17T18:40:32.090 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:32.091 DEBUG:boto:Port: 8001 2023-10-17T18:40:32.091 DEBUG:boto:Params: {} 2023-10-17T18:40:32.092 DEBUG:boto:Token: None 2023-10-17T18:40:32.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/ 2023-10-17T18:40:32.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28= 2023-10-17T18:40:32.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28=', 'Content-Length': '0'} 2023-10-17T18:40:32.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35dd20dfb9f3aee-00652ed520-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.138 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:32.139 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:32.139 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:32.140 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.140 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.141 DEBUG:boto:Method: GET 2023-10-17T18:40:32.141 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:40:32.141 DEBUG:boto:Data: 2023-10-17T18:40:32.142 DEBUG:boto:Headers: {} 2023-10-17T18:40:32.142 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:32.143 DEBUG:boto:Port: 8000 2023-10-17T18:40:32.143 DEBUG:boto:Params: {} 2023-10-17T18:40:32.144 DEBUG:boto:Token: None 2023-10-17T18:40:32.144 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:40:32 GMT /hbmuog-49/ 2023-10-17T18:40:32.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28= 2023-10-17T18:40:32.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAoTFZF0ciFR3Sa+r3fNNrSQF28=', 'Content-Length': '0'} 2023-10-17T18:40:32.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d2f75f8594144f-00652ed520-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:32.149 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:40:32.149 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-49 zones={a2, a1} 2023-10-17T18:40:32.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:32.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:32.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:40:32.205 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:32.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:32.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hbmuog-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:32.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hbmuog-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:32.281 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hbmuog-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:40:32.333 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:32.333 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:32.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:32.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:32.402 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:40:32.452 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:32.452 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:32.482 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:40:32.495 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:40:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568032.387520_587.1", 2023-10-17T18:40:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:32.387520Z" 2023-10-17T18:40:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:40:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:40:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:40:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:40:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.499 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:40:32.500 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568032.387520_587.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:32.500 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:40:32.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:32.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:32.502 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:40:32.552 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:32.553 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:32.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:32.581+0000 7f39a53f0780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:32.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:32.582+0000 7f39a53f0780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:32.595 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:40:32.595 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:40:32.595 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:40:32.595 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:40:32.595 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:40:32.596 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:40:32.596 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:32.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.596 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:40:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:40:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:32.597 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697567864.375672_558.1", 2023-10-17T18:40:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:40:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:37:44.375672Z", 2023-10-17T18:40:32.598 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:32.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:40:32.600 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:32.600 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:32.600 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:40:32.600 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:32.600 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:32.601 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:32.601 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:40:32.601 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:32.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.602 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:40:32.602 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:32.602 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:32.602 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:32.603 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:32.603 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:32.603 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:32.603 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:32.603 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:32.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:32.604 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:40:32.604 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:32.605 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:32.606 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:32.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:32.606 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:40:32.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.607 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:40:32.607 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:40:32.607 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:40:32.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:32.608 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:40:32Z" 2023-10-17T18:40:32.608 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:40:32.609 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697567864.375672_558.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:37:44.375672Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:40:32Z" } 2023-10-17T18:40:32.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697567864.375672_558.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:37:44.375672Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:40:32.610 DEBUG:rgw_multi.tests:log_status={0: '1_1697568032.387520_587.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:32.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1697567864.375672_558.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:32.611 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568032.387520_587.1 target=1_1697567864.375672_558.1 2023-10-17T18:40:37.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:37.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:37.613 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:40:37.672 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:37.672 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:37.705 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:37.705+0000 7fe838258780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:37.706 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:37.706+0000 7fe838258780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:37.719 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:40:37.720 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:40:37.720 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:40:37.720 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:40:37.720 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:40:37.721 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:40:37.721 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:37.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.721 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:40:37.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:37.722 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:40:37.722 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:37.722 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:37.722 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568032.261735_584.1", 2023-10-17T18:40:37.723 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:37.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:40:37.723 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:37.723 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:32.261735Z", 2023-10-17T18:40:37.724 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:37.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:37.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:37.724 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:37.725 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:37.726 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:40:37.726 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:37.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:37.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:37.728 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:37.728 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:37.728 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:37.728 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:37.728 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:37.729 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:37.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:37.729 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:40:37.729 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:37.730 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:37.730 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:37.730 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:37.730 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:37.730 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:40:37.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.732 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:40:37.732 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:40:37.732 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:40:37.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:37.733 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:40:37Z" 2023-10-17T18:40:37.733 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:40:37.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568032.261735_584.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:40:32.261735Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:40:37Z" } 2023-10-17T18:40:37.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568032.261735_584.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:40:32.261735Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:40:37.735 DEBUG:rgw_multi.tests:log_status={0: '1_1697568032.387520_587.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:37.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568032.261735_584.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:37.735 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568032.387520_587.1 target=1_1697568032.261735_584.1 2023-10-17T18:40:42.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:42.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:40:42.738 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:40:42.797 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:42.798 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:42.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:42.831+0000 7fb641469780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:42.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:42.832+0000 7fb641469780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:40:42.846 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:40:42.847 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:40:42.847 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:40:42.847 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:40:42.847 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:40:42.848 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:40:42.848 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:42.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.848 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:40:42.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:42.849 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:40:42.849 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:42.850 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:42.850 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568032.387520_587.1", 2023-10-17T18:40:42.850 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:42.850 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:40:42.851 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:42.851 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:32.387520Z", 2023-10-17T18:40:42.851 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:42.851 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:42.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:42.852 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:40:42.852 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:42.852 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:42.853 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:40:42.853 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:42.853 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:42.853 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:42.853 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:40:42.854 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:42.854 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:42.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:42.854 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:40:42.855 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:42.855 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:42.855 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:42.855 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:42.856 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:42.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:42.856 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:42.856 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:42.856 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:42.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:40:42.857 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:40:42.857 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:40:42.857 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:40:42.858 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:40:42.858 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:40:42.858 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:40:42.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:40:42.859 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:40:42.859 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:40:42.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:42.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:40:42.859 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:40:42.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.860 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:40:42.860 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:40:42.860 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:40:42.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:40:42.861 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:40:42Z" 2023-10-17T18:40:42.861 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:40:42.862 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568032.387520_587.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:40:32.387520Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:40:42Z" } 2023-10-17T18:40:42.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568032.387520_587.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:40:32.387520Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:40:42.863 DEBUG:rgw_multi.tests:log_status={0: '1_1697568032.387520_587.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:42.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568032.387520_587.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:40:42.864 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:40:42.864 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:42.865 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:42.865 DEBUG:boto:Method: HEAD 2023-10-17T18:40:42.865 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:42.866 DEBUG:boto:Data: 2023-10-17T18:40:42.866 DEBUG:boto:Headers: {} 2023-10-17T18:40:42.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.867 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.867 DEBUG:boto:Params: {} 2023-10-17T18:40:42.868 DEBUG:boto:Token: None 2023-10-17T18:40:42.868 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/ 2023-10-17T18:40:42.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo= 2023-10-17T18:40:42.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo=', 'Content-Length': '0'} 2023-10-17T18:40:42.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924b855def1608c6-00652ed52a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.872 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:40:42.873 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:40:42.873 DEBUG:boto:Method: PUT 2023-10-17T18:40:42.874 DEBUG:boto:Path: /hbmuog-48/obj5 2023-10-17T18:40:42.874 DEBUG:boto:Data: 2023-10-17T18:40:42.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:42.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.875 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.876 DEBUG:boto:Params: {} 2023-10-17T18:40:42.876 DEBUG:boto:Token: None 2023-10-17T18:40:42.877 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/obj5 2023-10-17T18:40:42.877 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hzvmxFk0L1qHPj4OOKmaSVKeTYM= 2023-10-17T18:40:42.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hzvmxFk0L1qHPj4OOKmaSVKeTYM='} 2023-10-17T18:40:42.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004be174d0cdf3ca32-00652ed52a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.883 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:42.884 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:42.884 DEBUG:boto:Method: HEAD 2023-10-17T18:40:42.885 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:42.885 DEBUG:boto:Data: 2023-10-17T18:40:42.886 DEBUG:boto:Headers: {} 2023-10-17T18:40:42.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.886 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.887 DEBUG:boto:Params: {} 2023-10-17T18:40:42.887 DEBUG:boto:Token: None 2023-10-17T18:40:42.888 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/ 2023-10-17T18:40:42.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo= 2023-10-17T18:40:42.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo=', 'Content-Length': '0'} 2023-10-17T18:40:42.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb5d0cc366711c19-00652ed52a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.891 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:40:42.892 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:40:42.892 DEBUG:boto:Method: PUT 2023-10-17T18:40:42.893 DEBUG:boto:Path: /hbmuog-48/obj6 2023-10-17T18:40:42.893 DEBUG:boto:Data: 2023-10-17T18:40:42.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:42.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.895 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.895 DEBUG:boto:Params: {} 2023-10-17T18:40:42.895 DEBUG:boto:Token: None 2023-10-17T18:40:42.896 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/obj6 2023-10-17T18:40:42.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gAralYhO2+z/abZ9hsfYh25TR0k= 2023-10-17T18:40:42.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gAralYhO2+z/abZ9hsfYh25TR0k='} 2023-10-17T18:40:42.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ce30b46086880b3-00652ed52a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.943 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:42.944 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:42.944 DEBUG:boto:Method: HEAD 2023-10-17T18:40:42.944 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:42.945 DEBUG:boto:Data: 2023-10-17T18:40:42.945 DEBUG:boto:Headers: {} 2023-10-17T18:40:42.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.946 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.946 DEBUG:boto:Params: {} 2023-10-17T18:40:42.947 DEBUG:boto:Token: None 2023-10-17T18:40:42.947 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/ 2023-10-17T18:40:42.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo= 2023-10-17T18:40:42.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo=', 'Content-Length': '0'} 2023-10-17T18:40:42.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6a669e8f44fa0e4-00652ed52a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.951 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:40:42.952 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:40:42.952 DEBUG:boto:Method: PUT 2023-10-17T18:40:42.953 DEBUG:boto:Path: /hbmuog-48/obj7 2023-10-17T18:40:42.953 DEBUG:boto:Data: 2023-10-17T18:40:42.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:42.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.954 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.955 DEBUG:boto:Params: {} 2023-10-17T18:40:42.955 DEBUG:boto:Token: None 2023-10-17T18:40:42.956 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/obj7 2023-10-17T18:40:42.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DGRqtV7W9SpXvZbyyHH6RZ6t92I= 2023-10-17T18:40:42.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DGRqtV7W9SpXvZbyyHH6RZ6t92I='} 2023-10-17T18:40:42.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aba5abbae4c7124f-00652ed52a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.962 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:40:42.963 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:40:42.963 DEBUG:boto:Method: HEAD 2023-10-17T18:40:42.963 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:40:42.964 DEBUG:boto:Data: 2023-10-17T18:40:42.964 DEBUG:boto:Headers: {} 2023-10-17T18:40:42.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.965 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.965 DEBUG:boto:Params: {} 2023-10-17T18:40:42.966 DEBUG:boto:Token: None 2023-10-17T18:40:42.966 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/ 2023-10-17T18:40:42.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo= 2023-10-17T18:40:42.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SWIhx7/NCAYfFqhmZl+MzYAKCFo=', 'Content-Length': '0'} 2023-10-17T18:40:42.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f114c21d4f58fe73-00652ed52a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:42.970 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:40:42.970 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:40:42.971 DEBUG:boto:Method: PUT 2023-10-17T18:40:42.971 DEBUG:boto:Path: /hbmuog-48/obj8 2023-10-17T18:40:42.972 DEBUG:boto:Data: 2023-10-17T18:40:42.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:42.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:40:42.973 DEBUG:boto:Port: 8000 2023-10-17T18:40:42.973 DEBUG:boto:Params: {} 2023-10-17T18:40:42.974 DEBUG:boto:Token: None 2023-10-17T18:40:42.974 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:42 GMT /hbmuog-48/obj8 2023-10-17T18:40:42.975 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HWPrTpb7mG6VME/feAYEswugXHQ= 2023-10-17T18:40:42.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HWPrTpb7mG6VME/feAYEswugXHQ='} 2023-10-17T18:40:43.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b4572444b8d2775-00652ed52a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.019 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:43.020 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:43.020 DEBUG:boto:Method: HEAD 2023-10-17T18:40:43.020 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:43.021 DEBUG:boto:Data: 2023-10-17T18:40:43.021 DEBUG:boto:Headers: {} 2023-10-17T18:40:43.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.022 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.022 DEBUG:boto:Params: {} 2023-10-17T18:40:43.023 DEBUG:boto:Token: None 2023-10-17T18:40:43.023 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/ 2023-10-17T18:40:43.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0= 2023-10-17T18:40:43.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0=', 'Content-Length': '0'} 2023-10-17T18:40:43.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a96a23ea49234ab-00652ed52b-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.027 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:40:43.028 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:40:43.028 DEBUG:boto:Method: PUT 2023-10-17T18:40:43.028 DEBUG:boto:Path: /hbmuog-49/obj5 2023-10-17T18:40:43.029 DEBUG:boto:Data: 2023-10-17T18:40:43.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:43.029 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.030 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.030 DEBUG:boto:Params: {} 2023-10-17T18:40:43.031 DEBUG:boto:Token: None 2023-10-17T18:40:43.031 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/obj5 2023-10-17T18:40:43.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rOKoSpdz93IV9FHmDp8/VEtOobQ= 2023-10-17T18:40:43.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rOKoSpdz93IV9FHmDp8/VEtOobQ='} 2023-10-17T18:40:43.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efe353c1ab9602a1-00652ed52b-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.037 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:43.037 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:43.037 DEBUG:boto:Method: HEAD 2023-10-17T18:40:43.038 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:43.038 DEBUG:boto:Data: 2023-10-17T18:40:43.039 DEBUG:boto:Headers: {} 2023-10-17T18:40:43.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.039 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.040 DEBUG:boto:Params: {} 2023-10-17T18:40:43.040 DEBUG:boto:Token: None 2023-10-17T18:40:43.041 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/ 2023-10-17T18:40:43.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0= 2023-10-17T18:40:43.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0=', 'Content-Length': '0'} 2023-10-17T18:40:43.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bee5d08d48298267-00652ed52b-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.044 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:40:43.045 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:40:43.045 DEBUG:boto:Method: PUT 2023-10-17T18:40:43.046 DEBUG:boto:Path: /hbmuog-49/obj6 2023-10-17T18:40:43.046 DEBUG:boto:Data: 2023-10-17T18:40:43.046 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:43.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.047 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.048 DEBUG:boto:Params: {} 2023-10-17T18:40:43.048 DEBUG:boto:Token: None 2023-10-17T18:40:43.048 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/obj6 2023-10-17T18:40:43.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L1wawC60vAsVHpRSUgO2CfnvkN4= 2023-10-17T18:40:43.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L1wawC60vAsVHpRSUgO2CfnvkN4='} 2023-10-17T18:40:43.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f39e963692d2c01-00652ed52b-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.054 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:43.054 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:43.055 DEBUG:boto:Method: HEAD 2023-10-17T18:40:43.055 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:43.055 DEBUG:boto:Data: 2023-10-17T18:40:43.056 DEBUG:boto:Headers: {} 2023-10-17T18:40:43.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.056 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.057 DEBUG:boto:Params: {} 2023-10-17T18:40:43.057 DEBUG:boto:Token: None 2023-10-17T18:40:43.058 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/ 2023-10-17T18:40:43.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0= 2023-10-17T18:40:43.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0=', 'Content-Length': '0'} 2023-10-17T18:40:43.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000787abc0e21b78b84-00652ed52b-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.062 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:40:43.062 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:40:43.063 DEBUG:boto:Method: PUT 2023-10-17T18:40:43.063 DEBUG:boto:Path: /hbmuog-49/obj7 2023-10-17T18:40:43.063 DEBUG:boto:Data: 2023-10-17T18:40:43.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:43.064 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.065 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.065 DEBUG:boto:Params: {} 2023-10-17T18:40:43.065 DEBUG:boto:Token: None 2023-10-17T18:40:43.066 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/obj7 2023-10-17T18:40:43.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ydng4feeOsIijjcoSPL6dXuvMvs= 2023-10-17T18:40:43.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ydng4feeOsIijjcoSPL6dXuvMvs='} 2023-10-17T18:40:43.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000454398af9cad3edf-00652ed52b-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.071 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:40:43.071 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:40:43.072 DEBUG:boto:Method: HEAD 2023-10-17T18:40:43.072 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:40:43.072 DEBUG:boto:Data: 2023-10-17T18:40:43.073 DEBUG:boto:Headers: {} 2023-10-17T18:40:43.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.074 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.074 DEBUG:boto:Params: {} 2023-10-17T18:40:43.074 DEBUG:boto:Token: None 2023-10-17T18:40:43.075 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/ 2023-10-17T18:40:43.075 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0= 2023-10-17T18:40:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4c0BKCziNhHQWT3ngBoNvzccFI0=', 'Content-Length': '0'} 2023-10-17T18:40:43.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295803b4fd575f06-00652ed52b-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.079 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:40:43.079 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:40:43.080 DEBUG:boto:Method: PUT 2023-10-17T18:40:43.080 DEBUG:boto:Path: /hbmuog-49/obj8 2023-10-17T18:40:43.080 DEBUG:boto:Data: 2023-10-17T18:40:43.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T18:40:43.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:40:43.082 DEBUG:boto:Port: 8001 2023-10-17T18:40:43.082 DEBUG:boto:Params: {} 2023-10-17T18:40:43.082 DEBUG:boto:Token: None 2023-10-17T18:40:43.083 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 18:40:43 GMT /hbmuog-49/obj8 2023-10-17T18:40:43.083 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vRpZEAhYjTq3EQMART4p5xAQXaE= 2023-10-17T18:40:43.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:40:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vRpZEAhYjTq3EQMART4p5xAQXaE='} 2023-10-17T18:40:43.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3f926abc4d29ba3-00652ed52b-1630-a2'), ('Date', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:40:43.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hbmuog-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:43.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hbmuog-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:43.128 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hbmuog-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:40:43.183 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:43.183 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:43.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hbmuog-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:43.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hbmuog-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:40:43.250 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hbmuog-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:40:43.302 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:43.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:43.369 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-48 2023-10-17T18:40:43.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:43.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:40:43.370 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:40:43.429 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:40:43.429 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:40:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:43.464+0000 7f5a14125780 1 waiting to reach incremental sync.. 2023-10-17T18:40:48.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:48.464+0000 7f5a14125780 1 waiting to reach incremental sync.. 2023-10-17T18:40:53.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:53.465+0000 7f5a14125780 1 waiting to reach incremental sync.. 2023-10-17T18:40:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:40:58.465+0000 7f5a14125780 1 waiting to reach incremental sync.. 2023-10-17T18:41:03.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:03.467+0000 7f5a14125780 1 bucket sync caught up with source: 2023-10-17T18:41:03.469 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.27.1, 00000000002.38.1, 00000000004.10.1, 00000000002.34.1, 00000000004.39.1, 00000000002.27.1, 00000000002.28.1, 00000000004.35.1, 00000000002.21.1, 00000000002.28.1, 00000000002.102.1] 2023-10-17T18:41:03.469 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.27.1, 00000000002.38.1, 00000000004.10.1, 00000000002.34.1, 00000000004.39.1, 00000000002.27.1, 00000000002.28.1, 00000000004.35.1, 00000000002.21.1, 00000000002.28.1, 00000000002.102.1] 2023-10-17T18:41:03.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:03.467+0000 7f5a14125780 0 bucket checkpoint complete 2023-10-17T18:41:03.486 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-48 2023-10-17T18:41:03.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:41:03.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:41:03.488 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:41:03.552 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:41:03.553 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:41:03.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:03.590+0000 7fca4d990780 1 waiting for incremental sync to catch up: 2023-10-17T18:41:03.592 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.200.1, 00000000002.205.1, 00000000003.13.1, 00000000002.109.1, 00000000003.208.1, 00000000002.238.1, 00000000002.245.1, 00000000003.274.1, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:03.592 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.206.6, 00000000002.205.1, 00000000004.19.6, 00000000002.109.1, 00000000004.212.6, 00000000002.238.1, 00000000002.245.1, 00000000004.279.6, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:08.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:08.592+0000 7fca4d990780 1 waiting for incremental sync to catch up: 2023-10-17T18:41:08.593 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.200.1, 00000000002.205.1, 00000000003.13.1, 00000000002.109.1, 00000000003.208.1, 00000000002.238.1, 00000000002.245.1, 00000000003.274.1, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:08.594 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.206.6, 00000000002.205.1, 00000000004.19.6, 00000000002.109.1, 00000000004.212.6, 00000000002.238.1, 00000000002.245.1, 00000000004.279.6, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:13.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:13.594+0000 7fca4d990780 1 bucket sync caught up with source: 2023-10-17T18:41:13.595 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.206.6, 00000000002.205.1, 00000000004.19.6, 00000000002.109.1, 00000000004.212.6, 00000000002.238.1, 00000000002.245.1, 00000000004.279.6, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:13.595 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.206.6, 00000000002.205.1, 00000000004.19.6, 00000000002.109.1, 00000000004.212.6, 00000000002.238.1, 00000000002.245.1, 00000000004.279.6, 00000000002.279.1, 00000000002.151.1, 00000000002.394.1] 2023-10-17T18:41:13.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:13.594+0000 7fca4d990780 0 bucket checkpoint complete 2023-10-17T18:41:13.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-48 zones={a2, a1} 2023-10-17T18:41:13.611 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.612 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.612 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.613 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:41:13.613 DEBUG:boto:Data: 2023-10-17T18:41:13.613 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.614 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.615 DEBUG:boto:Params: {} 2023-10-17T18:41:13.615 DEBUG:boto:Token: None 2023-10-17T18:41:13.616 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z9OemAM5YtlysdyS4tDJ3FJD2ME= 2023-10-17T18:41:13.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z9OemAM5YtlysdyS4tDJ3FJD2ME=', 'Content-Length': '0'} 2023-10-17T18:41:13.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd919811cceb016f-00652ed549-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.620 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.620 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.621 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.621 DEBUG:boto:Path: /hbmuog-48/ 2023-10-17T18:41:13.622 DEBUG:boto:Data: 2023-10-17T18:41:13.622 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.623 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.623 DEBUG:boto:Params: {} 2023-10-17T18:41:13.624 DEBUG:boto:Token: None 2023-10-17T18:41:13.624 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z9OemAM5YtlysdyS4tDJ3FJD2ME= 2023-10-17T18:41:13.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z9OemAM5YtlysdyS4tDJ3FJD2ME=', 'Content-Length': '0'} 2023-10-17T18:41:13.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b046f8ad45d1f03-00652ed549-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:41:13.629 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.629 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.629 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.630 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.630 DEBUG:boto:Method: GET 2023-10-17T18:41:13.631 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.631 DEBUG:boto:Data: 2023-10-17T18:41:13.632 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.632 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.633 DEBUG:boto:Params: {} 2023-10-17T18:41:13.633 DEBUG:boto:Token: None 2023-10-17T18:41:13.634 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/?versions 2023-10-17T18:41:13.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw= 2023-10-17T18:41:13.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw=', 'Content-Length': '0'} 2023-10-17T18:41:13.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed78f21008d7ee58-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.637 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:41:13.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:41:13.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:41:13.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:41:13.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:41:13.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:41:13.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:41:13.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:41:13.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:41:13.642 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.643 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.643 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.643 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.644 DEBUG:boto:Method: GET 2023-10-17T18:41:13.644 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.645 DEBUG:boto:Data: 2023-10-17T18:41:13.645 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.646 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.646 DEBUG:boto:Params: {} 2023-10-17T18:41:13.647 DEBUG:boto:Token: None 2023-10-17T18:41:13.647 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/?versions 2023-10-17T18:41:13.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw= 2023-10-17T18:41:13.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw=', 'Content-Length': '0'} 2023-10-17T18:41:13.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29765bef8d6da2d-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.651 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:41:13.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:41:13.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:41:13.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:41:13.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:41:13.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:41:13.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:41:13.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:41:13.655 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.656 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.656 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.657 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.657 DEBUG:boto:Method: GET 2023-10-17T18:41:13.657 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.658 DEBUG:boto:Data: 2023-10-17T18:41:13.658 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.659 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.659 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.659 DEBUG:boto:Params: {} 2023-10-17T18:41:13.660 DEBUG:boto:Token: None 2023-10-17T18:41:13.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/?versions 2023-10-17T18:41:13.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw= 2023-10-17T18:41:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw=', 'Content-Length': '0'} 2023-10-17T18:41:13.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50e5a5cae678362-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.664 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.665 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.666 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.666 DEBUG:boto:path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.666 DEBUG:boto:auth_path=/hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.667 DEBUG:boto:Method: GET 2023-10-17T18:41:13.667 DEBUG:boto:Path: /hbmuog-48/?versions&max-keys=999 2023-10-17T18:41:13.668 DEBUG:boto:Data: 2023-10-17T18:41:13.668 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.669 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.669 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.669 DEBUG:boto:Params: {} 2023-10-17T18:41:13.670 DEBUG:boto:Token: None 2023-10-17T18:41:13.670 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/?versions 2023-10-17T18:41:13.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw= 2023-10-17T18:41:13.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rsCgt1JxNPbmjxSkPkJRaE+rlcw=', 'Content-Length': '0'} 2023-10-17T18:41:13.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6ad98142318c67-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.674 DEBUG:boto:b'hbmuog-48999falseobj1nulltrue2023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:41:13.676 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.676 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.677 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.677 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.677 DEBUG:boto:Method: GET 2023-10-17T18:41:13.678 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.678 DEBUG:boto:Data: 2023-10-17T18:41:13.679 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.680 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.680 DEBUG:boto:Params: {} 2023-10-17T18:41:13.680 DEBUG:boto:Token: None 2023-10-17T18:41:13.681 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0= 2023-10-17T18:41:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0=', 'Content-Length': '0'} 2023-10-17T18:41:13.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008991328790d64a51-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.685 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.685 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.686 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.686 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.686 DEBUG:boto:Method: GET 2023-10-17T18:41:13.687 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.687 DEBUG:boto:Data: 2023-10-17T18:41:13.688 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.689 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.689 DEBUG:boto:Params: {} 2023-10-17T18:41:13.689 DEBUG:boto:Token: None 2023-10-17T18:41:13.690 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0= 2023-10-17T18:41:13.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0=', 'Content-Length': '0'} 2023-10-17T18:41:13.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002abc0d4b1529e7cf-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.693 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.694 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.694 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.695 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.695 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.695 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.696 DEBUG:boto:Data: 2023-10-17T18:41:13.696 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.697 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.697 DEBUG:boto:Params: {} 2023-10-17T18:41:13.698 DEBUG:boto:Token: None 2023-10-17T18:41:13.698 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nFf/Khqv3u4h1x3cbTg6558xDO4= 2023-10-17T18:41:13.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nFf/Khqv3u4h1x3cbTg6558xDO4=', 'Content-Length': '0'} 2023-10-17T18:41:13.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000033fe6a3237e5919-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.702 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.702 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.702 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.703 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.703 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.704 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.704 DEBUG:boto:Data: 2023-10-17T18:41:13.704 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.705 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.705 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.706 DEBUG:boto:Params: {} 2023-10-17T18:41:13.706 DEBUG:boto:Token: None 2023-10-17T18:41:13.707 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nFf/Khqv3u4h1x3cbTg6558xDO4= 2023-10-17T18:41:13.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nFf/Khqv3u4h1x3cbTg6558xDO4=', 'Content-Length': '0'} 2023-10-17T18:41:13.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002781ab64c630a45e-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:41:13.710 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.711 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.711 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.712 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.712 DEBUG:boto:Method: GET 2023-10-17T18:41:13.712 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.713 DEBUG:boto:Data: 2023-10-17T18:41:13.713 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.714 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.714 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.715 DEBUG:boto:Params: {} 2023-10-17T18:41:13.715 DEBUG:boto:Token: None 2023-10-17T18:41:13.716 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.716 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0= 2023-10-17T18:41:13.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0=', 'Content-Length': '0'} 2023-10-17T18:41:13.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012cb96bb1cfb97fb-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.719 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.720 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.720 DEBUG:boto:path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.720 DEBUG:boto:auth_path=/hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.721 DEBUG:boto:Method: GET 2023-10-17T18:41:13.721 DEBUG:boto:Path: /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.722 DEBUG:boto:Data: 2023-10-17T18:41:13.722 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.723 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.723 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.724 DEBUG:boto:Params: {} 2023-10-17T18:41:13.724 DEBUG:boto:Token: None 2023-10-17T18:41:13.725 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1?versionId=null 2023-10-17T18:41:13.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0= 2023-10-17T18:41:13.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dZaWOLjHljUe6srv0eDShODtJN0=', 'Content-Length': '0'} 2023-10-17T18:41:13.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005a659dc4211d509-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.728 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.729 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.729 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.729 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.730 DEBUG:boto:Method: GET 2023-10-17T18:41:13.730 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.731 DEBUG:boto:Data: 2023-10-17T18:41:13.731 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.732 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.732 DEBUG:boto:Params: {} 2023-10-17T18:41:13.733 DEBUG:boto:Token: None 2023-10-17T18:41:13.733 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.734 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000753b85cf58b2b0c-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.737 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.738 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.739 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.739 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.739 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.740 DEBUG:boto:Method: GET 2023-10-17T18:41:13.740 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.741 DEBUG:boto:Data: 2023-10-17T18:41:13.741 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.742 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.742 DEBUG:boto:Params: {} 2023-10-17T18:41:13.743 DEBUG:boto:Token: None 2023-10-17T18:41:13.743 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa3c8d1831b9e08-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.747 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.748 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.748 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.749 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.749 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:41:13.749 DEBUG:boto:Data: 2023-10-17T18:41:13.750 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.751 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.751 DEBUG:boto:Params: {} 2023-10-17T18:41:13.752 DEBUG:boto:Token: None 2023-10-17T18:41:13.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1 2023-10-17T18:41:13.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sijtM361jpgJNt/g9fMJRNSN06o= 2023-10-17T18:41:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sijtM361jpgJNt/g9fMJRNSN06o=', 'Content-Length': '0'} 2023-10-17T18:41:13.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011af7f67eaed5fda-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.755 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.756 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.756 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.757 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:41:13.757 DEBUG:boto:Data: 2023-10-17T18:41:13.758 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.758 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.759 DEBUG:boto:Params: {} 2023-10-17T18:41:13.759 DEBUG:boto:Token: None 2023-10-17T18:41:13.760 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1 2023-10-17T18:41:13.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sijtM361jpgJNt/g9fMJRNSN06o= 2023-10-17T18:41:13.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sijtM361jpgJNt/g9fMJRNSN06o=', 'Content-Length': '0'} 2023-10-17T18:41:13.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009918369224ef266d-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:41:13.764 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.764 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.765 DEBUG:boto:Method: GET 2023-10-17T18:41:13.765 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:41:13.765 DEBUG:boto:Data: 2023-10-17T18:41:13.766 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.767 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.767 DEBUG:boto:Params: {} 2023-10-17T18:41:13.768 DEBUG:boto:Token: None 2023-10-17T18:41:13.768 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1 2023-10-17T18:41:13.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pi0q2F+s4omXRbwFnbA2f7LijuU= 2023-10-17T18:41:13.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pi0q2F+s4omXRbwFnbA2f7LijuU=', 'Content-Length': '0'} 2023-10-17T18:41:13.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017552d1a54b3f2be-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.814 DEBUG:boto:path=/hbmuog-48/obj1 2023-10-17T18:41:13.815 DEBUG:boto:auth_path=/hbmuog-48/obj1 2023-10-17T18:41:13.816 DEBUG:boto:Method: GET 2023-10-17T18:41:13.816 DEBUG:boto:Path: /hbmuog-48/obj1 2023-10-17T18:41:13.816 DEBUG:boto:Data: 2023-10-17T18:41:13.817 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.818 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.818 DEBUG:boto:Params: {} 2023-10-17T18:41:13.819 DEBUG:boto:Token: None 2023-10-17T18:41:13.819 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj1 2023-10-17T18:41:13.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pi0q2F+s4omXRbwFnbA2f7LijuU= 2023-10-17T18:41:13.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pi0q2F+s4omXRbwFnbA2f7LijuU=', 'Content-Length': '0'} 2023-10-17T18:41:13.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066ea2b5d13e2d1b0-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.823 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.823 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.824 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.824 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.825 DEBUG:boto:Method: GET 2023-10-17T18:41:13.825 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.826 DEBUG:boto:Data: 2023-10-17T18:41:13.826 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.827 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.827 DEBUG:boto:Params: {} 2023-10-17T18:41:13.828 DEBUG:boto:Token: None 2023-10-17T18:41:13.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.829 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004194c59e16224389-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.832 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.833 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.834 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.834 DEBUG:boto:path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.835 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.835 DEBUG:boto:Method: GET 2023-10-17T18:41:13.835 DEBUG:boto:Path: /hbmuog-48/?prefix=obj1 2023-10-17T18:41:13.836 DEBUG:boto:Data: 2023-10-17T18:41:13.836 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.837 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.837 DEBUG:boto:Params: {} 2023-10-17T18:41:13.838 DEBUG:boto:Token: None 2023-10-17T18:41:13.838 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f871b66a045708-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.842 DEBUG:boto:b'hbmuog-48obj11000falseobj12023-10-17T18:37:44.400Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:41:13.843 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.844 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.844 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.845 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.845 DEBUG:boto:Method: GET 2023-10-17T18:41:13.845 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.846 DEBUG:boto:Data: 2023-10-17T18:41:13.846 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.847 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.848 DEBUG:boto:Params: {} 2023-10-17T18:41:13.848 DEBUG:boto:Token: None 2023-10-17T18:41:13.849 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus= 2023-10-17T18:41:13.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus=', 'Content-Length': '0'} 2023-10-17T18:41:13.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b470c3b2dcbff104-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.852 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.853 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.853 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.854 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.854 DEBUG:boto:Method: GET 2023-10-17T18:41:13.855 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.855 DEBUG:boto:Data: 2023-10-17T18:41:13.855 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.856 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.857 DEBUG:boto:Params: {} 2023-10-17T18:41:13.857 DEBUG:boto:Token: None 2023-10-17T18:41:13.858 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus= 2023-10-17T18:41:13.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus=', 'Content-Length': '0'} 2023-10-17T18:41:13.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030f723979ea785d0-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.861 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.862 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.862 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.863 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.863 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.864 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.864 DEBUG:boto:Data: 2023-10-17T18:41:13.864 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.865 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.866 DEBUG:boto:Params: {} 2023-10-17T18:41:13.866 DEBUG:boto:Token: None 2023-10-17T18:41:13.866 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OdMNnSu38L+145/d0kpsCK3pBjo= 2023-10-17T18:41:13.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OdMNnSu38L+145/d0kpsCK3pBjo=', 'Content-Length': '0'} 2023-10-17T18:41:13.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000355329eaf87e18ec-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.870 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.870 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.871 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.871 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.871 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.872 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.872 DEBUG:boto:Data: 2023-10-17T18:41:13.873 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.873 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.873 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.874 DEBUG:boto:Params: {} 2023-10-17T18:41:13.874 DEBUG:boto:Token: None 2023-10-17T18:41:13.875 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OdMNnSu38L+145/d0kpsCK3pBjo= 2023-10-17T18:41:13.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OdMNnSu38L+145/d0kpsCK3pBjo=', 'Content-Length': '0'} 2023-10-17T18:41:13.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba8ff9e49a8137c0-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:41:13.878 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.879 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.879 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.880 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.880 DEBUG:boto:Method: GET 2023-10-17T18:41:13.880 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.881 DEBUG:boto:Data: 2023-10-17T18:41:13.881 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.882 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.883 DEBUG:boto:Params: {} 2023-10-17T18:41:13.883 DEBUG:boto:Token: None 2023-10-17T18:41:13.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus= 2023-10-17T18:41:13.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus=', 'Content-Length': '0'} 2023-10-17T18:41:13.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021bc35fe10093b52-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.887 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.888 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.888 DEBUG:boto:path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.888 DEBUG:boto:auth_path=/hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.889 DEBUG:boto:Method: GET 2023-10-17T18:41:13.889 DEBUG:boto:Path: /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.890 DEBUG:boto:Data: 2023-10-17T18:41:13.890 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.891 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.892 DEBUG:boto:Params: {} 2023-10-17T18:41:13.892 DEBUG:boto:Token: None 2023-10-17T18:41:13.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2?versionId=null 2023-10-17T18:41:13.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus= 2023-10-17T18:41:13.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PfmwsP4SFof/T56FEyDLuSp1yus=', 'Content-Length': '0'} 2023-10-17T18:41:13.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0aacf4b9976e470-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.896 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.897 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.897 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.897 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.898 DEBUG:boto:Method: GET 2023-10-17T18:41:13.898 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.899 DEBUG:boto:Data: 2023-10-17T18:41:13.899 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.900 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.900 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.901 DEBUG:boto:Params: {} 2023-10-17T18:41:13.901 DEBUG:boto:Token: None 2023-10-17T18:41:13.902 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.902 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de71c97306fdca8c-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.905 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.906 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.907 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.907 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.908 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.908 DEBUG:boto:Method: GET 2023-10-17T18:41:13.908 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.909 DEBUG:boto:Data: 2023-10-17T18:41:13.909 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.910 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.911 DEBUG:boto:Params: {} 2023-10-17T18:41:13.911 DEBUG:boto:Token: None 2023-10-17T18:41:13.912 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dcf7e2ceb8d38e3-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.915 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.916 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.917 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.917 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.917 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:41:13.918 DEBUG:boto:Data: 2023-10-17T18:41:13.918 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.919 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.919 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.920 DEBUG:boto:Params: {} 2023-10-17T18:41:13.920 DEBUG:boto:Token: None 2023-10-17T18:41:13.921 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2 2023-10-17T18:41:13.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CJ+ZX5kfj++lDCjq/5Y14XisYAs= 2023-10-17T18:41:13.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CJ+ZX5kfj++lDCjq/5Y14XisYAs=', 'Content-Length': '0'} 2023-10-17T18:41:13.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007de3c63a724905c8-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.924 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.925 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.925 DEBUG:boto:Method: HEAD 2023-10-17T18:41:13.925 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:41:13.926 DEBUG:boto:Data: 2023-10-17T18:41:13.926 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.927 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.927 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.928 DEBUG:boto:Params: {} 2023-10-17T18:41:13.928 DEBUG:boto:Token: None 2023-10-17T18:41:13.929 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2 2023-10-17T18:41:13.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CJ+ZX5kfj++lDCjq/5Y14XisYAs= 2023-10-17T18:41:13.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CJ+ZX5kfj++lDCjq/5Y14XisYAs=', 'Content-Length': '0'} 2023-10-17T18:41:13.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f3f7532e1514b08-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:41:13.932 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.933 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.933 DEBUG:boto:Method: GET 2023-10-17T18:41:13.934 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:41:13.934 DEBUG:boto:Data: 2023-10-17T18:41:13.935 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.935 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.935 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.936 DEBUG:boto:Params: {} 2023-10-17T18:41:13.936 DEBUG:boto:Token: None 2023-10-17T18:41:13.937 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2 2023-10-17T18:41:13.937 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:umrYIVagL2h9KxxSFPotws07D04= 2023-10-17T18:41:13.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:umrYIVagL2h9KxxSFPotws07D04=', 'Content-Length': '0'} 2023-10-17T18:41:13.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4075700cba875bc-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.940 DEBUG:boto:path=/hbmuog-48/obj2 2023-10-17T18:41:13.940 DEBUG:boto:auth_path=/hbmuog-48/obj2 2023-10-17T18:41:13.941 DEBUG:boto:Method: GET 2023-10-17T18:41:13.941 DEBUG:boto:Path: /hbmuog-48/obj2 2023-10-17T18:41:13.942 DEBUG:boto:Data: 2023-10-17T18:41:13.942 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.943 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.943 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.943 DEBUG:boto:Params: {} 2023-10-17T18:41:13.944 DEBUG:boto:Token: None 2023-10-17T18:41:13.944 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj2 2023-10-17T18:41:13.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:umrYIVagL2h9KxxSFPotws07D04= 2023-10-17T18:41:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:umrYIVagL2h9KxxSFPotws07D04=', 'Content-Length': '0'} 2023-10-17T18:41:13.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c38a9be3f3da14f8-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.948 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.948 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.949 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.949 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.950 DEBUG:boto:Method: GET 2023-10-17T18:41:13.950 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.951 DEBUG:boto:Data: 2023-10-17T18:41:13.951 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.952 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.953 DEBUG:boto:Params: {} 2023-10-17T18:41:13.953 DEBUG:boto:Token: None 2023-10-17T18:41:13.954 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2fcbc5e517637b9-00652ed549-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.957 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.958 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:13.959 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:13.959 DEBUG:boto:path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.959 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.960 DEBUG:boto:Method: GET 2023-10-17T18:41:13.960 DEBUG:boto:Path: /hbmuog-48/?prefix=obj2 2023-10-17T18:41:13.961 DEBUG:boto:Data: 2023-10-17T18:41:13.961 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.962 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.962 DEBUG:boto:Params: {} 2023-10-17T18:41:13.963 DEBUG:boto:Token: None 2023-10-17T18:41:13.963 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/ 2023-10-17T18:41:13.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc= 2023-10-17T18:41:13.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ujXDC8TaRg7ehb3mMdSwsc9bqzc=', 'Content-Length': '0'} 2023-10-17T18:41:13.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d5699b0abaf1471-00652ed549-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.967 DEBUG:boto:b'hbmuog-48obj21000falseobj22023-10-17T18:37:44.418Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:13.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:41:13.968 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:13.969 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:13.969 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.970 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.970 DEBUG:boto:Method: GET 2023-10-17T18:41:13.971 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.971 DEBUG:boto:Data: 2023-10-17T18:41:13.971 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:13.972 DEBUG:boto:Port: 8001 2023-10-17T18:41:13.973 DEBUG:boto:Params: {} 2023-10-17T18:41:13.973 DEBUG:boto:Token: None 2023-10-17T18:41:13.974 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0H6nV6WYtNH3mo1ccSxXxlOp/8= 2023-10-17T18:41:13.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0H6nV6WYtNH3mo1ccSxXxlOp/8=', 'Content-Length': '0'} 2023-10-17T18:41:13.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d1b6646af43ae18-00652ed549-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:13.977 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:13.978 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:13.978 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.979 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.979 DEBUG:boto:Method: GET 2023-10-17T18:41:13.979 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.980 DEBUG:boto:Data: 2023-10-17T18:41:13.980 DEBUG:boto:Headers: {} 2023-10-17T18:41:13.981 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:13.981 DEBUG:boto:Port: 8000 2023-10-17T18:41:13.981 DEBUG:boto:Params: {} 2023-10-17T18:41:13.982 DEBUG:boto:Token: None 2023-10-17T18:41:13.982 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:13 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:13.983 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X0H6nV6WYtNH3mo1ccSxXxlOp/8= 2023-10-17T18:41:13.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X0H6nV6WYtNH3mo1ccSxXxlOp/8=', 'Content-Length': '0'} 2023-10-17T18:41:13.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006450ede369219bc1-00652ed549-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.026 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.027 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.027 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.028 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.028 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.029 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.029 DEBUG:boto:Data: 2023-10-17T18:41:14.029 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.030 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.031 DEBUG:boto:Params: {} 2023-10-17T18:41:14.031 DEBUG:boto:Token: None 2023-10-17T18:41:14.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JDAnFic/nTdJrsBs5yJwXKqqK3A= 2023-10-17T18:41:14.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JDAnFic/nTdJrsBs5yJwXKqqK3A=', 'Content-Length': '0'} 2023-10-17T18:41:14.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000557cb4e7069fde0e-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.035 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.036 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.036 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.036 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.037 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.037 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.038 DEBUG:boto:Data: 2023-10-17T18:41:14.038 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.039 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.039 DEBUG:boto:Params: {} 2023-10-17T18:41:14.040 DEBUG:boto:Token: None 2023-10-17T18:41:14.040 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JDAnFic/nTdJrsBs5yJwXKqqK3A= 2023-10-17T18:41:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JDAnFic/nTdJrsBs5yJwXKqqK3A=', 'Content-Length': '0'} 2023-10-17T18:41:14.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3d6d6dfec128900-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:41:14.044 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.044 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.045 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.045 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.046 DEBUG:boto:Method: GET 2023-10-17T18:41:14.046 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.047 DEBUG:boto:Data: 2023-10-17T18:41:14.047 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.048 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.048 DEBUG:boto:Params: {} 2023-10-17T18:41:14.049 DEBUG:boto:Token: None 2023-10-17T18:41:14.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C0fO4Uin8GONmfkD+olpvAa0qjU= 2023-10-17T18:41:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C0fO4Uin8GONmfkD+olpvAa0qjU=', 'Content-Length': '0'} 2023-10-17T18:41:14.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044f271affe433a31-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.094 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.095 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.095 DEBUG:boto:path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.096 DEBUG:boto:auth_path=/hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.096 DEBUG:boto:Method: GET 2023-10-17T18:41:14.097 DEBUG:boto:Path: /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.097 DEBUG:boto:Data: 2023-10-17T18:41:14.098 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.099 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.099 DEBUG:boto:Params: {} 2023-10-17T18:41:14.100 DEBUG:boto:Token: None 2023-10-17T18:41:14.100 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3?versionId=null 2023-10-17T18:41:14.101 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C0fO4Uin8GONmfkD+olpvAa0qjU= 2023-10-17T18:41:14.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C0fO4Uin8GONmfkD+olpvAa0qjU=', 'Content-Length': '0'} 2023-10-17T18:41:14.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000c6e6c801975042-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.104 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.105 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.105 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.105 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.106 DEBUG:boto:Method: GET 2023-10-17T18:41:14.106 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.107 DEBUG:boto:Data: 2023-10-17T18:41:14.107 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.108 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.108 DEBUG:boto:Params: {} 2023-10-17T18:41:14.109 DEBUG:boto:Token: None 2023-10-17T18:41:14.109 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.110 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f3b5460e35468b9-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.113 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.114 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.115 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.115 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.115 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.116 DEBUG:boto:Method: GET 2023-10-17T18:41:14.116 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.117 DEBUG:boto:Data: 2023-10-17T18:41:14.117 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.118 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.118 DEBUG:boto:Params: {} 2023-10-17T18:41:14.119 DEBUG:boto:Token: None 2023-10-17T18:41:14.119 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e65585765811317-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.124 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.124 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.125 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.125 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.126 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:41:14.126 DEBUG:boto:Data: 2023-10-17T18:41:14.127 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.127 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.128 DEBUG:boto:Params: {} 2023-10-17T18:41:14.128 DEBUG:boto:Token: None 2023-10-17T18:41:14.129 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3 2023-10-17T18:41:14.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GyxhowNGJygLqQgRHYIPrBbmZrc= 2023-10-17T18:41:14.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GyxhowNGJygLqQgRHYIPrBbmZrc=', 'Content-Length': '0'} 2023-10-17T18:41:14.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077103c49398ccc1e-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.132 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.133 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.133 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.134 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:41:14.134 DEBUG:boto:Data: 2023-10-17T18:41:14.135 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.135 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.136 DEBUG:boto:Params: {} 2023-10-17T18:41:14.136 DEBUG:boto:Token: None 2023-10-17T18:41:14.137 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3 2023-10-17T18:41:14.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GyxhowNGJygLqQgRHYIPrBbmZrc= 2023-10-17T18:41:14.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GyxhowNGJygLqQgRHYIPrBbmZrc=', 'Content-Length': '0'} 2023-10-17T18:41:14.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c33c3e0cc32daddc-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:41:14.141 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.141 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.141 DEBUG:boto:Method: GET 2023-10-17T18:41:14.142 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:41:14.142 DEBUG:boto:Data: 2023-10-17T18:41:14.143 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.144 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.144 DEBUG:boto:Params: {} 2023-10-17T18:41:14.144 DEBUG:boto:Token: None 2023-10-17T18:41:14.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3 2023-10-17T18:41:14.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ndkdGDBpwmCRMUgFvUKsM55xMto= 2023-10-17T18:41:14.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ndkdGDBpwmCRMUgFvUKsM55xMto=', 'Content-Length': '0'} 2023-10-17T18:41:14.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fa12741bbac9195-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.148 DEBUG:boto:path=/hbmuog-48/obj3 2023-10-17T18:41:14.149 DEBUG:boto:auth_path=/hbmuog-48/obj3 2023-10-17T18:41:14.149 DEBUG:boto:Method: GET 2023-10-17T18:41:14.150 DEBUG:boto:Path: /hbmuog-48/obj3 2023-10-17T18:41:14.150 DEBUG:boto:Data: 2023-10-17T18:41:14.151 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.151 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.152 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.152 DEBUG:boto:Params: {} 2023-10-17T18:41:14.152 DEBUG:boto:Token: None 2023-10-17T18:41:14.153 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj3 2023-10-17T18:41:14.153 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ndkdGDBpwmCRMUgFvUKsM55xMto= 2023-10-17T18:41:14.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ndkdGDBpwmCRMUgFvUKsM55xMto=', 'Content-Length': '0'} 2023-10-17T18:41:14.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7ec7f54ef88bb8a-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.198 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.199 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.199 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.200 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.200 DEBUG:boto:Method: GET 2023-10-17T18:41:14.201 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.201 DEBUG:boto:Data: 2023-10-17T18:41:14.202 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.202 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.203 DEBUG:boto:Params: {} 2023-10-17T18:41:14.203 DEBUG:boto:Token: None 2023-10-17T18:41:14.204 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079dcab3aa56a754c-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.209 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.209 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.210 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.210 DEBUG:boto:path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.211 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.211 DEBUG:boto:Method: GET 2023-10-17T18:41:14.211 DEBUG:boto:Path: /hbmuog-48/?prefix=obj3 2023-10-17T18:41:14.212 DEBUG:boto:Data: 2023-10-17T18:41:14.212 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.213 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.213 DEBUG:boto:Params: {} 2023-10-17T18:41:14.214 DEBUG:boto:Token: None 2023-10-17T18:41:14.214 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed75ce9a6cca8dde-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.219 DEBUG:boto:b'hbmuog-48obj31000falseobj32023-10-17T18:37:44.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:41:14.222 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.222 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.223 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.223 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.224 DEBUG:boto:Method: GET 2023-10-17T18:41:14.224 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.224 DEBUG:boto:Data: 2023-10-17T18:41:14.225 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.226 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.226 DEBUG:boto:Params: {} 2023-10-17T18:41:14.227 DEBUG:boto:Token: None 2023-10-17T18:41:14.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo= 2023-10-17T18:41:14.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo=', 'Content-Length': '0'} 2023-10-17T18:41:14.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000506800b7f66fb642-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.231 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.231 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.232 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.232 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.233 DEBUG:boto:Method: GET 2023-10-17T18:41:14.233 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.234 DEBUG:boto:Data: 2023-10-17T18:41:14.234 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.234 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.235 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.235 DEBUG:boto:Params: {} 2023-10-17T18:41:14.236 DEBUG:boto:Token: None 2023-10-17T18:41:14.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo= 2023-10-17T18:41:14.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo=', 'Content-Length': '0'} 2023-10-17T18:41:14.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006076d3d6b0c1aab1-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.240 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.240 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.241 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.241 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.242 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.242 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.242 DEBUG:boto:Data: 2023-10-17T18:41:14.243 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.243 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.244 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.244 DEBUG:boto:Params: {} 2023-10-17T18:41:14.244 DEBUG:boto:Token: None 2023-10-17T18:41:14.245 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4qdcaMd9l0SC11+vmYXI6Ep2M5I= 2023-10-17T18:41:14.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4qdcaMd9l0SC11+vmYXI6Ep2M5I=', 'Content-Length': '0'} 2023-10-17T18:41:14.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081a066ef8d0ca03c-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.248 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.249 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.249 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.249 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.250 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.250 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.251 DEBUG:boto:Data: 2023-10-17T18:41:14.251 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.252 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.252 DEBUG:boto:Params: {} 2023-10-17T18:41:14.253 DEBUG:boto:Token: None 2023-10-17T18:41:14.253 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4qdcaMd9l0SC11+vmYXI6Ep2M5I= 2023-10-17T18:41:14.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4qdcaMd9l0SC11+vmYXI6Ep2M5I=', 'Content-Length': '0'} 2023-10-17T18:41:14.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095ea24a32670656e-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:41:14.257 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.257 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.258 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.258 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.259 DEBUG:boto:Method: GET 2023-10-17T18:41:14.259 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.259 DEBUG:boto:Data: 2023-10-17T18:41:14.260 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.261 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.261 DEBUG:boto:Params: {} 2023-10-17T18:41:14.262 DEBUG:boto:Token: None 2023-10-17T18:41:14.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo= 2023-10-17T18:41:14.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo=', 'Content-Length': '0'} 2023-10-17T18:41:14.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006551e779b321c5ca-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.306 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.307 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.307 DEBUG:boto:path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.307 DEBUG:boto:auth_path=/hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.308 DEBUG:boto:Method: GET 2023-10-17T18:41:14.308 DEBUG:boto:Path: /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.309 DEBUG:boto:Data: 2023-10-17T18:41:14.309 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.310 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.311 DEBUG:boto:Params: {} 2023-10-17T18:41:14.311 DEBUG:boto:Token: None 2023-10-17T18:41:14.311 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4?versionId=null 2023-10-17T18:41:14.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo= 2023-10-17T18:41:14.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jq5nSdP1uCRY5UkuXEAHGat4Mo=', 'Content-Length': '0'} 2023-10-17T18:41:14.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8cd52005c6b4e3-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.315 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.316 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.316 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.317 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.317 DEBUG:boto:Method: GET 2023-10-17T18:41:14.317 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.318 DEBUG:boto:Data: 2023-10-17T18:41:14.318 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.319 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.320 DEBUG:boto:Params: {} 2023-10-17T18:41:14.320 DEBUG:boto:Token: None 2023-10-17T18:41:14.320 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375a61ccc02ce855-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.325 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.325 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.326 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.326 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.326 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.327 DEBUG:boto:Method: GET 2023-10-17T18:41:14.327 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.328 DEBUG:boto:Data: 2023-10-17T18:41:14.328 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.328 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.329 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.329 DEBUG:boto:Params: {} 2023-10-17T18:41:14.330 DEBUG:boto:Token: None 2023-10-17T18:41:14.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.331 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059a9c3db4a4d5284-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.334 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.335 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.335 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.336 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.336 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:41:14.336 DEBUG:boto:Data: 2023-10-17T18:41:14.337 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.338 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.338 DEBUG:boto:Params: {} 2023-10-17T18:41:14.338 DEBUG:boto:Token: None 2023-10-17T18:41:14.339 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4 2023-10-17T18:41:14.339 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PSy613nvB251j5J+svc5ByVJ25k= 2023-10-17T18:41:14.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PSy613nvB251j5J+svc5ByVJ25k=', 'Content-Length': '0'} 2023-10-17T18:41:14.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b58d3303bc0f659-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.342 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.343 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.343 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.343 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:41:14.344 DEBUG:boto:Data: 2023-10-17T18:41:14.344 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.345 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.345 DEBUG:boto:Params: {} 2023-10-17T18:41:14.346 DEBUG:boto:Token: None 2023-10-17T18:41:14.346 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4 2023-10-17T18:41:14.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PSy613nvB251j5J+svc5ByVJ25k= 2023-10-17T18:41:14.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PSy613nvB251j5J+svc5ByVJ25k=', 'Content-Length': '0'} 2023-10-17T18:41:14.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012f583191d0e0552-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:41:14.350 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.350 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.351 DEBUG:boto:Method: GET 2023-10-17T18:41:14.351 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:41:14.352 DEBUG:boto:Data: 2023-10-17T18:41:14.352 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.353 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.353 DEBUG:boto:Params: {} 2023-10-17T18:41:14.354 DEBUG:boto:Token: None 2023-10-17T18:41:14.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4 2023-10-17T18:41:14.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mM6pYUDkVjusbF6+1PE3pLh6Qdk= 2023-10-17T18:41:14.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mM6pYUDkVjusbF6+1PE3pLh6Qdk=', 'Content-Length': '0'} 2023-10-17T18:41:14.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c930db502880177-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.358 DEBUG:boto:path=/hbmuog-48/obj4 2023-10-17T18:41:14.359 DEBUG:boto:auth_path=/hbmuog-48/obj4 2023-10-17T18:41:14.359 DEBUG:boto:Method: GET 2023-10-17T18:41:14.359 DEBUG:boto:Path: /hbmuog-48/obj4 2023-10-17T18:41:14.360 DEBUG:boto:Data: 2023-10-17T18:41:14.360 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.361 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.361 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.362 DEBUG:boto:Params: {} 2023-10-17T18:41:14.362 DEBUG:boto:Token: None 2023-10-17T18:41:14.363 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj4 2023-10-17T18:41:14.363 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mM6pYUDkVjusbF6+1PE3pLh6Qdk= 2023-10-17T18:41:14.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mM6pYUDkVjusbF6+1PE3pLh6Qdk=', 'Content-Length': '0'} 2023-10-17T18:41:14.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b985a93d86402fc3-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.366 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.367 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.367 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.368 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.368 DEBUG:boto:Method: GET 2023-10-17T18:41:14.368 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.369 DEBUG:boto:Data: 2023-10-17T18:41:14.369 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.370 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.371 DEBUG:boto:Params: {} 2023-10-17T18:41:14.371 DEBUG:boto:Token: None 2023-10-17T18:41:14.372 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca31beac66f6c340-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.376 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.377 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.377 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.378 DEBUG:boto:path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.378 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.379 DEBUG:boto:Method: GET 2023-10-17T18:41:14.379 DEBUG:boto:Path: /hbmuog-48/?prefix=obj4 2023-10-17T18:41:14.380 DEBUG:boto:Data: 2023-10-17T18:41:14.380 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.380 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.381 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.381 DEBUG:boto:Params: {} 2023-10-17T18:41:14.382 DEBUG:boto:Token: None 2023-10-17T18:41:14.382 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf964b93ea5dfbe-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.387 DEBUG:boto:b'hbmuog-48obj41000falseobj42023-10-17T18:37:44.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:41:14.388 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.388 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.389 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.389 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.389 DEBUG:boto:Method: GET 2023-10-17T18:41:14.390 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.390 DEBUG:boto:Data: 2023-10-17T18:41:14.391 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.392 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.392 DEBUG:boto:Params: {} 2023-10-17T18:41:14.392 DEBUG:boto:Token: None 2023-10-17T18:41:14.393 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U= 2023-10-17T18:41:14.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U=', 'Content-Length': '0'} 2023-10-17T18:41:14.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005430ae2abf33b662-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.396 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.397 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.397 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.398 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.398 DEBUG:boto:Method: GET 2023-10-17T18:41:14.399 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.399 DEBUG:boto:Data: 2023-10-17T18:41:14.400 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.401 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.401 DEBUG:boto:Params: {} 2023-10-17T18:41:14.401 DEBUG:boto:Token: None 2023-10-17T18:41:14.402 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U= 2023-10-17T18:41:14.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U=', 'Content-Length': '0'} 2023-10-17T18:41:14.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df26157f7e05eabc-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.405 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.406 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.406 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.407 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.407 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.408 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.408 DEBUG:boto:Data: 2023-10-17T18:41:14.408 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.409 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.410 DEBUG:boto:Params: {} 2023-10-17T18:41:14.410 DEBUG:boto:Token: None 2023-10-17T18:41:14.411 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e9M8/EdmXgQCq/Ly/iXeUVOdHkk= 2023-10-17T18:41:14.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e9M8/EdmXgQCq/Ly/iXeUVOdHkk=', 'Content-Length': '0'} 2023-10-17T18:41:14.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004db7b336f7d04936-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.414 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.415 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.415 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.416 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.416 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.416 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.417 DEBUG:boto:Data: 2023-10-17T18:41:14.417 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.418 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.419 DEBUG:boto:Params: {} 2023-10-17T18:41:14.419 DEBUG:boto:Token: None 2023-10-17T18:41:14.419 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e9M8/EdmXgQCq/Ly/iXeUVOdHkk= 2023-10-17T18:41:14.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e9M8/EdmXgQCq/Ly/iXeUVOdHkk=', 'Content-Length': '0'} 2023-10-17T18:41:14.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4912f7bbd47985f-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:41:14.424 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.424 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.425 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.425 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.425 DEBUG:boto:Method: GET 2023-10-17T18:41:14.426 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.426 DEBUG:boto:Data: 2023-10-17T18:41:14.427 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.428 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.429 DEBUG:boto:Params: {} 2023-10-17T18:41:14.430 DEBUG:boto:Token: None 2023-10-17T18:41:14.430 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.431 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U= 2023-10-17T18:41:14.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U=', 'Content-Length': '0'} 2023-10-17T18:41:14.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c35692ff44425390-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.434 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.435 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.435 DEBUG:boto:path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.436 DEBUG:boto:auth_path=/hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.436 DEBUG:boto:Method: GET 2023-10-17T18:41:14.437 DEBUG:boto:Path: /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.437 DEBUG:boto:Data: 2023-10-17T18:41:14.437 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.438 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.439 DEBUG:boto:Params: {} 2023-10-17T18:41:14.439 DEBUG:boto:Token: None 2023-10-17T18:41:14.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5?versionId=null 2023-10-17T18:41:14.440 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U= 2023-10-17T18:41:14.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PAlR0yOyvFIqsCP69pT2VEKxQ5U=', 'Content-Length': '0'} 2023-10-17T18:41:14.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008416edfeea6c5a18-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.486 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.487 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.487 DEBUG:boto:path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.488 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.488 DEBUG:boto:Method: GET 2023-10-17T18:41:14.488 DEBUG:boto:Path: /hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.489 DEBUG:boto:Data: 2023-10-17T18:41:14.489 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.490 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.491 DEBUG:boto:Params: {} 2023-10-17T18:41:14.491 DEBUG:boto:Token: None 2023-10-17T18:41:14.492 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.492 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000976e8fe0d99820e9-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.496 DEBUG:boto:b'hbmuog-48obj51000falseobj52023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.497 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.498 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.498 DEBUG:boto:path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.498 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.499 DEBUG:boto:Method: GET 2023-10-17T18:41:14.499 DEBUG:boto:Path: /hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.500 DEBUG:boto:Data: 2023-10-17T18:41:14.500 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.501 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.502 DEBUG:boto:Params: {} 2023-10-17T18:41:14.502 DEBUG:boto:Token: None 2023-10-17T18:41:14.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b196e1bb8b22f60c-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.507 DEBUG:boto:b'hbmuog-48obj51000falseobj52023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.508 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.508 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.509 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.509 DEBUG:boto:Path: /hbmuog-48/obj5 2023-10-17T18:41:14.510 DEBUG:boto:Data: 2023-10-17T18:41:14.510 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.510 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.511 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.511 DEBUG:boto:Params: {} 2023-10-17T18:41:14.512 DEBUG:boto:Token: None 2023-10-17T18:41:14.512 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5 2023-10-17T18:41:14.512 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X4lSsZVT4YueTSmNwXIN14NGpPI= 2023-10-17T18:41:14.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X4lSsZVT4YueTSmNwXIN14NGpPI=', 'Content-Length': '0'} 2023-10-17T18:41:14.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d9c083dc76f65c-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.515 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.516 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.516 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.517 DEBUG:boto:Path: /hbmuog-48/obj5 2023-10-17T18:41:14.517 DEBUG:boto:Data: 2023-10-17T18:41:14.517 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.518 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.518 DEBUG:boto:Params: {} 2023-10-17T18:41:14.519 DEBUG:boto:Token: None 2023-10-17T18:41:14.519 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5 2023-10-17T18:41:14.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X4lSsZVT4YueTSmNwXIN14NGpPI= 2023-10-17T18:41:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X4lSsZVT4YueTSmNwXIN14NGpPI=', 'Content-Length': '0'} 2023-10-17T18:41:14.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013b13d7f63b45b34-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:41:14.523 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.524 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.524 DEBUG:boto:Method: GET 2023-10-17T18:41:14.524 DEBUG:boto:Path: /hbmuog-48/obj5 2023-10-17T18:41:14.525 DEBUG:boto:Data: 2023-10-17T18:41:14.525 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.526 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.527 DEBUG:boto:Params: {} 2023-10-17T18:41:14.527 DEBUG:boto:Token: None 2023-10-17T18:41:14.528 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5 2023-10-17T18:41:14.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U5Zwmt6lfSloKm2RNbPVbvs0S5M= 2023-10-17T18:41:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U5Zwmt6lfSloKm2RNbPVbvs0S5M=', 'Content-Length': '0'} 2023-10-17T18:41:14.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376be12bb96956dd-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.574 DEBUG:boto:path=/hbmuog-48/obj5 2023-10-17T18:41:14.575 DEBUG:boto:auth_path=/hbmuog-48/obj5 2023-10-17T18:41:14.575 DEBUG:boto:Method: GET 2023-10-17T18:41:14.575 DEBUG:boto:Path: /hbmuog-48/obj5 2023-10-17T18:41:14.576 DEBUG:boto:Data: 2023-10-17T18:41:14.576 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.577 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.578 DEBUG:boto:Params: {} 2023-10-17T18:41:14.578 DEBUG:boto:Token: None 2023-10-17T18:41:14.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj5 2023-10-17T18:41:14.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U5Zwmt6lfSloKm2RNbPVbvs0S5M= 2023-10-17T18:41:14.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U5Zwmt6lfSloKm2RNbPVbvs0S5M=', 'Content-Length': '0'} 2023-10-17T18:41:14.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008281f9553c2ae2fa-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.582 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.583 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.583 DEBUG:boto:path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.584 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.584 DEBUG:boto:Method: GET 2023-10-17T18:41:14.584 DEBUG:boto:Path: /hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.585 DEBUG:boto:Data: 2023-10-17T18:41:14.585 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.586 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.586 DEBUG:boto:Params: {} 2023-10-17T18:41:14.587 DEBUG:boto:Token: None 2023-10-17T18:41:14.587 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d40a09666dcc4d7-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.591 DEBUG:boto:b'hbmuog-48obj51000falseobj52023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.592 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.592 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.593 DEBUG:boto:path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.593 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.594 DEBUG:boto:Method: GET 2023-10-17T18:41:14.594 DEBUG:boto:Path: /hbmuog-48/?prefix=obj5 2023-10-17T18:41:14.595 DEBUG:boto:Data: 2023-10-17T18:41:14.595 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.596 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.596 DEBUG:boto:Params: {} 2023-10-17T18:41:14.597 DEBUG:boto:Token: None 2023-10-17T18:41:14.597 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb2b4b8de24a037-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.601 DEBUG:boto:b'hbmuog-48obj51000falseobj52023-10-17T18:40:42.879Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:41:14.602 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.603 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.603 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.604 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.604 DEBUG:boto:Method: GET 2023-10-17T18:41:14.604 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.605 DEBUG:boto:Data: 2023-10-17T18:41:14.605 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.606 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.607 DEBUG:boto:Params: {} 2023-10-17T18:41:14.607 DEBUG:boto:Token: None 2023-10-17T18:41:14.608 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.608 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc= 2023-10-17T18:41:14.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc=', 'Content-Length': '0'} 2023-10-17T18:41:14.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f8d677c9ad5538f-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.611 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.612 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.612 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.612 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.613 DEBUG:boto:Method: GET 2023-10-17T18:41:14.613 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.614 DEBUG:boto:Data: 2023-10-17T18:41:14.614 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.615 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.615 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.615 DEBUG:boto:Params: {} 2023-10-17T18:41:14.616 DEBUG:boto:Token: None 2023-10-17T18:41:14.616 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.617 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc= 2023-10-17T18:41:14.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc=', 'Content-Length': '0'} 2023-10-17T18:41:14.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000163936cd33955859-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.620 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.620 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.621 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.621 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.621 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.622 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.622 DEBUG:boto:Data: 2023-10-17T18:41:14.623 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.623 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.624 DEBUG:boto:Params: {} 2023-10-17T18:41:14.624 DEBUG:boto:Token: None 2023-10-17T18:41:14.624 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ub/AQtMtSe/anPIkNvKX4X0blH8= 2023-10-17T18:41:14.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ub/AQtMtSe/anPIkNvKX4X0blH8=', 'Content-Length': '0'} 2023-10-17T18:41:14.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f25d47a6caa4f603-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.628 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.628 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.628 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.629 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.629 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.630 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.630 DEBUG:boto:Data: 2023-10-17T18:41:14.631 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.631 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.631 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.632 DEBUG:boto:Params: {} 2023-10-17T18:41:14.632 DEBUG:boto:Token: None 2023-10-17T18:41:14.633 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.633 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ub/AQtMtSe/anPIkNvKX4X0blH8= 2023-10-17T18:41:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ub/AQtMtSe/anPIkNvKX4X0blH8=', 'Content-Length': '0'} 2023-10-17T18:41:14.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebe0cc92b7adddee-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:41:14.637 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.637 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.638 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.638 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.638 DEBUG:boto:Method: GET 2023-10-17T18:41:14.639 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.639 DEBUG:boto:Data: 2023-10-17T18:41:14.640 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.641 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.641 DEBUG:boto:Params: {} 2023-10-17T18:41:14.642 DEBUG:boto:Token: None 2023-10-17T18:41:14.642 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc= 2023-10-17T18:41:14.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc=', 'Content-Length': '0'} 2023-10-17T18:41:14.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000955eedd6c12829e8-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.686 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.687 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.687 DEBUG:boto:path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.688 DEBUG:boto:auth_path=/hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.688 DEBUG:boto:Method: GET 2023-10-17T18:41:14.689 DEBUG:boto:Path: /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.689 DEBUG:boto:Data: 2023-10-17T18:41:14.690 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.691 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.691 DEBUG:boto:Params: {} 2023-10-17T18:41:14.692 DEBUG:boto:Token: None 2023-10-17T18:41:14.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6?versionId=null 2023-10-17T18:41:14.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc= 2023-10-17T18:41:14.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7uGbgJFeQ5B0jhrlIhi62Kurxcc=', 'Content-Length': '0'} 2023-10-17T18:41:14.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f2427a3c0bd2578-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.696 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.697 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.697 DEBUG:boto:path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.697 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.698 DEBUG:boto:Method: GET 2023-10-17T18:41:14.698 DEBUG:boto:Path: /hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.699 DEBUG:boto:Data: 2023-10-17T18:41:14.699 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.700 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.700 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.701 DEBUG:boto:Params: {} 2023-10-17T18:41:14.701 DEBUG:boto:Token: None 2023-10-17T18:41:14.702 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.703 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca943ce2ea0522e6-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.706 DEBUG:boto:b'hbmuog-48obj61000falseobj62023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.707 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.708 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.708 DEBUG:boto:path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.709 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.709 DEBUG:boto:Method: GET 2023-10-17T18:41:14.710 DEBUG:boto:Path: /hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.710 DEBUG:boto:Data: 2023-10-17T18:41:14.710 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.711 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.712 DEBUG:boto:Params: {} 2023-10-17T18:41:14.712 DEBUG:boto:Token: None 2023-10-17T18:41:14.713 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f968ea25c828f2f3-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.717 DEBUG:boto:b'hbmuog-48obj61000falseobj62023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.718 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.718 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.719 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.719 DEBUG:boto:Path: /hbmuog-48/obj6 2023-10-17T18:41:14.720 DEBUG:boto:Data: 2023-10-17T18:41:14.720 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.721 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.721 DEBUG:boto:Params: {} 2023-10-17T18:41:14.722 DEBUG:boto:Token: None 2023-10-17T18:41:14.722 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6 2023-10-17T18:41:14.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:byHDOfhIrIdXcF8TOZ9DcWV+wp0= 2023-10-17T18:41:14.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:byHDOfhIrIdXcF8TOZ9DcWV+wp0=', 'Content-Length': '0'} 2023-10-17T18:41:14.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db4d259dde902b1b-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.726 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.726 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.727 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.727 DEBUG:boto:Path: /hbmuog-48/obj6 2023-10-17T18:41:14.728 DEBUG:boto:Data: 2023-10-17T18:41:14.728 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.729 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.730 DEBUG:boto:Params: {} 2023-10-17T18:41:14.730 DEBUG:boto:Token: None 2023-10-17T18:41:14.731 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6 2023-10-17T18:41:14.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:byHDOfhIrIdXcF8TOZ9DcWV+wp0= 2023-10-17T18:41:14.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:byHDOfhIrIdXcF8TOZ9DcWV+wp0=', 'Content-Length': '0'} 2023-10-17T18:41:14.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c125b83422aad9-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:41:14.735 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.735 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.736 DEBUG:boto:Method: GET 2023-10-17T18:41:14.736 DEBUG:boto:Path: /hbmuog-48/obj6 2023-10-17T18:41:14.736 DEBUG:boto:Data: 2023-10-17T18:41:14.737 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.738 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.738 DEBUG:boto:Params: {} 2023-10-17T18:41:14.739 DEBUG:boto:Token: None 2023-10-17T18:41:14.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6 2023-10-17T18:41:14.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kgm+Rmi6K5xJK+FwyClKWMOeX78= 2023-10-17T18:41:14.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kgm+Rmi6K5xJK+FwyClKWMOeX78=', 'Content-Length': '0'} 2023-10-17T18:41:14.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000345ff7e0bf96830f-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.743 DEBUG:boto:path=/hbmuog-48/obj6 2023-10-17T18:41:14.743 DEBUG:boto:auth_path=/hbmuog-48/obj6 2023-10-17T18:41:14.744 DEBUG:boto:Method: GET 2023-10-17T18:41:14.744 DEBUG:boto:Path: /hbmuog-48/obj6 2023-10-17T18:41:14.745 DEBUG:boto:Data: 2023-10-17T18:41:14.745 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.746 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.747 DEBUG:boto:Params: {} 2023-10-17T18:41:14.747 DEBUG:boto:Token: None 2023-10-17T18:41:14.748 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj6 2023-10-17T18:41:14.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kgm+Rmi6K5xJK+FwyClKWMOeX78= 2023-10-17T18:41:14.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kgm+Rmi6K5xJK+FwyClKWMOeX78=', 'Content-Length': '0'} 2023-10-17T18:41:14.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb4a0b8707503752-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.751 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.752 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.752 DEBUG:boto:path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.752 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.753 DEBUG:boto:Method: GET 2023-10-17T18:41:14.753 DEBUG:boto:Path: /hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.754 DEBUG:boto:Data: 2023-10-17T18:41:14.754 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.755 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.756 DEBUG:boto:Params: {} 2023-10-17T18:41:14.756 DEBUG:boto:Token: None 2023-10-17T18:41:14.756 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22df3bd2505834c-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.761 DEBUG:boto:b'hbmuog-48obj61000falseobj62023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.761 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.762 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.762 DEBUG:boto:path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.763 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.763 DEBUG:boto:Method: GET 2023-10-17T18:41:14.764 DEBUG:boto:Path: /hbmuog-48/?prefix=obj6 2023-10-17T18:41:14.764 DEBUG:boto:Data: 2023-10-17T18:41:14.764 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.765 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.766 DEBUG:boto:Params: {} 2023-10-17T18:41:14.766 DEBUG:boto:Token: None 2023-10-17T18:41:14.767 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637c07d2b9dca8a2-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.770 DEBUG:boto:b'hbmuog-48obj61000falseobj62023-10-17T18:40:42.899Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:41:14.771 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.772 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.772 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.773 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.773 DEBUG:boto:Method: GET 2023-10-17T18:41:14.774 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.774 DEBUG:boto:Data: 2023-10-17T18:41:14.775 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.775 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.776 DEBUG:boto:Params: {} 2023-10-17T18:41:14.776 DEBUG:boto:Token: None 2023-10-17T18:41:14.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk= 2023-10-17T18:41:14.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk=', 'Content-Length': '0'} 2023-10-17T18:41:14.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000380e88a1a41f21a0-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.781 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.781 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.781 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.782 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.782 DEBUG:boto:Method: GET 2023-10-17T18:41:14.783 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.783 DEBUG:boto:Data: 2023-10-17T18:41:14.784 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.784 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.785 DEBUG:boto:Params: {} 2023-10-17T18:41:14.785 DEBUG:boto:Token: None 2023-10-17T18:41:14.786 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.786 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk= 2023-10-17T18:41:14.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk=', 'Content-Length': '0'} 2023-10-17T18:41:14.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063428d51bf8f3c1d-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.789 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.789 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.790 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.790 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.791 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.791 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.792 DEBUG:boto:Data: 2023-10-17T18:41:14.792 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.793 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.793 DEBUG:boto:Params: {} 2023-10-17T18:41:14.794 DEBUG:boto:Token: None 2023-10-17T18:41:14.794 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hqf93i6JXdvzTLIKObiEXTnN4oc= 2023-10-17T18:41:14.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hqf93i6JXdvzTLIKObiEXTnN4oc=', 'Content-Length': '0'} 2023-10-17T18:41:14.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8f3d0d15a58ae06-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.797 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.798 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.798 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.798 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.799 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.799 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.800 DEBUG:boto:Data: 2023-10-17T18:41:14.800 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.801 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.801 DEBUG:boto:Params: {} 2023-10-17T18:41:14.802 DEBUG:boto:Token: None 2023-10-17T18:41:14.802 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hqf93i6JXdvzTLIKObiEXTnN4oc= 2023-10-17T18:41:14.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hqf93i6JXdvzTLIKObiEXTnN4oc=', 'Content-Length': '0'} 2023-10-17T18:41:14.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216c9854bb57eb81-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:41:14.806 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.806 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.807 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.807 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.808 DEBUG:boto:Method: GET 2023-10-17T18:41:14.808 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.808 DEBUG:boto:Data: 2023-10-17T18:41:14.809 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.810 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.810 DEBUG:boto:Params: {} 2023-10-17T18:41:14.811 DEBUG:boto:Token: None 2023-10-17T18:41:14.811 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk= 2023-10-17T18:41:14.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk=', 'Content-Length': '0'} 2023-10-17T18:41:14.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbda31c2748cbe31-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.814 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.815 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.815 DEBUG:boto:path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.816 DEBUG:boto:auth_path=/hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.816 DEBUG:boto:Method: GET 2023-10-17T18:41:14.816 DEBUG:boto:Path: /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.817 DEBUG:boto:Data: 2023-10-17T18:41:14.817 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.818 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.819 DEBUG:boto:Params: {} 2023-10-17T18:41:14.819 DEBUG:boto:Token: None 2023-10-17T18:41:14.820 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7?versionId=null 2023-10-17T18:41:14.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk= 2023-10-17T18:41:14.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZTidSx/l+/9N20fgPFUGfvLKofk=', 'Content-Length': '0'} 2023-10-17T18:41:14.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017e78dd6e6845f51-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.823 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.823 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.824 DEBUG:boto:path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.824 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.825 DEBUG:boto:Method: GET 2023-10-17T18:41:14.825 DEBUG:boto:Path: /hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.826 DEBUG:boto:Data: 2023-10-17T18:41:14.826 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.827 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.827 DEBUG:boto:Params: {} 2023-10-17T18:41:14.828 DEBUG:boto:Token: None 2023-10-17T18:41:14.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.829 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132a6071399a4a9e-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.832 DEBUG:boto:b'hbmuog-48obj71000falseobj72023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.833 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.833 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.834 DEBUG:boto:path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.834 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.835 DEBUG:boto:Method: GET 2023-10-17T18:41:14.835 DEBUG:boto:Path: /hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.835 DEBUG:boto:Data: 2023-10-17T18:41:14.836 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.837 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.837 DEBUG:boto:Params: {} 2023-10-17T18:41:14.838 DEBUG:boto:Token: None 2023-10-17T18:41:14.838 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095486f3437da6758-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.842 DEBUG:boto:b'hbmuog-48obj71000falseobj72023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.843 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.843 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.843 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.844 DEBUG:boto:Path: /hbmuog-48/obj7 2023-10-17T18:41:14.844 DEBUG:boto:Data: 2023-10-17T18:41:14.845 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.846 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.846 DEBUG:boto:Params: {} 2023-10-17T18:41:14.846 DEBUG:boto:Token: None 2023-10-17T18:41:14.847 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7 2023-10-17T18:41:14.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zBZiSXI0QK/sHygVtKyZCDJlhqg= 2023-10-17T18:41:14.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zBZiSXI0QK/sHygVtKyZCDJlhqg=', 'Content-Length': '0'} 2023-10-17T18:41:14.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009586a662a012e4e3-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.850 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.851 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.851 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.851 DEBUG:boto:Path: /hbmuog-48/obj7 2023-10-17T18:41:14.852 DEBUG:boto:Data: 2023-10-17T18:41:14.852 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.853 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.853 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.853 DEBUG:boto:Params: {} 2023-10-17T18:41:14.854 DEBUG:boto:Token: None 2023-10-17T18:41:14.854 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7 2023-10-17T18:41:14.855 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zBZiSXI0QK/sHygVtKyZCDJlhqg= 2023-10-17T18:41:14.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zBZiSXI0QK/sHygVtKyZCDJlhqg=', 'Content-Length': '0'} 2023-10-17T18:41:14.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be612cc7c2041a48-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:41:14.858 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.859 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.859 DEBUG:boto:Method: GET 2023-10-17T18:41:14.860 DEBUG:boto:Path: /hbmuog-48/obj7 2023-10-17T18:41:14.860 DEBUG:boto:Data: 2023-10-17T18:41:14.860 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.861 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.861 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.862 DEBUG:boto:Params: {} 2023-10-17T18:41:14.862 DEBUG:boto:Token: None 2023-10-17T18:41:14.863 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7 2023-10-17T18:41:14.863 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XENipVhmIJyowY49HA8DeMSTh7A= 2023-10-17T18:41:14.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XENipVhmIJyowY49HA8DeMSTh7A=', 'Content-Length': '0'} 2023-10-17T18:41:14.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e23b0d83a0643a-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.906 DEBUG:boto:path=/hbmuog-48/obj7 2023-10-17T18:41:14.907 DEBUG:boto:auth_path=/hbmuog-48/obj7 2023-10-17T18:41:14.907 DEBUG:boto:Method: GET 2023-10-17T18:41:14.908 DEBUG:boto:Path: /hbmuog-48/obj7 2023-10-17T18:41:14.908 DEBUG:boto:Data: 2023-10-17T18:41:14.908 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.909 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.909 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.910 DEBUG:boto:Params: {} 2023-10-17T18:41:14.910 DEBUG:boto:Token: None 2023-10-17T18:41:14.911 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj7 2023-10-17T18:41:14.911 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XENipVhmIJyowY49HA8DeMSTh7A= 2023-10-17T18:41:14.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XENipVhmIJyowY49HA8DeMSTh7A=', 'Content-Length': '0'} 2023-10-17T18:41:14.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abf9d1890c726a31-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.914 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.915 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.915 DEBUG:boto:path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.915 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.916 DEBUG:boto:Method: GET 2023-10-17T18:41:14.916 DEBUG:boto:Path: /hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.917 DEBUG:boto:Data: 2023-10-17T18:41:14.917 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.918 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.918 DEBUG:boto:Params: {} 2023-10-17T18:41:14.919 DEBUG:boto:Token: None 2023-10-17T18:41:14.919 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3a0764d09098d6-00652ed54a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.923 DEBUG:boto:b'hbmuog-48obj71000falseobj72023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.924 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:14.924 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:14.924 DEBUG:boto:path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.925 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.925 DEBUG:boto:Method: GET 2023-10-17T18:41:14.926 DEBUG:boto:Path: /hbmuog-48/?prefix=obj7 2023-10-17T18:41:14.926 DEBUG:boto:Data: 2023-10-17T18:41:14.926 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.927 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.927 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.928 DEBUG:boto:Params: {} 2023-10-17T18:41:14.928 DEBUG:boto:Token: None 2023-10-17T18:41:14.929 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/ 2023-10-17T18:41:14.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U= 2023-10-17T18:41:14.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gHzXFKX0+FKGWgGMq54YHxkIZ0U=', 'Content-Length': '0'} 2023-10-17T18:41:14.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38748be1f580cda-00652ed54a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.932 DEBUG:boto:b'hbmuog-48obj71000falseobj72023-10-17T18:40:42.958Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:14.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:41:14.933 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:14.934 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:14.934 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.935 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.935 DEBUG:boto:Method: GET 2023-10-17T18:41:14.936 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.936 DEBUG:boto:Data: 2023-10-17T18:41:14.936 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.937 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.938 DEBUG:boto:Params: {} 2023-10-17T18:41:14.938 DEBUG:boto:Token: None 2023-10-17T18:41:14.939 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU= 2023-10-17T18:41:14.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU=', 'Content-Length': '0'} 2023-10-17T18:41:14.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5c5754e76521bb-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.942 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:14.943 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:14.943 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.944 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.944 DEBUG:boto:Method: GET 2023-10-17T18:41:14.945 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.945 DEBUG:boto:Data: 2023-10-17T18:41:14.946 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.946 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.947 DEBUG:boto:Params: {} 2023-10-17T18:41:14.947 DEBUG:boto:Token: None 2023-10-17T18:41:14.948 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU= 2023-10-17T18:41:14.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU=', 'Content-Length': '0'} 2023-10-17T18:41:14.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1f1d43ecf86acfd-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.951 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:14.952 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:14.952 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.952 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.953 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.953 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.953 DEBUG:boto:Data: 2023-10-17T18:41:14.954 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.955 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.955 DEBUG:boto:Params: {} 2023-10-17T18:41:14.956 DEBUG:boto:Token: None 2023-10-17T18:41:14.956 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.957 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b0uyVYxMBxSnpLz1DLKj8kneMkU= 2023-10-17T18:41:14.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b0uyVYxMBxSnpLz1DLKj8kneMkU=', 'Content-Length': '0'} 2023-10-17T18:41:14.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041c4b09529389420-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.960 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:14.960 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:14.960 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.961 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.961 DEBUG:boto:Method: HEAD 2023-10-17T18:41:14.962 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.962 DEBUG:boto:Data: 2023-10-17T18:41:14.962 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.963 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:14.963 DEBUG:boto:Port: 8000 2023-10-17T18:41:14.964 DEBUG:boto:Params: {} 2023-10-17T18:41:14.964 DEBUG:boto:Token: None 2023-10-17T18:41:14.964 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.965 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b0uyVYxMBxSnpLz1DLKj8kneMkU= 2023-10-17T18:41:14.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b0uyVYxMBxSnpLz1DLKj8kneMkU=', 'Content-Length': '0'} 2023-10-17T18:41:14.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff46b0bbf6e32253-00652ed54a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:14.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:41:14.968 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:14.969 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:14.969 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.969 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.970 DEBUG:boto:Method: GET 2023-10-17T18:41:14.970 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.971 DEBUG:boto:Data: 2023-10-17T18:41:14.971 DEBUG:boto:Headers: {} 2023-10-17T18:41:14.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:14.972 DEBUG:boto:Port: 8001 2023-10-17T18:41:14.973 DEBUG:boto:Params: {} 2023-10-17T18:41:14.973 DEBUG:boto:Token: None 2023-10-17T18:41:14.973 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:14 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:14.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU= 2023-10-17T18:41:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iBpII/VQ4KfKqxCPS+k4DnjMtIU=', 'Content-Length': '0'} 2023-10-17T18:41:14.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089ea3c7c3aab48d9-00652ed54a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.018 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:15.019 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:15.019 DEBUG:boto:path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:15.020 DEBUG:boto:auth_path=/hbmuog-48/obj8?versionId=null 2023-10-17T18:41:15.020 DEBUG:boto:Method: GET 2023-10-17T18:41:15.020 DEBUG:boto:Path: /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:15.021 DEBUG:boto:Data: 2023-10-17T18:41:15.021 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:15.022 DEBUG:boto:Port: 8000 2023-10-17T18:41:15.023 DEBUG:boto:Params: {} 2023-10-17T18:41:15.023 DEBUG:boto:Token: None 2023-10-17T18:41:15.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/obj8?versionId=null 2023-10-17T18:41:15.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ovm+C5iTu3P9xHzkI/0Sak2sFsU= 2023-10-17T18:41:15.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ovm+C5iTu3P9xHzkI/0Sak2sFsU=', 'Content-Length': '0'} 2023-10-17T18:41:15.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad76981dd425259-00652ed54b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.027 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:15.028 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:15.028 DEBUG:boto:path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.028 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.029 DEBUG:boto:Method: GET 2023-10-17T18:41:15.029 DEBUG:boto:Path: /hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.030 DEBUG:boto:Data: 2023-10-17T18:41:15.030 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.031 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:15.031 DEBUG:boto:Port: 8001 2023-10-17T18:41:15.031 DEBUG:boto:Params: {} 2023-10-17T18:41:15.032 DEBUG:boto:Token: None 2023-10-17T18:41:15.032 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/ 2023-10-17T18:41:15.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk= 2023-10-17T18:41:15.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk=', 'Content-Length': '0'} 2023-10-17T18:41:15.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236cf16a7d07efeb-00652ed54b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.036 DEBUG:boto:b'hbmuog-48obj81000falseobj82023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:15.037 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:15.037 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:15.038 DEBUG:boto:path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.038 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.039 DEBUG:boto:Method: GET 2023-10-17T18:41:15.039 DEBUG:boto:Path: /hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.040 DEBUG:boto:Data: 2023-10-17T18:41:15.040 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:15.041 DEBUG:boto:Port: 8000 2023-10-17T18:41:15.041 DEBUG:boto:Params: {} 2023-10-17T18:41:15.042 DEBUG:boto:Token: None 2023-10-17T18:41:15.042 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/ 2023-10-17T18:41:15.043 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk= 2023-10-17T18:41:15.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk=', 'Content-Length': '0'} 2023-10-17T18:41:15.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31ae56b292f6a8b-00652ed54b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.046 DEBUG:boto:b'hbmuog-48obj81000falseobj82023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:15.047 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:15.047 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:15.048 DEBUG:boto:Method: HEAD 2023-10-17T18:41:15.048 DEBUG:boto:Path: /hbmuog-48/obj8 2023-10-17T18:41:15.048 DEBUG:boto:Data: 2023-10-17T18:41:15.049 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.049 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:15.050 DEBUG:boto:Port: 8001 2023-10-17T18:41:15.050 DEBUG:boto:Params: {} 2023-10-17T18:41:15.051 DEBUG:boto:Token: None 2023-10-17T18:41:15.051 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/obj8 2023-10-17T18:41:15.051 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zXrS3Vu9xFdYKEVbkBO4QL6btXI= 2023-10-17T18:41:15.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zXrS3Vu9xFdYKEVbkBO4QL6btXI=', 'Content-Length': '0'} 2023-10-17T18:41:15.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f30af374dbc5dc6-00652ed54b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.054 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:15.055 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:15.055 DEBUG:boto:Method: HEAD 2023-10-17T18:41:15.055 DEBUG:boto:Path: /hbmuog-48/obj8 2023-10-17T18:41:15.056 DEBUG:boto:Data: 2023-10-17T18:41:15.056 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:15.057 DEBUG:boto:Port: 8000 2023-10-17T18:41:15.057 DEBUG:boto:Params: {} 2023-10-17T18:41:15.058 DEBUG:boto:Token: None 2023-10-17T18:41:15.058 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/obj8 2023-10-17T18:41:15.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zXrS3Vu9xFdYKEVbkBO4QL6btXI= 2023-10-17T18:41:15.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zXrS3Vu9xFdYKEVbkBO4QL6btXI=', 'Content-Length': '0'} 2023-10-17T18:41:15.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd6ec90cf2eb9ad9-00652ed54b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:41:15.062 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:15.063 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:15.063 DEBUG:boto:Method: GET 2023-10-17T18:41:15.063 DEBUG:boto:Path: /hbmuog-48/obj8 2023-10-17T18:41:15.064 DEBUG:boto:Data: 2023-10-17T18:41:15.064 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:15.065 DEBUG:boto:Port: 8001 2023-10-17T18:41:15.066 DEBUG:boto:Params: {} 2023-10-17T18:41:15.066 DEBUG:boto:Token: None 2023-10-17T18:41:15.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/obj8 2023-10-17T18:41:15.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kAJYlP8XkHIWRMdWxYNtRl5X7q0= 2023-10-17T18:41:15.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kAJYlP8XkHIWRMdWxYNtRl5X7q0=', 'Content-Length': '0'} 2023-10-17T18:41:15.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-48:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000569b18b06678d3f3-00652ed54b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.070 DEBUG:boto:path=/hbmuog-48/obj8 2023-10-17T18:41:15.071 DEBUG:boto:auth_path=/hbmuog-48/obj8 2023-10-17T18:41:15.071 DEBUG:boto:Method: GET 2023-10-17T18:41:15.072 DEBUG:boto:Path: /hbmuog-48/obj8 2023-10-17T18:41:15.072 DEBUG:boto:Data: 2023-10-17T18:41:15.072 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:15.074 DEBUG:boto:Port: 8000 2023-10-17T18:41:15.074 DEBUG:boto:Params: {} 2023-10-17T18:41:15.075 DEBUG:boto:Token: None 2023-10-17T18:41:15.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/obj8 2023-10-17T18:41:15.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kAJYlP8XkHIWRMdWxYNtRl5X7q0= 2023-10-17T18:41:15.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kAJYlP8XkHIWRMdWxYNtRl5X7q0=', 'Content-Length': '0'} 2023-10-17T18:41:15.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a98574ac4b6109d-00652ed54b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.079 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:15.079 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:15.080 DEBUG:boto:path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.080 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.081 DEBUG:boto:Method: GET 2023-10-17T18:41:15.081 DEBUG:boto:Path: /hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.082 DEBUG:boto:Data: 2023-10-17T18:41:15.082 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:41:15.083 DEBUG:boto:Port: 8001 2023-10-17T18:41:15.083 DEBUG:boto:Params: {} 2023-10-17T18:41:15.084 DEBUG:boto:Token: None 2023-10-17T18:41:15.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/ 2023-10-17T18:41:15.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk= 2023-10-17T18:41:15.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk=', 'Content-Length': '0'} 2023-10-17T18:41:15.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006382eed74412bd76-00652ed54b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.088 DEBUG:boto:b'hbmuog-48obj81000falseobj82023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:15.089 DEBUG:boto:path=/hbmuog-48/ 2023-10-17T18:41:15.089 DEBUG:boto:auth_path=/hbmuog-48/ 2023-10-17T18:41:15.090 DEBUG:boto:path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.090 DEBUG:boto:auth_path=/hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.091 DEBUG:boto:Method: GET 2023-10-17T18:41:15.091 DEBUG:boto:Path: /hbmuog-48/?prefix=obj8 2023-10-17T18:41:15.092 DEBUG:boto:Data: 2023-10-17T18:41:15.092 DEBUG:boto:Headers: {} 2023-10-17T18:41:15.092 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:41:15.093 DEBUG:boto:Port: 8000 2023-10-17T18:41:15.093 DEBUG:boto:Params: {} 2023-10-17T18:41:15.094 DEBUG:boto:Token: None 2023-10-17T18:41:15.094 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:41:15 GMT /hbmuog-48/ 2023-10-17T18:41:15.095 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk= 2023-10-17T18:41:15.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:41:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qHga9mTi+JInJvWUShXdFUzA5Vk=', 'Content-Length': '0'} 2023-10-17T18:41:15.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce12abf91fa0e8e-00652ed54b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:41:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:41:15.098 DEBUG:boto:b'hbmuog-48obj81000falseobj82023-10-17T18:40:42.977Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:41:15.099 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-48 zones={a2, a1} 2023-10-17T18:41:15.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-49 2023-10-17T18:41:15.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:41:15.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:41:15.101 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:41:15.164 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:41:15.164 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:41:15.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:15.204+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:20.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:20.204+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:25.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:25.204+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:30.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:30.204+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:35.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:35.205+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:40.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:40.205+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:45.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:45.208+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:50.208+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:41:55.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:41:55.209+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:00.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:00.208+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:05.209+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:10.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:10.210+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:15.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:15.210+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:20.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:20.212+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:25.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:25.213+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:30.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:30.213+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:35.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:35.213+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:40.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:40.213+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:45.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:45.214+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:50.215+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:42:55.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:42:55.216+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:43:00.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:00.217+0000 7f1913df6780 1 waiting to reach incremental sync.. 2023-10-17T18:43:05.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:05.219+0000 7f1913df6780 1 bucket sync caught up with source: 2023-10-17T18:43:05.220 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.23.6, 00000000002.14.1, 00000000004.13.6, 00000000002.48.1, 00000000004.40.6, 00000000002.35.1, 00000000002.36.1, 00000000004.39.6, 00000000002.37.1, 00000000002.40.1, 00000000002.21.1] 2023-10-17T18:43:05.220 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.23.6, 00000000002.14.1, 00000000004.13.6, 00000000002.48.1, 00000000004.40.6, 00000000002.35.1, 00000000002.36.1, 00000000004.39.6, 00000000002.37.1, 00000000002.40.1, 00000000002.21.1] 2023-10-17T18:43:05.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:05.219+0000 7f1913df6780 0 bucket checkpoint complete 2023-10-17T18:43:05.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-49 2023-10-17T18:43:05.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:05.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:05.240 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:43:05.305 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:05.305 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:05.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:05.343+0000 7f2b440f5780 1 bucket sync caught up with source: 2023-10-17T18:43:05.344 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000004.98.1, 00000000002.182.1, 00000000004.135.1, 00000000002.140.1, 00000000004.251.1, 00000000002.106.1, 00000000002.268.1, 00000000004.107.1, 00000000002.108.1, 00000000002.196.1, 00000000002.99.1] 2023-10-17T18:43:05.344 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000004.98.1, 00000000002.182.1, 00000000004.135.1, 00000000002.140.1, 00000000004.251.1, 00000000002.106.1, 00000000002.268.1, 00000000004.107.1, 00000000002.108.1, 00000000002.196.1, 00000000002.99.1] 2023-10-17T18:43:05.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:05.343+0000 7f2b440f5780 0 bucket checkpoint complete 2023-10-17T18:43:05.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-49 zones={a2, a1} 2023-10-17T18:43:05.358 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.358 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.359 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.359 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:43:05.359 DEBUG:boto:Data: 2023-10-17T18:43:05.360 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.360 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.361 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.361 DEBUG:boto:Params: {} 2023-10-17T18:43:05.362 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:43:05.362 DEBUG:boto:Token: None 2023-10-17T18:43:05.363 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.363 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkVJ9aguQVDJ4n8a191Imlt+RWc= 2023-10-17T18:43:05.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkVJ9aguQVDJ4n8a191Imlt+RWc=', 'Content-Length': '0'} 2023-10-17T18:43:05.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1ea2aaba188cb58-00652ed5b9-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.369 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.369 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.370 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.370 DEBUG:boto:Path: /hbmuog-49/ 2023-10-17T18:43:05.370 DEBUG:boto:Data: 2023-10-17T18:43:05.371 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.371 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.372 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.372 DEBUG:boto:Params: {} 2023-10-17T18:43:05.373 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:43:05.373 DEBUG:boto:Token: None 2023-10-17T18:43:05.374 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkVJ9aguQVDJ4n8a191Imlt+RWc= 2023-10-17T18:43:05.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkVJ9aguQVDJ4n8a191Imlt+RWc=', 'Content-Length': '0'} 2023-10-17T18:43:05.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5140b8019b1a085-00652ed5b9-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.379 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:43:05.379 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.380 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.380 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.380 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.381 DEBUG:boto:Method: GET 2023-10-17T18:43:05.381 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.382 DEBUG:boto:Data: 2023-10-17T18:43:05.382 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.383 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.383 DEBUG:boto:Params: {} 2023-10-17T18:43:05.384 DEBUG:boto:Token: None 2023-10-17T18:43:05.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/?versions 2023-10-17T18:43:05.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q= 2023-10-17T18:43:05.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q=', 'Content-Length': '0'} 2023-10-17T18:43:05.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032393f765006b291-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.389 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:43:05.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:43:05.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:43:05.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:43:05.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:43:05.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:43:05.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:43:05.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:43:05.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:43:05.394 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.394 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.394 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.395 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.395 DEBUG:boto:Method: GET 2023-10-17T18:43:05.396 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.396 DEBUG:boto:Data: 2023-10-17T18:43:05.396 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.397 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.398 DEBUG:boto:Params: {} 2023-10-17T18:43:05.398 DEBUG:boto:Token: None 2023-10-17T18:43:05.399 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/?versions 2023-10-17T18:43:05.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q= 2023-10-17T18:43:05.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q=', 'Content-Length': '0'} 2023-10-17T18:43:05.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d064b086c3738245-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.403 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-17T18:43:05.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-17T18:43:05.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-17T18:43:05.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-17T18:43:05.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-17T18:43:05.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-17T18:43:05.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-17T18:43:05.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-17T18:43:05.406 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.407 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.407 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.408 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.408 DEBUG:boto:Method: GET 2023-10-17T18:43:05.408 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.409 DEBUG:boto:Data: 2023-10-17T18:43:05.409 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.410 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.410 DEBUG:boto:Params: {} 2023-10-17T18:43:05.411 DEBUG:boto:Token: None 2023-10-17T18:43:05.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/?versions 2023-10-17T18:43:05.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q= 2023-10-17T18:43:05.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q=', 'Content-Length': '0'} 2023-10-17T18:43:05.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c6853dff93020c-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.415 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.416 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.417 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.417 DEBUG:boto:path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.417 DEBUG:boto:auth_path=/hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.418 DEBUG:boto:Method: GET 2023-10-17T18:43:05.418 DEBUG:boto:Path: /hbmuog-49/?versions&max-keys=999 2023-10-17T18:43:05.418 DEBUG:boto:Data: 2023-10-17T18:43:05.419 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.420 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.420 DEBUG:boto:Params: {} 2023-10-17T18:43:05.420 DEBUG:boto:Token: None 2023-10-17T18:43:05.421 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/?versions 2023-10-17T18:43:05.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q= 2023-10-17T18:43:05.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0qdyEp4KUuqZjzOseykIwPDFA9Q=', 'Content-Length': '0'} 2023-10-17T18:43:05.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c935e66cbe86e25-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.424 DEBUG:boto:b'hbmuog-49999falseobj1nulltrue2023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:43:05.426 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.426 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.426 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.427 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.427 DEBUG:boto:Method: GET 2023-10-17T18:43:05.427 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.428 DEBUG:boto:Data: 2023-10-17T18:43:05.428 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.429 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.429 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.429 DEBUG:boto:Params: {} 2023-10-17T18:43:05.430 DEBUG:boto:Token: None 2023-10-17T18:43:05.430 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.431 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E= 2023-10-17T18:43:05.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E=', 'Content-Length': '0'} 2023-10-17T18:43:05.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a23f53594df75656-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.434 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.434 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.434 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.435 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.435 DEBUG:boto:Method: GET 2023-10-17T18:43:05.435 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.436 DEBUG:boto:Data: 2023-10-17T18:43:05.436 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.437 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.437 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.437 DEBUG:boto:Params: {} 2023-10-17T18:43:05.438 DEBUG:boto:Token: None 2023-10-17T18:43:05.438 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E= 2023-10-17T18:43:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E=', 'Content-Length': '0'} 2023-10-17T18:43:05.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d86d1c46e505f002-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.442 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.442 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.442 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.443 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.443 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.444 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.444 DEBUG:boto:Data: 2023-10-17T18:43:05.444 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.445 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.445 DEBUG:boto:Params: {} 2023-10-17T18:43:05.446 DEBUG:boto:Token: None 2023-10-17T18:43:05.446 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FDnkb6X69JToS3tGj4HN2aq0PDg= 2023-10-17T18:43:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FDnkb6X69JToS3tGj4HN2aq0PDg=', 'Content-Length': '0'} 2023-10-17T18:43:05.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f7c13d4980629b4-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.449 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.449 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.450 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.450 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.451 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.451 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.451 DEBUG:boto:Data: 2023-10-17T18:43:05.452 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.452 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.452 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.453 DEBUG:boto:Params: {} 2023-10-17T18:43:05.453 DEBUG:boto:Token: None 2023-10-17T18:43:05.454 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.454 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FDnkb6X69JToS3tGj4HN2aq0PDg= 2023-10-17T18:43:05.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FDnkb6X69JToS3tGj4HN2aq0PDg=', 'Content-Length': '0'} 2023-10-17T18:43:05.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a467f10ab67d11c-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:43:05.457 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.457 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.458 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.458 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.458 DEBUG:boto:Method: GET 2023-10-17T18:43:05.459 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.459 DEBUG:boto:Data: 2023-10-17T18:43:05.460 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.460 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.460 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.461 DEBUG:boto:Params: {} 2023-10-17T18:43:05.461 DEBUG:boto:Token: None 2023-10-17T18:43:05.462 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.462 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E= 2023-10-17T18:43:05.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E=', 'Content-Length': '0'} 2023-10-17T18:43:05.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017a88ce996d491e9-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.465 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.465 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.466 DEBUG:boto:path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.466 DEBUG:boto:auth_path=/hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.467 DEBUG:boto:Method: GET 2023-10-17T18:43:05.467 DEBUG:boto:Path: /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.467 DEBUG:boto:Data: 2023-10-17T18:43:05.468 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.468 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.469 DEBUG:boto:Params: {} 2023-10-17T18:43:05.469 DEBUG:boto:Token: None 2023-10-17T18:43:05.470 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1?versionId=null 2023-10-17T18:43:05.470 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E= 2023-10-17T18:43:05.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BO1t0pH7iEF9If0XOSSxxNG2H7E=', 'Content-Length': '0'} 2023-10-17T18:43:05.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f4bb6568a6ec50-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.473 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.474 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.474 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.474 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.475 DEBUG:boto:Method: GET 2023-10-17T18:43:05.475 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.476 DEBUG:boto:Data: 2023-10-17T18:43:05.476 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.477 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.477 DEBUG:boto:Params: {} 2023-10-17T18:43:05.478 DEBUG:boto:Token: None 2023-10-17T18:43:05.478 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000963872c04f0c12c6-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.482 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.483 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.483 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.484 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.484 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.484 DEBUG:boto:Method: GET 2023-10-17T18:43:05.485 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.485 DEBUG:boto:Data: 2023-10-17T18:43:05.485 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.486 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.487 DEBUG:boto:Params: {} 2023-10-17T18:43:05.487 DEBUG:boto:Token: None 2023-10-17T18:43:05.487 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce585122d7202ac-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.491 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.492 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.492 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.492 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.493 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:43:05.493 DEBUG:boto:Data: 2023-10-17T18:43:05.493 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.494 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.494 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.495 DEBUG:boto:Params: {} 2023-10-17T18:43:05.495 DEBUG:boto:Token: None 2023-10-17T18:43:05.495 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1 2023-10-17T18:43:05.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Fu4fd/wB5wUzGgaJzoW6sjJKZYc= 2023-10-17T18:43:05.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Fu4fd/wB5wUzGgaJzoW6sjJKZYc=', 'Content-Length': '0'} 2023-10-17T18:43:05.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006175714e44ef56c1-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.498 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.499 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.499 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.499 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:43:05.500 DEBUG:boto:Data: 2023-10-17T18:43:05.500 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.501 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.501 DEBUG:boto:Params: {} 2023-10-17T18:43:05.502 DEBUG:boto:Token: None 2023-10-17T18:43:05.502 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1 2023-10-17T18:43:05.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Fu4fd/wB5wUzGgaJzoW6sjJKZYc= 2023-10-17T18:43:05.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Fu4fd/wB5wUzGgaJzoW6sjJKZYc=', 'Content-Length': '0'} 2023-10-17T18:43:05.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b64f73efdb4e1b91-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-17T18:43:05.506 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.506 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.506 DEBUG:boto:Method: GET 2023-10-17T18:43:05.507 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:43:05.507 DEBUG:boto:Data: 2023-10-17T18:43:05.508 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.508 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.508 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.509 DEBUG:boto:Params: {} 2023-10-17T18:43:05.509 DEBUG:boto:Token: None 2023-10-17T18:43:05.510 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1 2023-10-17T18:43:05.510 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BsogWU8K6h2OTXj9uQtXNY7fBfs= 2023-10-17T18:43:05.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BsogWU8K6h2OTXj9uQtXNY7fBfs=', 'Content-Length': '0'} 2023-10-17T18:43:05.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a03fdaacff799d4-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.513 DEBUG:boto:path=/hbmuog-49/obj1 2023-10-17T18:43:05.513 DEBUG:boto:auth_path=/hbmuog-49/obj1 2023-10-17T18:43:05.513 DEBUG:boto:Method: GET 2023-10-17T18:43:05.514 DEBUG:boto:Path: /hbmuog-49/obj1 2023-10-17T18:43:05.514 DEBUG:boto:Data: 2023-10-17T18:43:05.515 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.515 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.516 DEBUG:boto:Params: {} 2023-10-17T18:43:05.516 DEBUG:boto:Token: None 2023-10-17T18:43:05.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj1 2023-10-17T18:43:05.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BsogWU8K6h2OTXj9uQtXNY7fBfs= 2023-10-17T18:43:05.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BsogWU8K6h2OTXj9uQtXNY7fBfs=', 'Content-Length': '0'} 2023-10-17T18:43:05.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a41d637afa696815-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.562 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.563 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.563 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.564 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.564 DEBUG:boto:Method: GET 2023-10-17T18:43:05.564 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.565 DEBUG:boto:Data: 2023-10-17T18:43:05.565 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.566 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.566 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.566 DEBUG:boto:Params: {} 2023-10-17T18:43:05.567 DEBUG:boto:Token: None 2023-10-17T18:43:05.567 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.568 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b984336846882130-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.571 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.573 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.574 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.574 DEBUG:boto:path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.575 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.575 DEBUG:boto:Method: GET 2023-10-17T18:43:05.576 DEBUG:boto:Path: /hbmuog-49/?prefix=obj1 2023-10-17T18:43:05.576 DEBUG:boto:Data: 2023-10-17T18:43:05.577 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.578 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.578 DEBUG:boto:Params: {} 2023-10-17T18:43:05.579 DEBUG:boto:Token: None 2023-10-17T18:43:05.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038cb3040e46df357-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.583 DEBUG:boto:b'hbmuog-49obj11000falseobj12023-10-17T18:37:44.550Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:43:05.584 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.585 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.585 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.585 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.586 DEBUG:boto:Method: GET 2023-10-17T18:43:05.586 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.587 DEBUG:boto:Data: 2023-10-17T18:43:05.587 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.588 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.588 DEBUG:boto:Params: {} 2023-10-17T18:43:05.588 DEBUG:boto:Token: None 2023-10-17T18:43:05.589 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU= 2023-10-17T18:43:05.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU=', 'Content-Length': '0'} 2023-10-17T18:43:05.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a430f72c84a5ac5-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.593 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.593 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.593 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.594 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.594 DEBUG:boto:Method: GET 2023-10-17T18:43:05.595 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.595 DEBUG:boto:Data: 2023-10-17T18:43:05.595 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.596 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.596 DEBUG:boto:Params: {} 2023-10-17T18:43:05.597 DEBUG:boto:Token: None 2023-10-17T18:43:05.597 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU= 2023-10-17T18:43:05.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU=', 'Content-Length': '0'} 2023-10-17T18:43:05.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008facd1d47756c3d7-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.600 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.601 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.601 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.601 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.602 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.602 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.603 DEBUG:boto:Data: 2023-10-17T18:43:05.603 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.603 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.604 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.604 DEBUG:boto:Params: {} 2023-10-17T18:43:05.604 DEBUG:boto:Token: None 2023-10-17T18:43:05.605 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.605 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BFoD8hQqkDXQuLXf1Ei5uVtqdzs= 2023-10-17T18:43:05.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BFoD8hQqkDXQuLXf1Ei5uVtqdzs=', 'Content-Length': '0'} 2023-10-17T18:43:05.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b1a1677ef46e9f-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.608 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.608 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.608 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.609 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.609 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.610 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.610 DEBUG:boto:Data: 2023-10-17T18:43:05.610 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.611 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.611 DEBUG:boto:Params: {} 2023-10-17T18:43:05.612 DEBUG:boto:Token: None 2023-10-17T18:43:05.612 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BFoD8hQqkDXQuLXf1Ei5uVtqdzs= 2023-10-17T18:43:05.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BFoD8hQqkDXQuLXf1Ei5uVtqdzs=', 'Content-Length': '0'} 2023-10-17T18:43:05.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039dd5fd58208b2dd-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:43:05.616 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.616 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.617 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.617 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.618 DEBUG:boto:Method: GET 2023-10-17T18:43:05.618 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.618 DEBUG:boto:Data: 2023-10-17T18:43:05.619 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.620 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.620 DEBUG:boto:Params: {} 2023-10-17T18:43:05.620 DEBUG:boto:Token: None 2023-10-17T18:43:05.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU= 2023-10-17T18:43:05.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU=', 'Content-Length': '0'} 2023-10-17T18:43:05.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004206d8c416e83efa-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.624 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.624 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.625 DEBUG:boto:path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.625 DEBUG:boto:auth_path=/hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.625 DEBUG:boto:Method: GET 2023-10-17T18:43:05.626 DEBUG:boto:Path: /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.626 DEBUG:boto:Data: 2023-10-17T18:43:05.627 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.627 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.627 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.628 DEBUG:boto:Params: {} 2023-10-17T18:43:05.628 DEBUG:boto:Token: None 2023-10-17T18:43:05.629 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2?versionId=null 2023-10-17T18:43:05.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU= 2023-10-17T18:43:05.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eg6X0NPvWOZ81rYH3vJmP6pQ/xU=', 'Content-Length': '0'} 2023-10-17T18:43:05.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f0fb33c834433af-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.674 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.675 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.675 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.675 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.676 DEBUG:boto:Method: GET 2023-10-17T18:43:05.676 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.677 DEBUG:boto:Data: 2023-10-17T18:43:05.677 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.678 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.678 DEBUG:boto:Params: {} 2023-10-17T18:43:05.679 DEBUG:boto:Token: None 2023-10-17T18:43:05.679 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.679 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c249d315705e1ad8-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.683 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.683 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.684 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.684 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.684 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.685 DEBUG:boto:Method: GET 2023-10-17T18:43:05.685 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.686 DEBUG:boto:Data: 2023-10-17T18:43:05.686 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.687 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.687 DEBUG:boto:Params: {} 2023-10-17T18:43:05.688 DEBUG:boto:Token: None 2023-10-17T18:43:05.688 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.688 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a2ba7ae4773379-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.692 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.693 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.693 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.694 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.694 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:43:05.695 DEBUG:boto:Data: 2023-10-17T18:43:05.695 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.695 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.696 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.696 DEBUG:boto:Params: {} 2023-10-17T18:43:05.697 DEBUG:boto:Token: None 2023-10-17T18:43:05.697 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2 2023-10-17T18:43:05.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vdw/Wt0mqC7B4Z1ZJDoABfkb7t0= 2023-10-17T18:43:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vdw/Wt0mqC7B4Z1ZJDoABfkb7t0=', 'Content-Length': '0'} 2023-10-17T18:43:05.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000815e54802a843d3d-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.700 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.700 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.701 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.701 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:43:05.701 DEBUG:boto:Data: 2023-10-17T18:43:05.702 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.703 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.703 DEBUG:boto:Params: {} 2023-10-17T18:43:05.703 DEBUG:boto:Token: None 2023-10-17T18:43:05.704 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2 2023-10-17T18:43:05.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vdw/Wt0mqC7B4Z1ZJDoABfkb7t0= 2023-10-17T18:43:05.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vdw/Wt0mqC7B4Z1ZJDoABfkb7t0=', 'Content-Length': '0'} 2023-10-17T18:43:05.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005715c8fad225e97a-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-17T18:43:05.707 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.708 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.708 DEBUG:boto:Method: GET 2023-10-17T18:43:05.708 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:43:05.709 DEBUG:boto:Data: 2023-10-17T18:43:05.709 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.710 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.710 DEBUG:boto:Params: {} 2023-10-17T18:43:05.711 DEBUG:boto:Token: None 2023-10-17T18:43:05.711 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2 2023-10-17T18:43:05.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/EzFAwThEXadMv4bVesUNe0Z59U= 2023-10-17T18:43:05.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/EzFAwThEXadMv4bVesUNe0Z59U=', 'Content-Length': '0'} 2023-10-17T18:43:05.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006253ccabcdf6992c-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.754 DEBUG:boto:path=/hbmuog-49/obj2 2023-10-17T18:43:05.755 DEBUG:boto:auth_path=/hbmuog-49/obj2 2023-10-17T18:43:05.755 DEBUG:boto:Method: GET 2023-10-17T18:43:05.755 DEBUG:boto:Path: /hbmuog-49/obj2 2023-10-17T18:43:05.756 DEBUG:boto:Data: 2023-10-17T18:43:05.756 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.757 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.757 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.757 DEBUG:boto:Params: {} 2023-10-17T18:43:05.758 DEBUG:boto:Token: None 2023-10-17T18:43:05.758 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj2 2023-10-17T18:43:05.759 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/EzFAwThEXadMv4bVesUNe0Z59U= 2023-10-17T18:43:05.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/EzFAwThEXadMv4bVesUNe0Z59U=', 'Content-Length': '0'} 2023-10-17T18:43:05.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008055cf2281ced6a7-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.762 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.762 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.762 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.763 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.763 DEBUG:boto:Method: GET 2023-10-17T18:43:05.763 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.764 DEBUG:boto:Data: 2023-10-17T18:43:05.764 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.765 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.765 DEBUG:boto:Params: {} 2023-10-17T18:43:05.766 DEBUG:boto:Token: None 2023-10-17T18:43:05.766 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c526bd99d03bb595-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.770 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.771 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.771 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.771 DEBUG:boto:path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.772 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.772 DEBUG:boto:Method: GET 2023-10-17T18:43:05.773 DEBUG:boto:Path: /hbmuog-49/?prefix=obj2 2023-10-17T18:43:05.773 DEBUG:boto:Data: 2023-10-17T18:43:05.773 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.774 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.775 DEBUG:boto:Params: {} 2023-10-17T18:43:05.775 DEBUG:boto:Token: None 2023-10-17T18:43:05.775 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010e82aafb0f47838-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.779 DEBUG:boto:b'hbmuog-49obj21000falseobj22023-10-17T18:37:44.570Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:43:05.780 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.781 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.781 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.781 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.782 DEBUG:boto:Method: GET 2023-10-17T18:43:05.782 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.782 DEBUG:boto:Data: 2023-10-17T18:43:05.783 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.784 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.784 DEBUG:boto:Params: {} 2023-10-17T18:43:05.784 DEBUG:boto:Token: None 2023-10-17T18:43:05.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE= 2023-10-17T18:43:05.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE=', 'Content-Length': '0'} 2023-10-17T18:43:05.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f780d9fb7ace8535-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.788 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.788 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.789 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.789 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.790 DEBUG:boto:Method: GET 2023-10-17T18:43:05.790 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.790 DEBUG:boto:Data: 2023-10-17T18:43:05.791 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.791 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.792 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.792 DEBUG:boto:Params: {} 2023-10-17T18:43:05.792 DEBUG:boto:Token: None 2023-10-17T18:43:05.793 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.793 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE= 2023-10-17T18:43:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE=', 'Content-Length': '0'} 2023-10-17T18:43:05.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000002d3ff3b4ce58b7-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.838 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.839 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.839 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.839 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.840 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.840 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.840 DEBUG:boto:Data: 2023-10-17T18:43:05.841 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.842 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.842 DEBUG:boto:Params: {} 2023-10-17T18:43:05.842 DEBUG:boto:Token: None 2023-10-17T18:43:05.843 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Fzjkd1f6deLuGoG6GVDWn4IJd6o= 2023-10-17T18:43:05.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Fzjkd1f6deLuGoG6GVDWn4IJd6o=', 'Content-Length': '0'} 2023-10-17T18:43:05.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6b1ef68bcf56c79-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.846 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.846 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.847 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.847 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.848 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.848 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.848 DEBUG:boto:Data: 2023-10-17T18:43:05.849 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.849 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.850 DEBUG:boto:Params: {} 2023-10-17T18:43:05.850 DEBUG:boto:Token: None 2023-10-17T18:43:05.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.851 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Fzjkd1f6deLuGoG6GVDWn4IJd6o= 2023-10-17T18:43:05.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Fzjkd1f6deLuGoG6GVDWn4IJd6o=', 'Content-Length': '0'} 2023-10-17T18:43:05.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc397b9872e0258d-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:43:05.854 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.854 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.855 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.855 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.856 DEBUG:boto:Method: GET 2023-10-17T18:43:05.856 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.856 DEBUG:boto:Data: 2023-10-17T18:43:05.857 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.857 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.858 DEBUG:boto:Params: {} 2023-10-17T18:43:05.858 DEBUG:boto:Token: None 2023-10-17T18:43:05.859 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE= 2023-10-17T18:43:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE=', 'Content-Length': '0'} 2023-10-17T18:43:05.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013d7150b4bf14dd3-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.862 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.863 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.863 DEBUG:boto:path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.863 DEBUG:boto:auth_path=/hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.864 DEBUG:boto:Method: GET 2023-10-17T18:43:05.864 DEBUG:boto:Path: /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.864 DEBUG:boto:Data: 2023-10-17T18:43:05.865 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.866 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.866 DEBUG:boto:Params: {} 2023-10-17T18:43:05.866 DEBUG:boto:Token: None 2023-10-17T18:43:05.867 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3?versionId=null 2023-10-17T18:43:05.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE= 2023-10-17T18:43:05.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JS82+8X6QeyZaje3IKiDXXRNcUE=', 'Content-Length': '0'} 2023-10-17T18:43:05.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007265d46f2d25f2ab-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.870 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.871 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.871 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.871 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.872 DEBUG:boto:Method: GET 2023-10-17T18:43:05.872 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.872 DEBUG:boto:Data: 2023-10-17T18:43:05.873 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.873 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.874 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.874 DEBUG:boto:Params: {} 2023-10-17T18:43:05.874 DEBUG:boto:Token: None 2023-10-17T18:43:05.875 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f7adfc0ba18f42-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.878 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.879 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.879 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.880 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.880 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.881 DEBUG:boto:Method: GET 2023-10-17T18:43:05.881 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.881 DEBUG:boto:Data: 2023-10-17T18:43:05.882 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.882 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.883 DEBUG:boto:Params: {} 2023-10-17T18:43:05.883 DEBUG:boto:Token: None 2023-10-17T18:43:05.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca211222d9da86f0-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.888 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.888 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.888 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.889 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.889 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:43:05.890 DEBUG:boto:Data: 2023-10-17T18:43:05.890 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.891 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.891 DEBUG:boto:Params: {} 2023-10-17T18:43:05.891 DEBUG:boto:Token: None 2023-10-17T18:43:05.892 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3 2023-10-17T18:43:05.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GX/1ctGRobflwtR4dm9dyV9dQfk= 2023-10-17T18:43:05.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GX/1ctGRobflwtR4dm9dyV9dQfk=', 'Content-Length': '0'} 2023-10-17T18:43:05.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095061db7b4fc2397-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.895 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.895 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.896 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.896 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:43:05.896 DEBUG:boto:Data: 2023-10-17T18:43:05.897 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.897 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.898 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.898 DEBUG:boto:Params: {} 2023-10-17T18:43:05.898 DEBUG:boto:Token: None 2023-10-17T18:43:05.899 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3 2023-10-17T18:43:05.899 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GX/1ctGRobflwtR4dm9dyV9dQfk= 2023-10-17T18:43:05.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GX/1ctGRobflwtR4dm9dyV9dQfk=', 'Content-Length': '0'} 2023-10-17T18:43:05.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c09c13838b3393f-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-17T18:43:05.903 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.903 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.903 DEBUG:boto:Method: GET 2023-10-17T18:43:05.904 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:43:05.904 DEBUG:boto:Data: 2023-10-17T18:43:05.905 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.905 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.905 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.906 DEBUG:boto:Params: {} 2023-10-17T18:43:05.906 DEBUG:boto:Token: None 2023-10-17T18:43:05.907 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3 2023-10-17T18:43:05.907 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:93j5Npu95gBUhUttw2rLAT4vWM0= 2023-10-17T18:43:05.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:93j5Npu95gBUhUttw2rLAT4vWM0=', 'Content-Length': '0'} 2023-10-17T18:43:05.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007829fd5a38ca4f29-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.910 DEBUG:boto:path=/hbmuog-49/obj3 2023-10-17T18:43:05.911 DEBUG:boto:auth_path=/hbmuog-49/obj3 2023-10-17T18:43:05.911 DEBUG:boto:Method: GET 2023-10-17T18:43:05.911 DEBUG:boto:Path: /hbmuog-49/obj3 2023-10-17T18:43:05.912 DEBUG:boto:Data: 2023-10-17T18:43:05.912 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.912 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.913 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.913 DEBUG:boto:Params: {} 2023-10-17T18:43:05.914 DEBUG:boto:Token: None 2023-10-17T18:43:05.914 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj3 2023-10-17T18:43:05.914 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:93j5Npu95gBUhUttw2rLAT4vWM0= 2023-10-17T18:43:05.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:93j5Npu95gBUhUttw2rLAT4vWM0=', 'Content-Length': '0'} 2023-10-17T18:43:05.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcb44aa48e537592-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.917 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.918 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.918 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.918 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.919 DEBUG:boto:Method: GET 2023-10-17T18:43:05.919 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.920 DEBUG:boto:Data: 2023-10-17T18:43:05.920 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.920 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.921 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.921 DEBUG:boto:Params: {} 2023-10-17T18:43:05.921 DEBUG:boto:Token: None 2023-10-17T18:43:05.922 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.922 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c1108321448406-00652ed5b9-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.926 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.926 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:05.927 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:05.927 DEBUG:boto:path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.927 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.928 DEBUG:boto:Method: GET 2023-10-17T18:43:05.928 DEBUG:boto:Path: /hbmuog-49/?prefix=obj3 2023-10-17T18:43:05.929 DEBUG:boto:Data: 2023-10-17T18:43:05.929 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.930 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.930 DEBUG:boto:Params: {} 2023-10-17T18:43:05.931 DEBUG:boto:Token: None 2023-10-17T18:43:05.931 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/ 2023-10-17T18:43:05.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8= 2023-10-17T18:43:05.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gh4HO670OHPP9CM1xyIOo+0/lC8=', 'Content-Length': '0'} 2023-10-17T18:43:05.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a451edb97043d2-00652ed5b9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.935 DEBUG:boto:b'hbmuog-49obj31000falseobj32023-10-17T18:37:44.628Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:05.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:43:05.936 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:05.936 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:05.937 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.937 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.937 DEBUG:boto:Method: GET 2023-10-17T18:43:05.938 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.938 DEBUG:boto:Data: 2023-10-17T18:43:05.938 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.939 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.940 DEBUG:boto:Params: {} 2023-10-17T18:43:05.940 DEBUG:boto:Token: None 2023-10-17T18:43:05.941 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QKsFOnCvYIf8EPEmB5VWhm9i7CY= 2023-10-17T18:43:05.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QKsFOnCvYIf8EPEmB5VWhm9i7CY=', 'Content-Length': '0'} 2023-10-17T18:43:05.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000954f9d8ae6a4d4d-00652ed5b9-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.944 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:05.944 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:05.945 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.945 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.945 DEBUG:boto:Method: GET 2023-10-17T18:43:05.946 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.946 DEBUG:boto:Data: 2023-10-17T18:43:05.947 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.947 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:05.947 DEBUG:boto:Port: 8000 2023-10-17T18:43:05.948 DEBUG:boto:Params: {} 2023-10-17T18:43:05.948 DEBUG:boto:Token: None 2023-10-17T18:43:05.949 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QKsFOnCvYIf8EPEmB5VWhm9i7CY= 2023-10-17T18:43:05.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QKsFOnCvYIf8EPEmB5VWhm9i7CY=', 'Content-Length': '0'} 2023-10-17T18:43:05.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007121253ea9a57bd9-00652ed5b9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:05.994 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:05.995 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:05.995 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.995 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.996 DEBUG:boto:Method: HEAD 2023-10-17T18:43:05.996 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.997 DEBUG:boto:Data: 2023-10-17T18:43:05.997 DEBUG:boto:Headers: {} 2023-10-17T18:43:05.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:05.998 DEBUG:boto:Port: 8001 2023-10-17T18:43:05.998 DEBUG:boto:Params: {} 2023-10-17T18:43:05.998 DEBUG:boto:Token: None 2023-10-17T18:43:05.999 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:05 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:05.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Rvm9n9W7ApFHasn9B+5gD33mYUA= 2023-10-17T18:43:06.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rvm9n9W7ApFHasn9B+5gD33mYUA=', 'Content-Length': '0'} 2023-10-17T18:43:06.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f641c27a558ab5a4-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.002 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.002 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.003 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.003 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.003 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.004 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.004 DEBUG:boto:Data: 2023-10-17T18:43:06.005 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.005 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.006 DEBUG:boto:Params: {} 2023-10-17T18:43:06.006 DEBUG:boto:Token: None 2023-10-17T18:43:06.006 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.007 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7mSXdyJEQQOuJK5TsDMAIFX4184= 2023-10-17T18:43:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7mSXdyJEQQOuJK5TsDMAIFX4184=', 'Content-Length': '0'} 2023-10-17T18:43:06.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997ac1beaea3272f-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:43:06.010 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.011 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.011 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.011 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.012 DEBUG:boto:Method: GET 2023-10-17T18:43:06.012 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.012 DEBUG:boto:Data: 2023-10-17T18:43:06.013 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.013 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.014 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.014 DEBUG:boto:Params: {} 2023-10-17T18:43:06.014 DEBUG:boto:Token: None 2023-10-17T18:43:06.015 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.015 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iXCmY7OMV7RHOrQ4c2JujxxQq7M= 2023-10-17T18:43:06.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iXCmY7OMV7RHOrQ4c2JujxxQq7M=', 'Content-Length': '0'} 2023-10-17T18:43:06.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f86c29d728b2b9a-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.018 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.019 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.019 DEBUG:boto:path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.019 DEBUG:boto:auth_path=/hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.020 DEBUG:boto:Method: GET 2023-10-17T18:43:06.020 DEBUG:boto:Path: /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.021 DEBUG:boto:Data: 2023-10-17T18:43:06.021 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.021 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.022 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.022 DEBUG:boto:Params: {} 2023-10-17T18:43:06.023 DEBUG:boto:Token: None 2023-10-17T18:43:06.023 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4?versionId=null 2023-10-17T18:43:06.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iXCmY7OMV7RHOrQ4c2JujxxQq7M= 2023-10-17T18:43:06.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iXCmY7OMV7RHOrQ4c2JujxxQq7M=', 'Content-Length': '0'} 2023-10-17T18:43:06.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003167b7dd5b123480-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.026 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.027 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.027 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.027 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.028 DEBUG:boto:Method: GET 2023-10-17T18:43:06.028 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.028 DEBUG:boto:Data: 2023-10-17T18:43:06.029 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.029 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.030 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.030 DEBUG:boto:Params: {} 2023-10-17T18:43:06.030 DEBUG:boto:Token: None 2023-10-17T18:43:06.031 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.031 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008746f08e679ccafe-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.035 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.035 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.036 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.036 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.036 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.037 DEBUG:boto:Method: GET 2023-10-17T18:43:06.037 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.038 DEBUG:boto:Data: 2023-10-17T18:43:06.038 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.039 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.039 DEBUG:boto:Params: {} 2023-10-17T18:43:06.040 DEBUG:boto:Token: None 2023-10-17T18:43:06.040 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076c1a268ea166f68-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.044 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.045 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.045 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.046 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.046 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:43:06.046 DEBUG:boto:Data: 2023-10-17T18:43:06.047 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.047 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.048 DEBUG:boto:Params: {} 2023-10-17T18:43:06.048 DEBUG:boto:Token: None 2023-10-17T18:43:06.049 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4 2023-10-17T18:43:06.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+tlW1WXY2ihkO4zjtZP+nszNhuI= 2023-10-17T18:43:06.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+tlW1WXY2ihkO4zjtZP+nszNhuI=', 'Content-Length': '0'} 2023-10-17T18:43:06.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e6e3e2095bdee8-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.052 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.052 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.052 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.053 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:43:06.053 DEBUG:boto:Data: 2023-10-17T18:43:06.054 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.054 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.055 DEBUG:boto:Params: {} 2023-10-17T18:43:06.055 DEBUG:boto:Token: None 2023-10-17T18:43:06.055 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4 2023-10-17T18:43:06.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+tlW1WXY2ihkO4zjtZP+nszNhuI= 2023-10-17T18:43:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+tlW1WXY2ihkO4zjtZP+nszNhuI=', 'Content-Length': '0'} 2023-10-17T18:43:06.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2520cb3883a4a7c-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-17T18:43:06.059 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.059 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.060 DEBUG:boto:Method: GET 2023-10-17T18:43:06.060 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:43:06.060 DEBUG:boto:Data: 2023-10-17T18:43:06.061 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.061 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.062 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.062 DEBUG:boto:Params: {} 2023-10-17T18:43:06.062 DEBUG:boto:Token: None 2023-10-17T18:43:06.063 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4 2023-10-17T18:43:06.063 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XUq22lPXQaBzwnWE3gj699V6ur0= 2023-10-17T18:43:06.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUq22lPXQaBzwnWE3gj699V6ur0=', 'Content-Length': '0'} 2023-10-17T18:43:06.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bbf59008fdcb268-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.106 DEBUG:boto:path=/hbmuog-49/obj4 2023-10-17T18:43:06.107 DEBUG:boto:auth_path=/hbmuog-49/obj4 2023-10-17T18:43:06.107 DEBUG:boto:Method: GET 2023-10-17T18:43:06.107 DEBUG:boto:Path: /hbmuog-49/obj4 2023-10-17T18:43:06.108 DEBUG:boto:Data: 2023-10-17T18:43:06.108 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.109 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.109 DEBUG:boto:Params: {} 2023-10-17T18:43:06.110 DEBUG:boto:Token: None 2023-10-17T18:43:06.110 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj4 2023-10-17T18:43:06.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XUq22lPXQaBzwnWE3gj699V6ur0= 2023-10-17T18:43:06.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUq22lPXQaBzwnWE3gj699V6ur0=', 'Content-Length': '0'} 2023-10-17T18:43:06.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:37:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000152513079574ae01-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.114 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.114 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.114 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.115 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.115 DEBUG:boto:Method: GET 2023-10-17T18:43:06.116 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.116 DEBUG:boto:Data: 2023-10-17T18:43:06.116 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.117 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.117 DEBUG:boto:Params: {} 2023-10-17T18:43:06.118 DEBUG:boto:Token: None 2023-10-17T18:43:06.118 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.119 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026bade8351e205e3-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.122 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.123 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.123 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.124 DEBUG:boto:path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.124 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.125 DEBUG:boto:Method: GET 2023-10-17T18:43:06.125 DEBUG:boto:Path: /hbmuog-49/?prefix=obj4 2023-10-17T18:43:06.125 DEBUG:boto:Data: 2023-10-17T18:43:06.126 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.126 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.127 DEBUG:boto:Params: {} 2023-10-17T18:43:06.127 DEBUG:boto:Token: None 2023-10-17T18:43:06.128 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4b6a96ab5828714-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.131 DEBUG:boto:b'hbmuog-49obj41000falseobj42023-10-17T18:37:44.647Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:43:06.132 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.132 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.133 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.133 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.134 DEBUG:boto:Method: GET 2023-10-17T18:43:06.134 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.134 DEBUG:boto:Data: 2023-10-17T18:43:06.135 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.136 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.136 DEBUG:boto:Params: {} 2023-10-17T18:43:06.136 DEBUG:boto:Token: None 2023-10-17T18:43:06.137 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE= 2023-10-17T18:43:06.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE=', 'Content-Length': '0'} 2023-10-17T18:43:06.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001293ba4e770a5ae5-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.142 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.142 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.143 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.143 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.143 DEBUG:boto:Method: GET 2023-10-17T18:43:06.144 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.144 DEBUG:boto:Data: 2023-10-17T18:43:06.144 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.145 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.145 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.146 DEBUG:boto:Params: {} 2023-10-17T18:43:06.146 DEBUG:boto:Token: None 2023-10-17T18:43:06.147 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.147 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE= 2023-10-17T18:43:06.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE=', 'Content-Length': '0'} 2023-10-17T18:43:06.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000139f87a84e8c5d1-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.190 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.191 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.191 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.191 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.192 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.192 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.192 DEBUG:boto:Data: 2023-10-17T18:43:06.193 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.193 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.194 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.194 DEBUG:boto:Params: {} 2023-10-17T18:43:06.195 DEBUG:boto:Token: None 2023-10-17T18:43:06.195 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.195 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mJzbDykRkgOlPErYyC5h8l3PO9g= 2023-10-17T18:43:06.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mJzbDykRkgOlPErYyC5h8l3PO9g=', 'Content-Length': '0'} 2023-10-17T18:43:06.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e2dba7abfa54992-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.198 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.198 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.199 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.199 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.199 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.200 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.200 DEBUG:boto:Data: 2023-10-17T18:43:06.201 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.201 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.201 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.202 DEBUG:boto:Params: {} 2023-10-17T18:43:06.202 DEBUG:boto:Token: None 2023-10-17T18:43:06.203 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.203 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mJzbDykRkgOlPErYyC5h8l3PO9g= 2023-10-17T18:43:06.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mJzbDykRkgOlPErYyC5h8l3PO9g=', 'Content-Length': '0'} 2023-10-17T18:43:06.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c72465dd30a8e5a3-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:43:06.206 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.206 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.207 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.207 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.208 DEBUG:boto:Method: GET 2023-10-17T18:43:06.208 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.208 DEBUG:boto:Data: 2023-10-17T18:43:06.209 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.210 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.210 DEBUG:boto:Params: {} 2023-10-17T18:43:06.210 DEBUG:boto:Token: None 2023-10-17T18:43:06.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE= 2023-10-17T18:43:06.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE=', 'Content-Length': '0'} 2023-10-17T18:43:06.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9cb86efdae5a3bc-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.214 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.215 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.215 DEBUG:boto:path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.216 DEBUG:boto:auth_path=/hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.216 DEBUG:boto:Method: GET 2023-10-17T18:43:06.216 DEBUG:boto:Path: /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.217 DEBUG:boto:Data: 2023-10-17T18:43:06.217 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.218 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.218 DEBUG:boto:Params: {} 2023-10-17T18:43:06.219 DEBUG:boto:Token: None 2023-10-17T18:43:06.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5?versionId=null 2023-10-17T18:43:06.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE= 2023-10-17T18:43:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ap9ZmD48UjO6ie/0+fd6VQIU2PE=', 'Content-Length': '0'} 2023-10-17T18:43:06.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca0a8cb34d0fa478-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.223 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.223 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.223 DEBUG:boto:path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.224 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.224 DEBUG:boto:Method: GET 2023-10-17T18:43:06.225 DEBUG:boto:Path: /hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.225 DEBUG:boto:Data: 2023-10-17T18:43:06.225 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.226 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.226 DEBUG:boto:Params: {} 2023-10-17T18:43:06.227 DEBUG:boto:Token: None 2023-10-17T18:43:06.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007349cfd798b3194a-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.231 DEBUG:boto:b'hbmuog-49obj51000falseobj52023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.232 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.232 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.232 DEBUG:boto:path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.233 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.233 DEBUG:boto:Method: GET 2023-10-17T18:43:06.234 DEBUG:boto:Path: /hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.234 DEBUG:boto:Data: 2023-10-17T18:43:06.234 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.235 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.235 DEBUG:boto:Params: {} 2023-10-17T18:43:06.236 DEBUG:boto:Token: None 2023-10-17T18:43:06.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ac67d6c4fa7c23-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.240 DEBUG:boto:b'hbmuog-49obj51000falseobj52023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.240 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.241 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.241 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.242 DEBUG:boto:Path: /hbmuog-49/obj5 2023-10-17T18:43:06.242 DEBUG:boto:Data: 2023-10-17T18:43:06.242 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.243 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.243 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.244 DEBUG:boto:Params: {} 2023-10-17T18:43:06.244 DEBUG:boto:Token: None 2023-10-17T18:43:06.244 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5 2023-10-17T18:43:06.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwTfa18d9ksAfJycFuDSctBqybA= 2023-10-17T18:43:06.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwTfa18d9ksAfJycFuDSctBqybA=', 'Content-Length': '0'} 2023-10-17T18:43:06.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002fa39f367fc5df1-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.248 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.248 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.249 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.249 DEBUG:boto:Path: /hbmuog-49/obj5 2023-10-17T18:43:06.249 DEBUG:boto:Data: 2023-10-17T18:43:06.250 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.250 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.250 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.251 DEBUG:boto:Params: {} 2023-10-17T18:43:06.251 DEBUG:boto:Token: None 2023-10-17T18:43:06.252 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5 2023-10-17T18:43:06.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PwTfa18d9ksAfJycFuDSctBqybA= 2023-10-17T18:43:06.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PwTfa18d9ksAfJycFuDSctBqybA=', 'Content-Length': '0'} 2023-10-17T18:43:06.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b30c642318d59ff6-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-17T18:43:06.255 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.256 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.256 DEBUG:boto:Method: GET 2023-10-17T18:43:06.256 DEBUG:boto:Path: /hbmuog-49/obj5 2023-10-17T18:43:06.257 DEBUG:boto:Data: 2023-10-17T18:43:06.257 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.258 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.258 DEBUG:boto:Params: {} 2023-10-17T18:43:06.259 DEBUG:boto:Token: None 2023-10-17T18:43:06.259 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5 2023-10-17T18:43:06.260 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vNaup8Hgxtm4FDnjN4mHG6HYaq0= 2023-10-17T18:43:06.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vNaup8Hgxtm4FDnjN4mHG6HYaq0=', 'Content-Length': '0'} 2023-10-17T18:43:06.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a27a4d100ccc2a79-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.302 DEBUG:boto:path=/hbmuog-49/obj5 2023-10-17T18:43:06.303 DEBUG:boto:auth_path=/hbmuog-49/obj5 2023-10-17T18:43:06.303 DEBUG:boto:Method: GET 2023-10-17T18:43:06.303 DEBUG:boto:Path: /hbmuog-49/obj5 2023-10-17T18:43:06.304 DEBUG:boto:Data: 2023-10-17T18:43:06.304 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.305 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.305 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.305 DEBUG:boto:Params: {} 2023-10-17T18:43:06.306 DEBUG:boto:Token: None 2023-10-17T18:43:06.306 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj5 2023-10-17T18:43:06.307 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vNaup8Hgxtm4FDnjN4mHG6HYaq0= 2023-10-17T18:43:06.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vNaup8Hgxtm4FDnjN4mHG6HYaq0=', 'Content-Length': '0'} 2023-10-17T18:43:06.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef1bf8c0ee647230-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.310 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.310 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.310 DEBUG:boto:path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.311 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.311 DEBUG:boto:Method: GET 2023-10-17T18:43:06.312 DEBUG:boto:Path: /hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.312 DEBUG:boto:Data: 2023-10-17T18:43:06.312 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.313 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.313 DEBUG:boto:Params: {} 2023-10-17T18:43:06.314 DEBUG:boto:Token: None 2023-10-17T18:43:06.314 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e26307d41e1483-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.319 DEBUG:boto:b'hbmuog-49obj51000falseobj52023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.319 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.320 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.320 DEBUG:boto:path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.320 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.321 DEBUG:boto:Method: GET 2023-10-17T18:43:06.321 DEBUG:boto:Path: /hbmuog-49/?prefix=obj5 2023-10-17T18:43:06.321 DEBUG:boto:Data: 2023-10-17T18:43:06.322 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.323 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.323 DEBUG:boto:Params: {} 2023-10-17T18:43:06.323 DEBUG:boto:Token: None 2023-10-17T18:43:06.324 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004713f14ac03e2e89-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.327 DEBUG:boto:b'hbmuog-49obj51000falseobj52023-10-17T18:40:43.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:43:06.328 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.329 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.329 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.330 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.330 DEBUG:boto:Method: GET 2023-10-17T18:43:06.330 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.331 DEBUG:boto:Data: 2023-10-17T18:43:06.331 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.332 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.332 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.332 DEBUG:boto:Params: {} 2023-10-17T18:43:06.333 DEBUG:boto:Token: None 2023-10-17T18:43:06.333 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs= 2023-10-17T18:43:06.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs=', 'Content-Length': '0'} 2023-10-17T18:43:06.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000383c5cb1a4e29e6f-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.336 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.337 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.337 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.337 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.338 DEBUG:boto:Method: GET 2023-10-17T18:43:06.338 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.339 DEBUG:boto:Data: 2023-10-17T18:43:06.339 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.340 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.340 DEBUG:boto:Params: {} 2023-10-17T18:43:06.341 DEBUG:boto:Token: None 2023-10-17T18:43:06.341 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs= 2023-10-17T18:43:06.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs=', 'Content-Length': '0'} 2023-10-17T18:43:06.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7dc60d4cb43744b-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.390 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.391 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.391 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.391 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.392 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.392 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.393 DEBUG:boto:Data: 2023-10-17T18:43:06.393 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.393 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.394 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.394 DEBUG:boto:Params: {} 2023-10-17T18:43:06.395 DEBUG:boto:Token: None 2023-10-17T18:43:06.395 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.395 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9dl17BaXE3oBUhev4JRGEsCbVXA= 2023-10-17T18:43:06.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9dl17BaXE3oBUhev4JRGEsCbVXA=', 'Content-Length': '0'} 2023-10-17T18:43:06.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec9cb75b17245624-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.398 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.399 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.399 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.399 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.400 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.400 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.400 DEBUG:boto:Data: 2023-10-17T18:43:06.401 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.401 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.402 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.402 DEBUG:boto:Params: {} 2023-10-17T18:43:06.402 DEBUG:boto:Token: None 2023-10-17T18:43:06.403 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.403 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9dl17BaXE3oBUhev4JRGEsCbVXA= 2023-10-17T18:43:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9dl17BaXE3oBUhev4JRGEsCbVXA=', 'Content-Length': '0'} 2023-10-17T18:43:06.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000520b38d775560f8b-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:43:06.406 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.407 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.407 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.407 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.408 DEBUG:boto:Method: GET 2023-10-17T18:43:06.408 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.409 DEBUG:boto:Data: 2023-10-17T18:43:06.409 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.410 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.410 DEBUG:boto:Params: {} 2023-10-17T18:43:06.411 DEBUG:boto:Token: None 2023-10-17T18:43:06.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs= 2023-10-17T18:43:06.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs=', 'Content-Length': '0'} 2023-10-17T18:43:06.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd42c0035d901ad5-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.415 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.415 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.415 DEBUG:boto:path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.416 DEBUG:boto:auth_path=/hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.416 DEBUG:boto:Method: GET 2023-10-17T18:43:06.417 DEBUG:boto:Path: /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.417 DEBUG:boto:Data: 2023-10-17T18:43:06.417 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.418 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.418 DEBUG:boto:Params: {} 2023-10-17T18:43:06.419 DEBUG:boto:Token: None 2023-10-17T18:43:06.419 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6?versionId=null 2023-10-17T18:43:06.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs= 2023-10-17T18:43:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1hon0QpQLO3LmK49yRbtDYtxHvs=', 'Content-Length': '0'} 2023-10-17T18:43:06.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20476945dfb5a47-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.423 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.423 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.424 DEBUG:boto:path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.424 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.424 DEBUG:boto:Method: GET 2023-10-17T18:43:06.425 DEBUG:boto:Path: /hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.425 DEBUG:boto:Data: 2023-10-17T18:43:06.426 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.426 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.427 DEBUG:boto:Params: {} 2023-10-17T18:43:06.427 DEBUG:boto:Token: None 2023-10-17T18:43:06.428 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.428 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df116a67e47f6f5c-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.432 DEBUG:boto:b'hbmuog-49obj61000falseobj62023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.432 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.433 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.433 DEBUG:boto:path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.434 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.434 DEBUG:boto:Method: GET 2023-10-17T18:43:06.434 DEBUG:boto:Path: /hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.435 DEBUG:boto:Data: 2023-10-17T18:43:06.435 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.436 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.436 DEBUG:boto:Params: {} 2023-10-17T18:43:06.437 DEBUG:boto:Token: None 2023-10-17T18:43:06.437 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.438 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8dbd295cc501a1b-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.441 DEBUG:boto:b'hbmuog-49obj61000falseobj62023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.442 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.442 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.442 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.443 DEBUG:boto:Path: /hbmuog-49/obj6 2023-10-17T18:43:06.443 DEBUG:boto:Data: 2023-10-17T18:43:06.443 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.444 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.445 DEBUG:boto:Params: {} 2023-10-17T18:43:06.445 DEBUG:boto:Token: None 2023-10-17T18:43:06.445 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6 2023-10-17T18:43:06.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ixfkTis9X7pskszYjDkpKTCzsaE= 2023-10-17T18:43:06.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ixfkTis9X7pskszYjDkpKTCzsaE=', 'Content-Length': '0'} 2023-10-17T18:43:06.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080aae59ba4e740f8-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.448 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.449 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.449 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.449 DEBUG:boto:Path: /hbmuog-49/obj6 2023-10-17T18:43:06.450 DEBUG:boto:Data: 2023-10-17T18:43:06.450 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.451 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.451 DEBUG:boto:Params: {} 2023-10-17T18:43:06.452 DEBUG:boto:Token: None 2023-10-17T18:43:06.452 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6 2023-10-17T18:43:06.452 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ixfkTis9X7pskszYjDkpKTCzsaE= 2023-10-17T18:43:06.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ixfkTis9X7pskszYjDkpKTCzsaE=', 'Content-Length': '0'} 2023-10-17T18:43:06.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036ce01cc4278e620-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-17T18:43:06.456 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.456 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.457 DEBUG:boto:Method: GET 2023-10-17T18:43:06.457 DEBUG:boto:Path: /hbmuog-49/obj6 2023-10-17T18:43:06.457 DEBUG:boto:Data: 2023-10-17T18:43:06.458 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.459 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.459 DEBUG:boto:Params: {} 2023-10-17T18:43:06.459 DEBUG:boto:Token: None 2023-10-17T18:43:06.460 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6 2023-10-17T18:43:06.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:owirZr0F/hsbxgOW9ak0p2wwQBg= 2023-10-17T18:43:06.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:owirZr0F/hsbxgOW9ak0p2wwQBg=', 'Content-Length': '0'} 2023-10-17T18:43:06.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c565498e63ee4802-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.463 DEBUG:boto:path=/hbmuog-49/obj6 2023-10-17T18:43:06.464 DEBUG:boto:auth_path=/hbmuog-49/obj6 2023-10-17T18:43:06.464 DEBUG:boto:Method: GET 2023-10-17T18:43:06.464 DEBUG:boto:Path: /hbmuog-49/obj6 2023-10-17T18:43:06.465 DEBUG:boto:Data: 2023-10-17T18:43:06.465 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.466 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.466 DEBUG:boto:Params: {} 2023-10-17T18:43:06.467 DEBUG:boto:Token: None 2023-10-17T18:43:06.467 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj6 2023-10-17T18:43:06.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:owirZr0F/hsbxgOW9ak0p2wwQBg= 2023-10-17T18:43:06.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:owirZr0F/hsbxgOW9ak0p2wwQBg=', 'Content-Length': '0'} 2023-10-17T18:43:06.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059075e24014f0452-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.510 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.511 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.511 DEBUG:boto:path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.512 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.512 DEBUG:boto:Method: GET 2023-10-17T18:43:06.512 DEBUG:boto:Path: /hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.513 DEBUG:boto:Data: 2023-10-17T18:43:06.513 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.514 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.515 DEBUG:boto:Params: {} 2023-10-17T18:43:06.515 DEBUG:boto:Token: None 2023-10-17T18:43:06.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.516 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbc4b97671ebf45-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.520 DEBUG:boto:b'hbmuog-49obj61000falseobj62023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.520 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.521 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.521 DEBUG:boto:path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.521 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.522 DEBUG:boto:Method: GET 2023-10-17T18:43:06.522 DEBUG:boto:Path: /hbmuog-49/?prefix=obj6 2023-10-17T18:43:06.523 DEBUG:boto:Data: 2023-10-17T18:43:06.523 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.524 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.524 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.525 DEBUG:boto:Params: {} 2023-10-17T18:43:06.525 DEBUG:boto:Token: None 2023-10-17T18:43:06.525 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.526 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6abff917182b0d5-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.529 DEBUG:boto:b'hbmuog-49obj61000falseobj62023-10-17T18:40:43.051Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:43:06.530 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.531 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.531 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.532 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.532 DEBUG:boto:Method: GET 2023-10-17T18:43:06.532 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.533 DEBUG:boto:Data: 2023-10-17T18:43:06.533 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.534 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.535 DEBUG:boto:Params: {} 2023-10-17T18:43:06.535 DEBUG:boto:Token: None 2023-10-17T18:43:06.536 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.536 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8= 2023-10-17T18:43:06.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8=', 'Content-Length': '0'} 2023-10-17T18:43:06.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082e64df3b07874b2-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.582 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.583 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.583 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.584 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.584 DEBUG:boto:Method: GET 2023-10-17T18:43:06.584 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.585 DEBUG:boto:Data: 2023-10-17T18:43:06.585 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.586 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.587 DEBUG:boto:Params: {} 2023-10-17T18:43:06.587 DEBUG:boto:Token: None 2023-10-17T18:43:06.588 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8= 2023-10-17T18:43:06.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8=', 'Content-Length': '0'} 2023-10-17T18:43:06.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080b49c4ba0321fe2-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.591 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.592 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.592 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.593 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.593 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.594 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.594 DEBUG:boto:Data: 2023-10-17T18:43:06.595 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.595 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.596 DEBUG:boto:Params: {} 2023-10-17T18:43:06.596 DEBUG:boto:Token: None 2023-10-17T18:43:06.597 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W7BAbW/ul8w9+svNPGclHPySim4= 2023-10-17T18:43:06.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W7BAbW/ul8w9+svNPGclHPySim4=', 'Content-Length': '0'} 2023-10-17T18:43:06.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5c375f6afe09818-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.600 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.600 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.601 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.601 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.602 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.602 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.602 DEBUG:boto:Data: 2023-10-17T18:43:06.603 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.603 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.604 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.604 DEBUG:boto:Params: {} 2023-10-17T18:43:06.604 DEBUG:boto:Token: None 2023-10-17T18:43:06.605 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.605 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W7BAbW/ul8w9+svNPGclHPySim4= 2023-10-17T18:43:06.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W7BAbW/ul8w9+svNPGclHPySim4=', 'Content-Length': '0'} 2023-10-17T18:43:06.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d07c9dbe88c17ae8-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:43:06.609 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.609 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.610 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.610 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.611 DEBUG:boto:Method: GET 2023-10-17T18:43:06.611 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.611 DEBUG:boto:Data: 2023-10-17T18:43:06.612 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.613 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.613 DEBUG:boto:Params: {} 2023-10-17T18:43:06.613 DEBUG:boto:Token: None 2023-10-17T18:43:06.614 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8= 2023-10-17T18:43:06.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8=', 'Content-Length': '0'} 2023-10-17T18:43:06.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096b0eb609798d181-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.617 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.618 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.618 DEBUG:boto:path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.619 DEBUG:boto:auth_path=/hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.619 DEBUG:boto:Method: GET 2023-10-17T18:43:06.620 DEBUG:boto:Path: /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.620 DEBUG:boto:Data: 2023-10-17T18:43:06.620 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.621 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.622 DEBUG:boto:Params: {} 2023-10-17T18:43:06.622 DEBUG:boto:Token: None 2023-10-17T18:43:06.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7?versionId=null 2023-10-17T18:43:06.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8= 2023-10-17T18:43:06.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xVkex3ZZi8cXcOEPMUlwhnrDir8=', 'Content-Length': '0'} 2023-10-17T18:43:06.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019952fd3bc99ab2c-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.626 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.627 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.627 DEBUG:boto:path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.628 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.628 DEBUG:boto:Method: GET 2023-10-17T18:43:06.629 DEBUG:boto:Path: /hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.629 DEBUG:boto:Data: 2023-10-17T18:43:06.629 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.630 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.630 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.631 DEBUG:boto:Params: {} 2023-10-17T18:43:06.631 DEBUG:boto:Token: None 2023-10-17T18:43:06.632 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.632 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224fe4b4cd21202a-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.636 DEBUG:boto:b'hbmuog-49obj71000falseobj72023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.636 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.637 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.637 DEBUG:boto:path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.638 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.638 DEBUG:boto:Method: GET 2023-10-17T18:43:06.639 DEBUG:boto:Path: /hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.639 DEBUG:boto:Data: 2023-10-17T18:43:06.640 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.641 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.641 DEBUG:boto:Params: {} 2023-10-17T18:43:06.642 DEBUG:boto:Token: None 2023-10-17T18:43:06.642 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097619f4f36c0224d-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.647 DEBUG:boto:b'hbmuog-49obj71000falseobj72023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.648 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.648 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.649 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.649 DEBUG:boto:Path: /hbmuog-49/obj7 2023-10-17T18:43:06.650 DEBUG:boto:Data: 2023-10-17T18:43:06.650 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.651 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.651 DEBUG:boto:Params: {} 2023-10-17T18:43:06.652 DEBUG:boto:Token: None 2023-10-17T18:43:06.652 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7 2023-10-17T18:43:06.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iir1SR0AX9AOJ7ofe0B41/VZUVE= 2023-10-17T18:43:06.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iir1SR0AX9AOJ7ofe0B41/VZUVE=', 'Content-Length': '0'} 2023-10-17T18:43:06.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba40b974759f0569-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.656 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.656 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.656 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.657 DEBUG:boto:Path: /hbmuog-49/obj7 2023-10-17T18:43:06.657 DEBUG:boto:Data: 2023-10-17T18:43:06.657 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.658 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.659 DEBUG:boto:Params: {} 2023-10-17T18:43:06.659 DEBUG:boto:Token: None 2023-10-17T18:43:06.660 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7 2023-10-17T18:43:06.660 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iir1SR0AX9AOJ7ofe0B41/VZUVE= 2023-10-17T18:43:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iir1SR0AX9AOJ7ofe0B41/VZUVE=', 'Content-Length': '0'} 2023-10-17T18:43:06.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000696f5229f13af288-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-17T18:43:06.663 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.664 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.664 DEBUG:boto:Method: GET 2023-10-17T18:43:06.664 DEBUG:boto:Path: /hbmuog-49/obj7 2023-10-17T18:43:06.665 DEBUG:boto:Data: 2023-10-17T18:43:06.665 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.666 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.666 DEBUG:boto:Params: {} 2023-10-17T18:43:06.667 DEBUG:boto:Token: None 2023-10-17T18:43:06.667 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7 2023-10-17T18:43:06.668 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yeu7AukVX+o8k5SwBnPgExFriC0= 2023-10-17T18:43:06.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yeu7AukVX+o8k5SwBnPgExFriC0=', 'Content-Length': '0'} 2023-10-17T18:43:06.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039e0f320d770e934-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.710 DEBUG:boto:path=/hbmuog-49/obj7 2023-10-17T18:43:06.711 DEBUG:boto:auth_path=/hbmuog-49/obj7 2023-10-17T18:43:06.711 DEBUG:boto:Method: GET 2023-10-17T18:43:06.711 DEBUG:boto:Path: /hbmuog-49/obj7 2023-10-17T18:43:06.712 DEBUG:boto:Data: 2023-10-17T18:43:06.712 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.713 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.713 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.714 DEBUG:boto:Params: {} 2023-10-17T18:43:06.714 DEBUG:boto:Token: None 2023-10-17T18:43:06.715 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj7 2023-10-17T18:43:06.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yeu7AukVX+o8k5SwBnPgExFriC0= 2023-10-17T18:43:06.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yeu7AukVX+o8k5SwBnPgExFriC0=', 'Content-Length': '0'} 2023-10-17T18:43:06.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bda8eb9fc369cbfb-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.718 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.718 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.719 DEBUG:boto:path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.719 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.720 DEBUG:boto:Method: GET 2023-10-17T18:43:06.720 DEBUG:boto:Path: /hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.720 DEBUG:boto:Data: 2023-10-17T18:43:06.721 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.721 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.722 DEBUG:boto:Params: {} 2023-10-17T18:43:06.722 DEBUG:boto:Token: None 2023-10-17T18:43:06.723 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f0cf0e8593a344-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.727 DEBUG:boto:b'hbmuog-49obj71000falseobj72023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.727 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.728 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.728 DEBUG:boto:path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.729 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.729 DEBUG:boto:Method: GET 2023-10-17T18:43:06.729 DEBUG:boto:Path: /hbmuog-49/?prefix=obj7 2023-10-17T18:43:06.730 DEBUG:boto:Data: 2023-10-17T18:43:06.730 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.731 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.731 DEBUG:boto:Params: {} 2023-10-17T18:43:06.732 DEBUG:boto:Token: None 2023-10-17T18:43:06.732 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001127231bfc4998e-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.736 DEBUG:boto:b'hbmuog-49obj71000falseobj72023-10-17T18:40:43.068Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:43:06.737 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.737 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.738 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.738 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.739 DEBUG:boto:Method: GET 2023-10-17T18:43:06.739 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.740 DEBUG:boto:Data: 2023-10-17T18:43:06.740 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.740 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.741 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.741 DEBUG:boto:Params: {} 2023-10-17T18:43:06.742 DEBUG:boto:Token: None 2023-10-17T18:43:06.742 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.742 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g= 2023-10-17T18:43:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g=', 'Content-Length': '0'} 2023-10-17T18:43:06.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f16fbcf87803931e-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.745 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.745 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.746 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.746 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.747 DEBUG:boto:Method: GET 2023-10-17T18:43:06.747 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.747 DEBUG:boto:Data: 2023-10-17T18:43:06.748 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.748 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.749 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.749 DEBUG:boto:Params: {} 2023-10-17T18:43:06.749 DEBUG:boto:Token: None 2023-10-17T18:43:06.750 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.750 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g= 2023-10-17T18:43:06.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g=', 'Content-Length': '0'} 2023-10-17T18:43:06.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a39e4af4cc1f7d31-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.753 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.754 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.754 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.754 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.755 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.755 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.755 DEBUG:boto:Data: 2023-10-17T18:43:06.756 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.756 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.757 DEBUG:boto:Params: {} 2023-10-17T18:43:06.757 DEBUG:boto:Token: None 2023-10-17T18:43:06.757 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jw7IBImrQMUc9ZPnm36YaFV4aM4= 2023-10-17T18:43:06.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jw7IBImrQMUc9ZPnm36YaFV4aM4=', 'Content-Length': '0'} 2023-10-17T18:43:06.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c93a9f9f2abc7e1-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.761 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.761 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.761 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.762 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.762 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.762 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.763 DEBUG:boto:Data: 2023-10-17T18:43:06.763 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.764 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.764 DEBUG:boto:Params: {} 2023-10-17T18:43:06.765 DEBUG:boto:Token: None 2023-10-17T18:43:06.765 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jw7IBImrQMUc9ZPnm36YaFV4aM4= 2023-10-17T18:43:06.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jw7IBImrQMUc9ZPnm36YaFV4aM4=', 'Content-Length': '0'} 2023-10-17T18:43:06.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dde2cb1b3191646-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:43:06.769 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.769 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.770 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.770 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.770 DEBUG:boto:Method: GET 2023-10-17T18:43:06.771 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.771 DEBUG:boto:Data: 2023-10-17T18:43:06.771 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.772 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.772 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.773 DEBUG:boto:Params: {} 2023-10-17T18:43:06.773 DEBUG:boto:Token: None 2023-10-17T18:43:06.774 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g= 2023-10-17T18:43:06.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g=', 'Content-Length': '0'} 2023-10-17T18:43:06.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f749b2f0b1e2dc74-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.777 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.778 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.778 DEBUG:boto:path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.778 DEBUG:boto:auth_path=/hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.779 DEBUG:boto:Method: GET 2023-10-17T18:43:06.779 DEBUG:boto:Path: /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.780 DEBUG:boto:Data: 2023-10-17T18:43:06.780 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.780 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.781 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.781 DEBUG:boto:Params: {} 2023-10-17T18:43:06.782 DEBUG:boto:Token: None 2023-10-17T18:43:06.782 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8?versionId=null 2023-10-17T18:43:06.782 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g= 2023-10-17T18:43:06.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8Ids4hnThLbyBAcB9xMnglTp67g=', 'Content-Length': '0'} 2023-10-17T18:43:06.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000820e7675fecef10e-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.785 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.786 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.786 DEBUG:boto:path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.786 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.787 DEBUG:boto:Method: GET 2023-10-17T18:43:06.787 DEBUG:boto:Path: /hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.788 DEBUG:boto:Data: 2023-10-17T18:43:06.788 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.789 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.789 DEBUG:boto:Params: {} 2023-10-17T18:43:06.790 DEBUG:boto:Token: None 2023-10-17T18:43:06.790 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091933e7a6d8b1fc6-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.794 DEBUG:boto:b'hbmuog-49obj81000falseobj82023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.795 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.795 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.795 DEBUG:boto:path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.796 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.796 DEBUG:boto:Method: GET 2023-10-17T18:43:06.797 DEBUG:boto:Path: /hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.797 DEBUG:boto:Data: 2023-10-17T18:43:06.797 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.798 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.799 DEBUG:boto:Params: {} 2023-10-17T18:43:06.799 DEBUG:boto:Token: None 2023-10-17T18:43:06.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000221a83ae6d0cc092-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.803 DEBUG:boto:b'hbmuog-49obj81000falseobj82023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.804 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.804 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.805 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.805 DEBUG:boto:Path: /hbmuog-49/obj8 2023-10-17T18:43:06.805 DEBUG:boto:Data: 2023-10-17T18:43:06.806 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.807 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.807 DEBUG:boto:Params: {} 2023-10-17T18:43:06.807 DEBUG:boto:Token: None 2023-10-17T18:43:06.808 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8 2023-10-17T18:43:06.808 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+ZKKHTnW+jVKgxZ4b3WMBxnELLM= 2023-10-17T18:43:06.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+ZKKHTnW+jVKgxZ4b3WMBxnELLM=', 'Content-Length': '0'} 2023-10-17T18:43:06.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000473d59c7fcae9b7b-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.811 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.812 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.812 DEBUG:boto:Method: HEAD 2023-10-17T18:43:06.812 DEBUG:boto:Path: /hbmuog-49/obj8 2023-10-17T18:43:06.813 DEBUG:boto:Data: 2023-10-17T18:43:06.813 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.813 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.814 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.814 DEBUG:boto:Params: {} 2023-10-17T18:43:06.815 DEBUG:boto:Token: None 2023-10-17T18:43:06.815 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8 2023-10-17T18:43:06.815 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+ZKKHTnW+jVKgxZ4b3WMBxnELLM= 2023-10-17T18:43:06.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+ZKKHTnW+jVKgxZ4b3WMBxnELLM=', 'Content-Length': '0'} 2023-10-17T18:43:06.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd8c54cd19f9821d-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-17T18:43:06.818 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.819 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.819 DEBUG:boto:Method: GET 2023-10-17T18:43:06.820 DEBUG:boto:Path: /hbmuog-49/obj8 2023-10-17T18:43:06.820 DEBUG:boto:Data: 2023-10-17T18:43:06.821 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.821 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.822 DEBUG:boto:Params: {} 2023-10-17T18:43:06.822 DEBUG:boto:Token: None 2023-10-17T18:43:06.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8 2023-10-17T18:43:06.823 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ofjrwbLLiAAeS17VzynfsLNJGJ4= 2023-10-17T18:43:06.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ofjrwbLLiAAeS17VzynfsLNJGJ4=', 'Content-Length': '0'} 2023-10-17T18:43:06.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b1ba2f07a5f8f11-00652ed5ba-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.866 DEBUG:boto:path=/hbmuog-49/obj8 2023-10-17T18:43:06.867 DEBUG:boto:auth_path=/hbmuog-49/obj8 2023-10-17T18:43:06.867 DEBUG:boto:Method: GET 2023-10-17T18:43:06.867 DEBUG:boto:Path: /hbmuog-49/obj8 2023-10-17T18:43:06.868 DEBUG:boto:Data: 2023-10-17T18:43:06.868 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.869 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.869 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.869 DEBUG:boto:Params: {} 2023-10-17T18:43:06.870 DEBUG:boto:Token: None 2023-10-17T18:43:06.870 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/obj8 2023-10-17T18:43:06.871 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ofjrwbLLiAAeS17VzynfsLNJGJ4= 2023-10-17T18:43:06.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ofjrwbLLiAAeS17VzynfsLNJGJ4=', 'Content-Length': '0'} 2023-10-17T18:43:06.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:40:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-49:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088bc3aecd89265a8-00652ed5ba-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.874 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.874 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.875 DEBUG:boto:path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.875 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.875 DEBUG:boto:Method: GET 2023-10-17T18:43:06.876 DEBUG:boto:Path: /hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.876 DEBUG:boto:Data: 2023-10-17T18:43:06.877 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.877 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:06.877 DEBUG:boto:Port: 8001 2023-10-17T18:43:06.878 DEBUG:boto:Params: {} 2023-10-17T18:43:06.878 DEBUG:boto:Token: None 2023-10-17T18:43:06.879 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.879 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad565b5d4bde28f-00652ed5ba-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.882 DEBUG:boto:b'hbmuog-49obj81000falseobj82023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.883 DEBUG:boto:path=/hbmuog-49/ 2023-10-17T18:43:06.883 DEBUG:boto:auth_path=/hbmuog-49/ 2023-10-17T18:43:06.884 DEBUG:boto:path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.884 DEBUG:boto:auth_path=/hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.885 DEBUG:boto:Method: GET 2023-10-17T18:43:06.885 DEBUG:boto:Path: /hbmuog-49/?prefix=obj8 2023-10-17T18:43:06.885 DEBUG:boto:Data: 2023-10-17T18:43:06.886 DEBUG:boto:Headers: {} 2023-10-17T18:43:06.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:06.886 DEBUG:boto:Port: 8000 2023-10-17T18:43:06.887 DEBUG:boto:Params: {} 2023-10-17T18:43:06.887 DEBUG:boto:Token: None 2023-10-17T18:43:06.888 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:06 GMT /hbmuog-49/ 2023-10-17T18:43:06.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ= 2023-10-17T18:43:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2aA1GjoQNsN6G0QY4uoFGYXfouQ=', 'Content-Length': '0'} 2023-10-17T18:43:06.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000090a2d87d23dca96-00652ed5ba-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:06.891 DEBUG:boto:b'hbmuog-49obj81000falseobj82023-10-17T18:40:43.085Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:06.892 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-49 zones={a2, a1} 2023-10-17T18:43:06.892 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T18:43:06.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:06.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:06.894 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:06.944 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:06.944 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:06.991 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:43:06.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-10-17T18:43:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:43.351254Z" 2023-10-17T18:43:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:06.992 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-17T18:43:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:48.557155Z" 2023-10-17T18:43:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-10-17T18:43:06.993 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:49.649816Z" 2023-10-17T18:43:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-17T18:43:06.994 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:49.649846Z" 2023-10-17T18:43:06.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:06.995 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:43:06.996 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000006851', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000009807'} 2023-10-17T18:43:06.996 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:43:06.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:06.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:06.997 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:07.049 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:07.050 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:07.092 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:43:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:43:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:43:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:43:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:43:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 9938396449720538791 2023-10-17T18:43:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:43:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:43:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006851", 2023-10-17T18:43:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:43.351254Z" 2023-10-17T18:43:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:48.557155Z" 2023-10-17T18:43:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:43:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004595", 2023-10-17T18:43:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:49.649816Z" 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:43:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-17T18:43:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:49.649846Z" 2023-10-17T18:43:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:43:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:43:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:43:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:43:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:43:07Z" 2023-10-17T18:43:07.104 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:43:07.105 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9938396449720538791 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006851", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:40:43.351254Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:40:48.557155Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004595", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:40:49.649816Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009807", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:40:49.649846Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:43:07Z" } 2023-10-17T18:43:07.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006851', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:40:43.351254Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:40:48.557155Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004595', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:40:49.649816Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009807', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:40:49.649846Z'}}] 2023-10-17T18:43:07.106 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006851', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000009807'} 2023-10-17T18:43:07.106 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006851', 1: '00000000000000000000:00000000000000006576', 2: '00000000000000000000:00000000000000004595', 3: '00000000000000000000:00000000000000009807'} 2023-10-17T18:43:07.106 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T18:43:07.107 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T18:43:07.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:07.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:07.108 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:07.157 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:07.157 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:07.205 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:43:07.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.205 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004174", 2023-10-17T18:43:07.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:40:51.346744Z" 2023-10-17T18:43:07.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.206 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004739", 2023-10-17T18:43:07.206 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:41:13.353259Z" 2023-10-17T18:43:07.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.207 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003326", 2023-10-17T18:43:07.207 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:41:13.355008Z" 2023-10-17T18:43:07.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.208 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008679", 2023-10-17T18:43:07.208 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:41:13.356462Z" 2023-10-17T18:43:07.208 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.208 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:43:07.209 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004174', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000003326', 3: '00000000000000000000:00000000000000008679'} 2023-10-17T18:43:07.210 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:43:07.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:07.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:07.211 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:07.264 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:07.264 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:07.308 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:43:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:43:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:43:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:43:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:43:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T18:43:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:43:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:43:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004174", 2023-10-17T18:43:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:40:51.346744Z" 2023-10-17T18:43:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:43:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004739", 2023-10-17T18:43:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:41:13.353259Z" 2023-10-17T18:43:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:43:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000003326", 2023-10-17T18:43:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:41:13.355008Z" 2023-10-17T18:43:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008679", 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:41:13.356462Z" 2023-10-17T18:43:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:43:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:43:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T18:43:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T18:43:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:43:07Z" 2023-10-17T18:43:07.320 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:43:07.321 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:40:51.346744Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:41:13.353259Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:41:13.355008Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008679", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:41:13.356462Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T18:43:07Z" } 2023-10-17T18:43:07.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:40:51.346744Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:41:13.353259Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:41:13.355008Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008679', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:41:13.356462Z'}}] 2023-10-17T18:43:07.321 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004174', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000003326', 3: '00000000000000000000:00000000000000008679'} 2023-10-17T18:43:07.322 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004174', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000003326', 3: '00000000000000000000:00000000000000008679'} 2023-10-17T18:43:07.322 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T18:43:07.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-10-17T18:43:07.324 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-50 2023-10-17T18:43:07.324 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:07.325 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:07.325 DEBUG:boto:Method: PUT 2023-10-17T18:43:07.326 DEBUG:boto:Path: /hbmuog-50/ 2023-10-17T18:43:07.326 DEBUG:boto:Data: 2023-10-17T18:43:07.326 DEBUG:boto:Headers: {} 2023-10-17T18:43:07.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:07.327 DEBUG:boto:Port: 8000 2023-10-17T18:43:07.327 DEBUG:boto:Params: {} 2023-10-17T18:43:07.328 DEBUG:boto:Token: None 2023-10-17T18:43:07.328 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:43:07 GMT /hbmuog-50/ 2023-10-17T18:43:07.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:18V5Cha4vKBoUfOYd5AN7c7KQmE= 2023-10-17T18:43:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:18V5Cha4vKBoUfOYd5AN7c7KQmE=', 'Content-Length': '0'} 2023-10-17T18:43:07.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025b2738136b11040-00652ed5bb-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:07.343 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-51 2023-10-17T18:43:07.344 DEBUG:boto:path=/hbmuog-51/ 2023-10-17T18:43:07.344 DEBUG:boto:auth_path=/hbmuog-51/ 2023-10-17T18:43:07.344 DEBUG:boto:Method: PUT 2023-10-17T18:43:07.345 DEBUG:boto:Path: /hbmuog-51/ 2023-10-17T18:43:07.345 DEBUG:boto:Data: 2023-10-17T18:43:07.345 DEBUG:boto:Headers: {} 2023-10-17T18:43:07.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:07.346 DEBUG:boto:Port: 8001 2023-10-17T18:43:07.347 DEBUG:boto:Params: {} 2023-10-17T18:43:07.347 DEBUG:boto:Token: None 2023-10-17T18:43:07.347 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:43:07 GMT /hbmuog-51/ 2023-10-17T18:43:07.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LONPb/VDY6Xq9JEYlrEZ7WTqL7Y= 2023-10-17T18:43:07.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LONPb/VDY6Xq9JEYlrEZ7WTqL7Y=', 'Content-Length': '0'} 2023-10-17T18:43:07.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034db0d7a5f384e6c-00652ed5bb-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:07.372 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:07.372 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:07.373 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploads 2023-10-17T18:43:07.373 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploads 2023-10-17T18:43:07.373 DEBUG:boto:Method: POST 2023-10-17T18:43:07.374 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploads 2023-10-17T18:43:07.374 DEBUG:boto:Data: 2023-10-17T18:43:07.374 DEBUG:boto:Headers: {} 2023-10-17T18:43:07.375 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:07.375 DEBUG:boto:Port: 8000 2023-10-17T18:43:07.375 DEBUG:boto:Params: {} 2023-10-17T18:43:07.376 DEBUG:boto:Token: None 2023-10-17T18:43:07.376 DEBUG:boto:StringToSign: POST Tue, 17 Oct 2023 18:43:07 GMT /hbmuog-50/MULTIPART?uploads 2023-10-17T18:43:07.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1PET0xM6W1PDi8IRXh3nyuBF8bM= 2023-10-17T18:43:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1PET0xM6W1PDi8IRXh3nyuBF8bM=', 'Content-Length': '0'} 2023-10-17T18:43:09.399 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000726d06b95df69bbc-00652ed5bb-162c-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.442 DEBUG:boto:b'hbmuog-50MULTIPART2~8OyUO2x097gQTihSz9lupptYrISL9ph' 2023-10-17T18:43:09.476 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.504 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.505 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=1 2023-10-17T18:43:09.505 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=1 2023-10-17T18:43:09.505 DEBUG:boto:Method: PUT 2023-10-17T18:43:09.506 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=1 2023-10-17T18:43:09.506 DEBUG:boto:Data: 2023-10-17T18:43:09.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-17T18:43:09.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.507 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.508 DEBUG:boto:Params: {} 2023-10-17T18:43:09.508 DEBUG:boto:Token: None 2023-10-17T18:43:09.509 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?partNumber=1&uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gqZoZTds7inqd9t6m7ki9O1e6Jo= 2023-10-17T18:43:09.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gqZoZTds7inqd9t6m7ki9O1e6Jo='} 2023-10-17T18:43:09.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9e5eca1e4911f4b-00652ed5bd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.606 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.607 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.607 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=2 2023-10-17T18:43:09.608 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=2 2023-10-17T18:43:09.608 DEBUG:boto:Method: PUT 2023-10-17T18:43:09.609 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=2 2023-10-17T18:43:09.609 DEBUG:boto:Data: 2023-10-17T18:43:09.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-17T18:43:09.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.610 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.611 DEBUG:boto:Params: {} 2023-10-17T18:43:09.611 DEBUG:boto:Token: None 2023-10-17T18:43:09.611 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?partNumber=2&uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1AHRzuTT1bSOCmmzTrgY7FB2Bx0= 2023-10-17T18:43:09.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1AHRzuTT1bSOCmmzTrgY7FB2Bx0='} 2023-10-17T18:43:09.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000baef9d8ea6e68546-00652ed5bd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.682 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.683 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.683 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=3 2023-10-17T18:43:09.683 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=3 2023-10-17T18:43:09.684 DEBUG:boto:Method: PUT 2023-10-17T18:43:09.684 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=3 2023-10-17T18:43:09.685 DEBUG:boto:Data: 2023-10-17T18:43:09.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-17T18:43:09.685 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.686 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.686 DEBUG:boto:Params: {} 2023-10-17T18:43:09.686 DEBUG:boto:Token: None 2023-10-17T18:43:09.687 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?partNumber=3&uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.687 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZrkjyiMecqS+p3AdJm6Y4bargL4= 2023-10-17T18:43:09.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZrkjyiMecqS+p3AdJm6Y4bargL4='} 2023-10-17T18:43:09.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008687ac39505b4902-00652ed5bd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.758 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.758 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.759 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=4 2023-10-17T18:43:09.759 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=4 2023-10-17T18:43:09.760 DEBUG:boto:Method: PUT 2023-10-17T18:43:09.760 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph&partNumber=4 2023-10-17T18:43:09.760 DEBUG:boto:Data: 2023-10-17T18:43:09.761 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-17T18:43:09.761 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.762 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.762 DEBUG:boto:Params: {} 2023-10-17T18:43:09.762 DEBUG:boto:Token: None 2023-10-17T18:43:09.763 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?partNumber=4&uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.763 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kPbr/ukNluOExg/XbuLEe6lpMVU= 2023-10-17T18:43:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kPbr/ukNluOExg/XbuLEe6lpMVU='} 2023-10-17T18:43:09.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058839fe270de805c-00652ed5bd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.816 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.817 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.817 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.817 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.818 DEBUG:boto:Method: GET 2023-10-17T18:43:09.818 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.818 DEBUG:boto:Data: 2023-10-17T18:43:09.819 DEBUG:boto:Headers: {} 2023-10-17T18:43:09.819 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.820 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.820 DEBUG:boto:Params: {} 2023-10-17T18:43:09.820 DEBUG:boto:Token: None 2023-10-17T18:43:09.821 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ztmeTx6tckolOq2pNzlSPjXX6Gc= 2023-10-17T18:43:09.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ztmeTx6tckolOq2pNzlSPjXX6Gc=', 'Content-Length': '0'} 2023-10-17T18:43:09.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a50889ef13189a9-00652ed5bd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.866 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.867 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:09.867 DEBUG:boto:path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.868 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.868 DEBUG:boto:Method: POST 2023-10-17T18:43:09.868 DEBUG:boto:Path: /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.869 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-10-17T18:43:09.869 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-10-17T18:43:09.869 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:09.870 DEBUG:boto:Port: 8000 2023-10-17T18:43:09.870 DEBUG:boto:Params: {} 2023-10-17T18:43:09.870 DEBUG:boto:Token: None 2023-10-17T18:43:09.871 DEBUG:boto:StringToSign: POST text/xml Tue, 17 Oct 2023 18:43:09 GMT /hbmuog-50/MULTIPART?uploadId=2~8OyUO2x097gQTihSz9lupptYrISL9ph 2023-10-17T18:43:09.871 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r9mAyeULO/RG9dyScrtrOSwJKz0= 2023-10-17T18:43:09.872 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:09 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r9mAyeULO/RG9dyScrtrOSwJKz0=', 'Content-Length': '460'} 2023-10-17T18:43:09.880 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cf049edb34a3612-00652ed5bd-162c-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:09.881 DEBUG:boto:http://smithi049.front.sepia.ceph.com:8000/hbmuog-50/MULTIPARThbmuog-50MULTIPART 2023-10-17T18:43:09.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:09.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:09.882 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:09.932 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:09.932 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:09.962 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:43:09.974 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:43:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568187.362578_615.1", 2023-10-17T18:43:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:43:07.362578Z" 2023-10-17T18:43:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:43:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:09.977 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:43:09.978 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568187.362578_615.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:09.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:43:09.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:09.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:09.980 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:10.031 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:10.031 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:10.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:10.060+0000 7eff08691780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:10.062 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:10.061+0000 7eff08691780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:10.075 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:43:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:43:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:43:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:43:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:43:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:43:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:43:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:43:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568187.362578_615.1", 2023-10-17T18:43:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:43:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:07.362578Z", 2023-10-17T18:43:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:43:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:43:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:43:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:43:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:43:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:43:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:43:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:43:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:43:10Z" 2023-10-17T18:43:10.088 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:43:10.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568187.362578_615.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:43:07.362578Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:43:10Z" } 2023-10-17T18:43:10.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568187.362578_615.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:43:07.362578Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:43:10.090 DEBUG:rgw_multi.tests:log_status={0: '1_1697568187.362578_615.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:10.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568187.362578_615.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:10.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:43:10.091 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-50 2023-10-17T18:43:10.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:10.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:10.092 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:43:10.150 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:10.150 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:10.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:10.183+0000 7fc1f266d780 1 waiting to reach incremental sync.. 2023-10-17T18:43:15.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:15.183+0000 7fc1f266d780 1 waiting to reach incremental sync.. 2023-10-17T18:43:20.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:20.184+0000 7fc1f266d780 1 waiting to reach incremental sync.. 2023-10-17T18:43:25.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:25.186+0000 7fc1f266d780 1 bucket sync caught up with source: 2023-10-17T18:43:25.188 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000011.51.6, , , , , ] 2023-10-17T18:43:25.188 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000011.51.6, , , , , ] 2023-10-17T18:43:25.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:25.186+0000 7fc1f266d780 0 bucket checkpoint complete 2023-10-17T18:43:25.206 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-50 2023-10-17T18:43:25.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:25.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:25.208 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:43:25.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:25.272 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:25.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:25.310+0000 7effbbcfa780 1 waiting to reach incremental sync.. 2023-10-17T18:43:30.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:30.312+0000 7effbbcfa780 1 bucket sync caught up with source: 2023-10-17T18:43:30.314 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.282.6, , , , , ] 2023-10-17T18:43:30.314 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.282.6, , , , , ] 2023-10-17T18:43:30.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:30.312+0000 7effbbcfa780 0 bucket checkpoint complete 2023-10-17T18:43:30.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-50 zones={a2, a1} 2023-10-17T18:43:30.334 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.334 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.335 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.335 DEBUG:boto:Path: /hbmuog-50/ 2023-10-17T18:43:30.335 DEBUG:boto:Data: 2023-10-17T18:43:30.336 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.337 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.337 DEBUG:boto:Params: {} 2023-10-17T18:43:30.337 DEBUG:boto:Token: None 2023-10-17T18:43:30.338 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rY9ZQtTANWw60QzhJ1KRL7lXj5E= 2023-10-17T18:43:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rY9ZQtTANWw60QzhJ1KRL7lXj5E=', 'Content-Length': '0'} 2023-10-17T18:43:30.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047d161dc679b6f80-00652ed5d2-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.342 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.342 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.343 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.343 DEBUG:boto:Path: /hbmuog-50/ 2023-10-17T18:43:30.344 DEBUG:boto:Data: 2023-10-17T18:43:30.344 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.345 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.345 DEBUG:boto:Params: {} 2023-10-17T18:43:30.345 DEBUG:boto:Token: None 2023-10-17T18:43:30.346 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rY9ZQtTANWw60QzhJ1KRL7lXj5E= 2023-10-17T18:43:30.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rY9ZQtTANWw60QzhJ1KRL7lXj5E=', 'Content-Length': '0'} 2023-10-17T18:43:30.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d12b9effc8fc5d0b-00652ed5d2-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:43:30.350 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.351 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.351 DEBUG:boto:path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.351 DEBUG:boto:auth_path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.352 DEBUG:boto:Method: GET 2023-10-17T18:43:30.352 DEBUG:boto:Path: /hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.353 DEBUG:boto:Data: 2023-10-17T18:43:30.353 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.354 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.354 DEBUG:boto:Params: {} 2023-10-17T18:43:30.355 DEBUG:boto:Token: None 2023-10-17T18:43:30.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/?versions 2023-10-17T18:43:30.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU= 2023-10-17T18:43:30.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU=', 'Content-Length': '0'} 2023-10-17T18:43:30.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b5778f10a4054e-00652ed5d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.359 DEBUG:boto:b'hbmuog-50999falseMULTIPARTnulltrue2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.359 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-17T18:43:30.360 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:43:30.360 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.361 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.361 DEBUG:boto:path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.361 DEBUG:boto:auth_path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.362 DEBUG:boto:Method: GET 2023-10-17T18:43:30.362 DEBUG:boto:Path: /hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.362 DEBUG:boto:Data: 2023-10-17T18:43:30.363 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.364 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.364 DEBUG:boto:Params: {} 2023-10-17T18:43:30.364 DEBUG:boto:Token: None 2023-10-17T18:43:30.365 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/?versions 2023-10-17T18:43:30.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU= 2023-10-17T18:43:30.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU=', 'Content-Length': '0'} 2023-10-17T18:43:30.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053846f10e91f2a2d-00652ed5d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.368 DEBUG:boto:b'hbmuog-50999falseMULTIPARTnulltrue2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-17T18:43:30.370 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.370 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.370 DEBUG:boto:path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.371 DEBUG:boto:auth_path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.371 DEBUG:boto:Method: GET 2023-10-17T18:43:30.372 DEBUG:boto:Path: /hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.372 DEBUG:boto:Data: 2023-10-17T18:43:30.372 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.373 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.373 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.374 DEBUG:boto:Params: {} 2023-10-17T18:43:30.374 DEBUG:boto:Token: None 2023-10-17T18:43:30.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/?versions 2023-10-17T18:43:30.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU= 2023-10-17T18:43:30.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU=', 'Content-Length': '0'} 2023-10-17T18:43:30.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc9d788b9a7411c-00652ed5d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.378 DEBUG:boto:b'hbmuog-50999falseMULTIPARTnulltrue2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.379 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.379 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.379 DEBUG:boto:path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.380 DEBUG:boto:auth_path=/hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.380 DEBUG:boto:Method: GET 2023-10-17T18:43:30.381 DEBUG:boto:Path: /hbmuog-50/?versions&max-keys=999 2023-10-17T18:43:30.381 DEBUG:boto:Data: 2023-10-17T18:43:30.381 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.382 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.382 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.383 DEBUG:boto:Params: {} 2023-10-17T18:43:30.383 DEBUG:boto:Token: None 2023-10-17T18:43:30.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/?versions 2023-10-17T18:43:30.384 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU= 2023-10-17T18:43:30.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lc2Uma4/YL0cAmQ8jFnUYFPvfqU=', 'Content-Length': '0'} 2023-10-17T18:43:30.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c438bf0bc242be6-00652ed5d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.387 DEBUG:boto:b'hbmuog-50999falseMULTIPARTnulltrue2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-17T18:43:30.388 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.388 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.389 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.389 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.390 DEBUG:boto:Method: GET 2023-10-17T18:43:30.390 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.391 DEBUG:boto:Data: 2023-10-17T18:43:30.391 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.392 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.392 DEBUG:boto:Params: {} 2023-10-17T18:43:30.393 DEBUG:boto:Token: None 2023-10-17T18:43:30.393 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U= 2023-10-17T18:43:30.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U=', 'Content-Length': '0'} 2023-10-17T18:43:30.398 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-50:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000007c54c703525a1949-00652ed5d2-1630-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.466 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.466 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.467 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.467 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.468 DEBUG:boto:Method: GET 2023-10-17T18:43:30.468 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.468 DEBUG:boto:Data: 2023-10-17T18:43:30.469 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.469 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.470 DEBUG:boto:Params: {} 2023-10-17T18:43:30.470 DEBUG:boto:Token: None 2023-10-17T18:43:30.471 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U= 2023-10-17T18:43:30.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U=', 'Content-Length': '0'} 2023-10-17T18:43:30.481 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000405fbfc31acf7ffe-00652ed5d2-162c-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.565 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.566 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.566 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.566 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.567 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.567 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.567 DEBUG:boto:Data: 2023-10-17T18:43:30.568 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.569 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.569 DEBUG:boto:Params: {} 2023-10-17T18:43:30.569 DEBUG:boto:Token: None 2023-10-17T18:43:30.570 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5roxUwK0dylc5DpPudVirRjGAaA= 2023-10-17T18:43:30.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5roxUwK0dylc5DpPudVirRjGAaA=', 'Content-Length': '0'} 2023-10-17T18:43:30.573 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-50:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002429372a8b255a2c-00652ed5d2-1630-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.573 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.574 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.574 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.575 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.575 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.575 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.576 DEBUG:boto:Data: 2023-10-17T18:43:30.576 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.577 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.577 DEBUG:boto:Params: {} 2023-10-17T18:43:30.578 DEBUG:boto:Token: None 2023-10-17T18:43:30.578 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5roxUwK0dylc5DpPudVirRjGAaA= 2023-10-17T18:43:30.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5roxUwK0dylc5DpPudVirRjGAaA=', 'Content-Length': '0'} 2023-10-17T18:43:30.581 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009358edb8995c8383-00652ed5d2-162c-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-17T18:43:30.582 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.582 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.583 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.583 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.584 DEBUG:boto:Method: GET 2023-10-17T18:43:30.584 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.584 DEBUG:boto:Data: 2023-10-17T18:43:30.585 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.586 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.586 DEBUG:boto:Params: {} 2023-10-17T18:43:30.587 DEBUG:boto:Token: None 2023-10-17T18:43:30.587 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U= 2023-10-17T18:43:30.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U=', 'Content-Length': '0'} 2023-10-17T18:43:30.591 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-50:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002964cae98af8d1c0-00652ed5d2-1630-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.665 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.666 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.666 DEBUG:boto:path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.666 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.667 DEBUG:boto:Method: GET 2023-10-17T18:43:30.667 DEBUG:boto:Path: /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.668 DEBUG:boto:Data: 2023-10-17T18:43:30.668 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.669 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.669 DEBUG:boto:Params: {} 2023-10-17T18:43:30.670 DEBUG:boto:Token: None 2023-10-17T18:43:30.670 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART?versionId=null 2023-10-17T18:43:30.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U= 2023-10-17T18:43:30.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewtRtovlgbDHpHTurxmJBVdvP4U=', 'Content-Length': '0'} 2023-10-17T18:43:30.680 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000566eca26ad79c387-00652ed5d2-162c-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.764 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.765 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.765 DEBUG:boto:path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.766 DEBUG:boto:auth_path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.766 DEBUG:boto:Method: GET 2023-10-17T18:43:30.766 DEBUG:boto:Path: /hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.767 DEBUG:boto:Data: 2023-10-17T18:43:30.767 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.768 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.768 DEBUG:boto:Params: {} 2023-10-17T18:43:30.769 DEBUG:boto:Token: None 2023-10-17T18:43:30.769 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.770 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw= 2023-10-17T18:43:30.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw=', 'Content-Length': '0'} 2023-10-17T18:43:30.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce2bfc811af06441-00652ed5d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.773 DEBUG:boto:b'hbmuog-50MULTIPART1000falseMULTIPART2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.774 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.774 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.775 DEBUG:boto:path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.775 DEBUG:boto:auth_path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.776 DEBUG:boto:Method: GET 2023-10-17T18:43:30.776 DEBUG:boto:Path: /hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.776 DEBUG:boto:Data: 2023-10-17T18:43:30.777 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.777 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.778 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.778 DEBUG:boto:Params: {} 2023-10-17T18:43:30.778 DEBUG:boto:Token: None 2023-10-17T18:43:30.779 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.779 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw= 2023-10-17T18:43:30.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw=', 'Content-Length': '0'} 2023-10-17T18:43:30.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004120c08215fb74fa-00652ed5d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.783 DEBUG:boto:b'hbmuog-50MULTIPART1000falseMULTIPART2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.783 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.783 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.784 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.784 DEBUG:boto:Path: /hbmuog-50/MULTIPART 2023-10-17T18:43:30.785 DEBUG:boto:Data: 2023-10-17T18:43:30.785 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.786 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.786 DEBUG:boto:Params: {} 2023-10-17T18:43:30.786 DEBUG:boto:Token: None 2023-10-17T18:43:30.787 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART 2023-10-17T18:43:30.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ST2RbziI6Illtkpu7KMpWXcoyV4= 2023-10-17T18:43:30.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ST2RbziI6Illtkpu7KMpWXcoyV4=', 'Content-Length': '0'} 2023-10-17T18:43:30.790 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-50:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001d8a978f759e9dee-00652ed5d2-1630-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.790 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.791 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.791 DEBUG:boto:Method: HEAD 2023-10-17T18:43:30.791 DEBUG:boto:Path: /hbmuog-50/MULTIPART 2023-10-17T18:43:30.792 DEBUG:boto:Data: 2023-10-17T18:43:30.792 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.793 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.793 DEBUG:boto:Params: {} 2023-10-17T18:43:30.794 DEBUG:boto:Token: None 2023-10-17T18:43:30.794 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART 2023-10-17T18:43:30.795 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ST2RbziI6Illtkpu7KMpWXcoyV4= 2023-10-17T18:43:30.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ST2RbziI6Illtkpu7KMpWXcoyV4=', 'Content-Length': '0'} 2023-10-17T18:43:30.797 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000032549698d5848a26-00652ed5d2-162c-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-17T18:43:30.798 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.798 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.799 DEBUG:boto:Method: GET 2023-10-17T18:43:30.799 DEBUG:boto:Path: /hbmuog-50/MULTIPART 2023-10-17T18:43:30.799 DEBUG:boto:Data: 2023-10-17T18:43:30.800 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.801 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.801 DEBUG:boto:Params: {} 2023-10-17T18:43:30.801 DEBUG:boto:Token: None 2023-10-17T18:43:30.802 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART 2023-10-17T18:43:30.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RHPdQe60hiyetIbRUkzrdZjnRhE= 2023-10-17T18:43:30.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RHPdQe60hiyetIbRUkzrdZjnRhE=', 'Content-Length': '0'} 2023-10-17T18:43:30.806 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-50:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f8862adb7eba8701-00652ed5d2-1630-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.880 DEBUG:boto:path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.881 DEBUG:boto:auth_path=/hbmuog-50/MULTIPART 2023-10-17T18:43:30.881 DEBUG:boto:Method: GET 2023-10-17T18:43:30.882 DEBUG:boto:Path: /hbmuog-50/MULTIPART 2023-10-17T18:43:30.882 DEBUG:boto:Data: 2023-10-17T18:43:30.883 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.883 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.883 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.884 DEBUG:boto:Params: {} 2023-10-17T18:43:30.884 DEBUG:boto:Token: None 2023-10-17T18:43:30.885 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/MULTIPART 2023-10-17T18:43:30.885 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RHPdQe60hiyetIbRUkzrdZjnRhE= 2023-10-17T18:43:30.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RHPdQe60hiyetIbRUkzrdZjnRhE=', 'Content-Length': '0'} 2023-10-17T18:43:30.895 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f39a66c16d3a8471-00652ed5d2-162c-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.978 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.979 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.979 DEBUG:boto:path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.980 DEBUG:boto:auth_path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.980 DEBUG:boto:Method: GET 2023-10-17T18:43:30.981 DEBUG:boto:Path: /hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.981 DEBUG:boto:Data: 2023-10-17T18:43:30.981 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:30.982 DEBUG:boto:Port: 8001 2023-10-17T18:43:30.982 DEBUG:boto:Params: {} 2023-10-17T18:43:30.983 DEBUG:boto:Token: None 2023-10-17T18:43:30.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw= 2023-10-17T18:43:30.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw=', 'Content-Length': '0'} 2023-10-17T18:43:30.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e2811a538b9e38-00652ed5d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.987 DEBUG:boto:b'hbmuog-50MULTIPART1000falseMULTIPART2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.988 DEBUG:boto:path=/hbmuog-50/ 2023-10-17T18:43:30.988 DEBUG:boto:auth_path=/hbmuog-50/ 2023-10-17T18:43:30.989 DEBUG:boto:path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.989 DEBUG:boto:auth_path=/hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.990 DEBUG:boto:Method: GET 2023-10-17T18:43:30.990 DEBUG:boto:Path: /hbmuog-50/?prefix=MULTIPART 2023-10-17T18:43:30.991 DEBUG:boto:Data: 2023-10-17T18:43:30.991 DEBUG:boto:Headers: {} 2023-10-17T18:43:30.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:30.992 DEBUG:boto:Port: 8000 2023-10-17T18:43:30.992 DEBUG:boto:Params: {} 2023-10-17T18:43:30.993 DEBUG:boto:Token: None 2023-10-17T18:43:30.993 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:30 GMT /hbmuog-50/ 2023-10-17T18:43:30.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw= 2023-10-17T18:43:30.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:30 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kptOGXIPJRUTvcysFiWGMYt7zbw=', 'Content-Length': '0'} 2023-10-17T18:43:30.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf31ed7e713379f-00652ed5d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:43:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:30.997 DEBUG:boto:b'hbmuog-50MULTIPART1000falseMULTIPART2023-10-17T18:43:09.875Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:43:30.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-50 zones={a2, a1} 2023-10-17T18:43:30.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-10-17T18:43:30.999 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-52 2023-10-17T18:43:30.999 DEBUG:boto:path=/hbmuog-52/ 2023-10-17T18:43:31.000 DEBUG:boto:auth_path=/hbmuog-52/ 2023-10-17T18:43:31.000 DEBUG:boto:Method: PUT 2023-10-17T18:43:31.000 DEBUG:boto:Path: /hbmuog-52/ 2023-10-17T18:43:31.001 DEBUG:boto:Data: 2023-10-17T18:43:31.001 DEBUG:boto:Headers: {} 2023-10-17T18:43:31.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:31.002 DEBUG:boto:Port: 8000 2023-10-17T18:43:31.002 DEBUG:boto:Params: {} 2023-10-17T18:43:31.003 DEBUG:boto:Token: None 2023-10-17T18:43:31.003 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:43:31 GMT /hbmuog-52/ 2023-10-17T18:43:31.003 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H+pA9Fx8hp5zD3ST5+OKYrMFbLc= 2023-10-17T18:43:31.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H+pA9Fx8hp5zD3ST5+OKYrMFbLc=', 'Content-Length': '0'} 2023-10-17T18:43:31.018 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eff472a673581fe5-00652ed5d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:31.018 DEBUG:boto:path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:31.019 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:31.019 DEBUG:boto:Method: PUT 2023-10-17T18:43:31.020 DEBUG:boto:Path: /hbmuog-52/testobj-sse-c 2023-10-17T18:43:31.020 DEBUG:boto:Data: 2023-10-17T18:43:31.020 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:43:31.021 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:31.021 DEBUG:boto:Port: 8000 2023-10-17T18:43:31.021 DEBUG:boto:Params: {} 2023-10-17T18:43:31.022 DEBUG:boto:Token: None 2023-10-17T18:43:31.022 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 17 Oct 2023 18:43:31 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hbmuog-52/testobj-sse-c 2023-10-17T18:43:31.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OLgh3jmPlrnEHRA+kOb3dt3h4b0= 2023-10-17T18:43:31.023 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OLgh3jmPlrnEHRA+kOb3dt3h4b0='} 2023-10-17T18:43:31.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000d701523753779792-00652ed5d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:31.030 DEBUG:boto:path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:31.030 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:31.031 DEBUG:boto:Method: PUT 2023-10-17T18:43:31.031 DEBUG:boto:Path: /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:31.031 DEBUG:boto:Data: 2023-10-17T18:43:31.032 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-17T18:43:31.032 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:31.032 DEBUG:boto:Port: 8000 2023-10-17T18:43:31.033 DEBUG:boto:Params: {} 2023-10-17T18:43:31.033 DEBUG:boto:Token: None 2023-10-17T18:43:31.034 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 17 Oct 2023 18:43:31 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:31.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i5tIttedZx5xGS03a/Dv4X3796Q= 2023-10-17T18:43:31.035 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:31 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i5tIttedZx5xGS03a/Dv4X3796Q='} 2023-10-17T18:43:31.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6aGJtdW9nLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000f699da308323b29a-00652ed5d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:31.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:31.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:31.080 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:31.131 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:31.131 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:31.165 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:43:31.177 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:43:31.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.177 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568211.016568_619.1", 2023-10-17T18:43:31.178 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:43:31.016568Z" 2023-10-17T18:43:31.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.178 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:31.179 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:43:31.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.179 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:31.179 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:31.180 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:43:31.181 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568211.016568_619.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:31.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:43:31.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:31.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:31.183 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:31.234 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:31.234 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:31.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:31.263+0000 7f8ff1424780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:31.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:31.265+0000 7f8ff1424780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:31.279 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:43:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:43:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:43:31.279 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:43:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:43:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:43:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:31.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:43:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:43:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:31.281 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568211.016568_619.1", 2023-10-17T18:43:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:43:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:31.282 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:31.016568Z", 2023-10-17T18:43:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.283 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:43:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:31.284 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:43:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:31.286 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:31.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:43:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:31.288 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:31.289 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:43:31.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:43:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:43:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:43:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:31.291 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:43:31Z" 2023-10-17T18:43:31.292 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:43:31.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568211.016568_619.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:43:31.016568Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:43:31Z" } 2023-10-17T18:43:31.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568211.016568_619.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:43:31.016568Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:43:31.293 DEBUG:rgw_multi.tests:log_status={0: '1_1697568211.016568_619.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:31.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568211.016568_619.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:31.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:43:31.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:31.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:31.295 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:43:31.353 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:31.353 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:31.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:31.386+0000 7fb7f5775780 1 waiting to reach incremental sync.. 2023-10-17T18:43:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:36.388+0000 7fb7f5775780 1 waiting to reach incremental sync.. 2023-10-17T18:43:41.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:41.390+0000 7fb7f5775780 1 bucket sync caught up with source: 2023-10-17T18:43:41.391 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.32.6, , , , 00000000001.42.6, , , , , ] 2023-10-17T18:43:41.391 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.32.6, , , , 00000000001.42.6, , , , , ] 2023-10-17T18:43:41.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:41.390+0000 7fb7f5775780 0 bucket checkpoint complete 2023-10-17T18:43:41.407 DEBUG:boto:path=/hbmuog-52/ 2023-10-17T18:43:41.408 DEBUG:boto:auth_path=/hbmuog-52/ 2023-10-17T18:43:41.408 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.409 DEBUG:boto:Path: /hbmuog-52/ 2023-10-17T18:43:41.409 DEBUG:boto:Data: 2023-10-17T18:43:41.410 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:41.411 DEBUG:boto:Port: 8001 2023-10-17T18:43:41.411 DEBUG:boto:Params: {} 2023-10-17T18:43:41.412 DEBUG:boto:Token: None 2023-10-17T18:43:41.412 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-52/ 2023-10-17T18:43:41.413 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pwDltNQxB6g6kse7hhjbfcGPFOM= 2023-10-17T18:43:41.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pwDltNQxB6g6kse7hhjbfcGPFOM=', 'Content-Length': '0'} 2023-10-17T18:43:41.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092a70dc26056e3b1-00652ed5dd-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.417 DEBUG:boto:path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.417 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.418 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.418 DEBUG:boto:Path: /hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.418 DEBUG:boto:Data: 2023-10-17T18:43:41.419 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-17T18:43:41.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:41.420 DEBUG:boto:Port: 8001 2023-10-17T18:43:41.420 DEBUG:boto:Params: {} 2023-10-17T18:43:41.421 DEBUG:boto:Token: None 2023-10-17T18:43:41.421 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.422 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hKMAmOqfllQGQjZ1e2h9of2jeE8= 2023-10-17T18:43:41.422 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hKMAmOqfllQGQjZ1e2h9of2jeE8=', 'Content-Length': '0'} 2023-10-17T18:43:41.425 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-52:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000007402087192ea448b-00652ed5dd-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.425 DEBUG:boto:path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.426 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.426 DEBUG:boto:Method: GET 2023-10-17T18:43:41.427 DEBUG:boto:Path: /hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.427 DEBUG:boto:Data: 2023-10-17T18:43:41.428 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-17T18:43:41.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:41.428 DEBUG:boto:Port: 8001 2023-10-17T18:43:41.429 DEBUG:boto:Params: {} 2023-10-17T18:43:41.429 DEBUG:boto:Token: None 2023-10-17T18:43:41.430 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:41 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hbmuog-52/testobj-sse-c 2023-10-17T18:43:41.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sVEoHT9/QeeafMblBmtyRngq2dc= 2023-10-17T18:43:41.431 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVEoHT9/QeeafMblBmtyRngq2dc=', 'Content-Length': '0'} 2023-10-17T18:43:41.433 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-52:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000002be7ff22689f17ce-00652ed5dd-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.434 DEBUG:boto:path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.434 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.435 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.435 DEBUG:boto:Path: /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.436 DEBUG:boto:Data: 2023-10-17T18:43:41.436 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.437 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:41.437 DEBUG:boto:Port: 8001 2023-10-17T18:43:41.437 DEBUG:boto:Params: {} 2023-10-17T18:43:41.438 DEBUG:boto:Token: None 2023-10-17T18:43:41.438 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rZXF7fMcZhn/zLibGgzy9XDC0xQ= 2023-10-17T18:43:41.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rZXF7fMcZhn/zLibGgzy9XDC0xQ=', 'Content-Length': '0'} 2023-10-17T18:43:41.441 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-52:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000004441b004d8107b68-00652ed5dd-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.442 DEBUG:boto:path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.443 DEBUG:boto:auth_path=/hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.443 DEBUG:boto:Method: GET 2023-10-17T18:43:41.443 DEBUG:boto:Path: /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.444 DEBUG:boto:Data: 2023-10-17T18:43:41.444 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:43:41.445 DEBUG:boto:Port: 8001 2023-10-17T18:43:41.446 DEBUG:boto:Params: {} 2023-10-17T18:43:41.446 DEBUG:boto:Token: None 2023-10-17T18:43:41.447 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-52/testobj-sse-kms 2023-10-17T18:43:41.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4YJZum8yeq8ZJ8nXOkkZOIB8gL4= 2023-10-17T18:43:41.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4YJZum8yeq8ZJ8nXOkkZOIB8gL4=', 'Content-Length': '0'} 2023-10-17T18:43:41.450 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-52:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000c8ea213b62b26c46-00652ed5dd-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-10-17T18:43:41.451 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-53 2023-10-17T18:43:41.452 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:43:41.452 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:43:41.453 DEBUG:boto:Method: PUT 2023-10-17T18:43:41.453 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:43:41.454 DEBUG:boto:Data: 2023-10-17T18:43:41.454 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.455 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.455 DEBUG:boto:Params: {} 2023-10-17T18:43:41.456 DEBUG:boto:Token: None 2023-10-17T18:43:41.456 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/ 2023-10-17T18:43:41.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xk0J0QfS4KLeEvJqxerXSp9/pmQ= 2023-10-17T18:43:41.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xk0J0QfS4KLeEvJqxerXSp9/pmQ=', 'Content-Length': '0'} 2023-10-17T18:43:41.474 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000183d7a1cf7427de5-00652ed5dd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.475 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:43:41.475 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:43:41.476 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.476 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:43:41.477 DEBUG:boto:Data: 2023-10-17T18:43:41.477 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.478 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.478 DEBUG:boto:Params: {} 2023-10-17T18:43:41.479 DEBUG:boto:Token: None 2023-10-17T18:43:41.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/ 2023-10-17T18:43:41.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs= 2023-10-17T18:43:41.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs=', 'Content-Length': '0'} 2023-10-17T18:43:41.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bc68b9980d53b31-00652ed5dd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.483 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:43:41.484 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:43:41.484 DEBUG:boto:Method: PUT 2023-10-17T18:43:41.485 DEBUG:boto:Path: /hbmuog-53/a 2023-10-17T18:43:41.485 DEBUG:boto:Data: 2023-10-17T18:43:41.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:43:41.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.486 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.487 DEBUG:boto:Params: {} 2023-10-17T18:43:41.487 DEBUG:boto:Token: None 2023-10-17T18:43:41.487 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/a 2023-10-17T18:43:41.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dDQjVUEL5AWW4qOt/wP7LMXnnDY= 2023-10-17T18:43:41.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dDQjVUEL5AWW4qOt/wP7LMXnnDY='} 2023-10-17T18:43:41.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fc2897e1a9890d4-00652ed5dd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.496 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:43:41.496 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:43:41.496 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.497 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:43:41.497 DEBUG:boto:Data: 2023-10-17T18:43:41.498 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.498 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.499 DEBUG:boto:Params: {} 2023-10-17T18:43:41.499 DEBUG:boto:Token: None 2023-10-17T18:43:41.500 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/ 2023-10-17T18:43:41.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs= 2023-10-17T18:43:41.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs=', 'Content-Length': '0'} 2023-10-17T18:43:41.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043f6746342cd99ec-00652ed5dd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.503 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:43:41.504 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:43:41.504 DEBUG:boto:Method: PUT 2023-10-17T18:43:41.505 DEBUG:boto:Path: /hbmuog-53/b 2023-10-17T18:43:41.505 DEBUG:boto:Data: 2023-10-17T18:43:41.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:43:41.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.506 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.507 DEBUG:boto:Params: {} 2023-10-17T18:43:41.507 DEBUG:boto:Token: None 2023-10-17T18:43:41.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/b 2023-10-17T18:43:41.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QIX12cCvKqKcXx2Jd+Zt+yJK/2w= 2023-10-17T18:43:41.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QIX12cCvKqKcXx2Jd+Zt+yJK/2w='} 2023-10-17T18:43:41.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b16b81a44c6d6a3a-00652ed5dd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.559 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:43:41.560 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:43:41.560 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.561 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:43:41.561 DEBUG:boto:Data: 2023-10-17T18:43:41.561 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.562 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.562 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.563 DEBUG:boto:Params: {} 2023-10-17T18:43:41.563 DEBUG:boto:Token: None 2023-10-17T18:43:41.564 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/ 2023-10-17T18:43:41.564 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs= 2023-10-17T18:43:41.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs=', 'Content-Length': '0'} 2023-10-17T18:43:41.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5286138476f6ef7-00652ed5dd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.568 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:43:41.568 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:43:41.569 DEBUG:boto:Method: PUT 2023-10-17T18:43:41.569 DEBUG:boto:Path: /hbmuog-53/c 2023-10-17T18:43:41.570 DEBUG:boto:Data: 2023-10-17T18:43:41.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:43:41.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.571 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.571 DEBUG:boto:Params: {} 2023-10-17T18:43:41.572 DEBUG:boto:Token: None 2023-10-17T18:43:41.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/c 2023-10-17T18:43:41.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:J/Sf1S67JkOa1xnIepu2sNPAuWw= 2023-10-17T18:43:41.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:J/Sf1S67JkOa1xnIepu2sNPAuWw='} 2023-10-17T18:43:41.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8e248e31cb7f509-00652ed5dd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.579 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:43:41.580 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:43:41.580 DEBUG:boto:Method: HEAD 2023-10-17T18:43:41.581 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:43:41.581 DEBUG:boto:Data: 2023-10-17T18:43:41.582 DEBUG:boto:Headers: {} 2023-10-17T18:43:41.583 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.583 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.583 DEBUG:boto:Params: {} 2023-10-17T18:43:41.584 DEBUG:boto:Token: None 2023-10-17T18:43:41.584 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/ 2023-10-17T18:43:41.585 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs= 2023-10-17T18:43:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nS05ams3B5/w6UrPaedicQgwSLs=', 'Content-Length': '0'} 2023-10-17T18:43:41.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1e6e4fe93f03754-00652ed5dd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.588 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:43:41.588 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:43:41.589 DEBUG:boto:Method: PUT 2023-10-17T18:43:41.589 DEBUG:boto:Path: /hbmuog-53/d 2023-10-17T18:43:41.589 DEBUG:boto:Data: 2023-10-17T18:43:41.590 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:43:41.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:43:41.591 DEBUG:boto:Port: 8000 2023-10-17T18:43:41.591 DEBUG:boto:Params: {} 2023-10-17T18:43:41.591 DEBUG:boto:Token: None 2023-10-17T18:43:41.592 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:43:41 GMT /hbmuog-53/d 2023-10-17T18:43:41.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zle8EAePHwr2TkQ/Sj1VKQHYME8= 2023-10-17T18:43:41.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:43:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zle8EAePHwr2TkQ/Sj1VKQHYME8='} 2023-10-17T18:43:41.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c9e14c08a50cfbc-00652ed5dd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:43:41.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:41.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:41.636 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:41.689 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:41.690 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:41.721 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:43:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:43:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568221.472769_623.1", 2023-10-17T18:43:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:43:41.472769Z" 2023-10-17T18:43:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:43:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:43:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:43:41.739 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568221.472769_623.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:41.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:43:41.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:41.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:43:41.740 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:43:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:41.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:41.821+0000 7f67416fd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:41.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:41.821+0000 7f67416fd780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:43:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:43:41.834 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:43:41.834 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:43:41.835 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:43:41.835 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:43:41.835 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:43:41.835 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:41.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.836 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:43:41.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.836 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:43:41.836 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:41.836 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:41.837 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568221.472769_623.1", 2023-10-17T18:43:41.837 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:41.837 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:43:41.837 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:41.838 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.472769Z", 2023-10-17T18:43:41.838 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:41.838 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.839 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:43:41.839 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:41.839 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:41.839 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:43:41.839 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:41.840 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:41.840 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:41.840 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:43:41.840 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:41.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.841 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:43:41.841 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:41.841 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:41.842 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:41.842 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:41.842 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:41.842 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:41.843 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:41.843 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:41.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:43:41.844 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:43:41.844 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:43:41.844 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:43:41.844 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:43:41.844 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:43:41.845 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:43:41.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:43:41.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.846 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:43:41.846 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:43:41.847 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:43:41.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:43:41.847 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:43:41Z" 2023-10-17T18:43:41.847 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:43:41.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568221.472769_623.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:43:41.472769Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:43:41Z" } 2023-10-17T18:43:41.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568221.472769_623.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:43:41.472769Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:43:41.849 DEBUG:rgw_multi.tests:log_status={0: '1_1697568221.472769_623.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:41.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568221.472769_623.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:43:41.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:43:41.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-53 2023-10-17T18:43:41.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:41.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:43:41.851 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:43:41.909 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:43:41.909 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:43:41.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:41.944+0000 7f7bfc34e780 1 waiting to reach incremental sync.. 2023-10-17T18:43:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:46.943+0000 7f7bfc34e780 1 waiting to reach incremental sync.. 2023-10-17T18:43:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:51.944+0000 7f7bfc34e780 1 waiting to reach incremental sync.. 2023-10-17T18:43:56.946 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:43:56.945+0000 7f7bfc34e780 1 waiting to reach incremental sync.. 2023-10-17T18:44:01.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:01.946+0000 7f7bfc34e780 1 bucket sync caught up with source: 2023-10-17T18:44:01.948 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.20.6, , , , 00000000001.27.6, , , 00000000001.52.6, , 00000000001.18.6] 2023-10-17T18:44:01.948 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.20.6, , , , 00000000001.27.6, , , 00000000001.52.6, , 00000000001.18.6] 2023-10-17T18:44:01.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:01.946+0000 7f7bfc34e780 0 bucket checkpoint complete 2023-10-17T18:44:01.962 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-53 2023-10-17T18:44:01.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:01.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:01.964 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:44:02.024 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:02.025 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:02.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:02.060+0000 7fa3dca3d780 1 waiting to reach incremental sync.. 2023-10-17T18:44:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:07.061+0000 7fa3dca3d780 1 waiting to reach incremental sync.. 2023-10-17T18:44:12.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:12.063+0000 7fa3dca3d780 1 bucket sync caught up with source: 2023-10-17T18:44:12.064 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.60.6, , , , 00000000001.239.6, , , 00000000001.149.6, , 00000000001.191.6] 2023-10-17T18:44:12.064 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.60.6, , , , 00000000001.239.6, , , 00000000001.149.6, , 00000000001.191.6] 2023-10-17T18:44:12.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:12.063+0000 7fa3dca3d780 0 bucket checkpoint complete 2023-10-17T18:44:12.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-53 zones={a2, a1} 2023-10-17T18:44:12.081 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.081 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.081 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.082 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:44:12.082 DEBUG:boto:Data: 2023-10-17T18:44:12.083 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.083 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.083 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.084 DEBUG:boto:Params: {} 2023-10-17T18:44:12.084 DEBUG:boto:Token: None 2023-10-17T18:44:12.085 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dhIBsHl2jManWWLVWXlv4Rlt7HM= 2023-10-17T18:44:12.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dhIBsHl2jManWWLVWXlv4Rlt7HM=', 'Content-Length': '0'} 2023-10-17T18:44:12.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006777699d039190c1-00652ed5fc-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.089 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.089 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.090 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.090 DEBUG:boto:Path: /hbmuog-53/ 2023-10-17T18:44:12.091 DEBUG:boto:Data: 2023-10-17T18:44:12.091 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.092 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.092 DEBUG:boto:Params: {} 2023-10-17T18:44:12.093 DEBUG:boto:Token: None 2023-10-17T18:44:12.093 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dhIBsHl2jManWWLVWXlv4Rlt7HM= 2023-10-17T18:44:12.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dhIBsHl2jManWWLVWXlv4Rlt7HM=', 'Content-Length': '0'} 2023-10-17T18:44:12.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000438bfb64a81c646e-00652ed5fc-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.097 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:44:12.098 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.098 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.099 DEBUG:boto:path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.099 DEBUG:boto:auth_path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.099 DEBUG:boto:Method: GET 2023-10-17T18:44:12.100 DEBUG:boto:Path: /hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.100 DEBUG:boto:Data: 2023-10-17T18:44:12.100 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.101 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.101 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.102 DEBUG:boto:Params: {} 2023-10-17T18:44:12.102 DEBUG:boto:Token: None 2023-10-17T18:44:12.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/?versions 2023-10-17T18:44:12.103 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo= 2023-10-17T18:44:12.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo=', 'Content-Length': '0'} 2023-10-17T18:44:12.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009712fd62632a131-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.106 DEBUG:boto:b'hbmuog-53999falseanulltrue2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.107 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:44:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:44:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:44:12.109 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:44:12.109 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:44:12.109 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.110 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.110 DEBUG:boto:path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.111 DEBUG:boto:auth_path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.111 DEBUG:boto:Method: GET 2023-10-17T18:44:12.111 DEBUG:boto:Path: /hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.112 DEBUG:boto:Data: 2023-10-17T18:44:12.112 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.113 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.113 DEBUG:boto:Params: {} 2023-10-17T18:44:12.114 DEBUG:boto:Token: None 2023-10-17T18:44:12.114 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/?versions 2023-10-17T18:44:12.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo= 2023-10-17T18:44:12.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo=', 'Content-Length': '0'} 2023-10-17T18:44:12.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027dc8cb1e1c06da3-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.119 DEBUG:boto:b'hbmuog-53999falseanulltrue2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.119 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:44:12.120 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:44:12.120 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:44:12.120 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:44:12.121 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.121 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.122 DEBUG:boto:path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.122 DEBUG:boto:auth_path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.122 DEBUG:boto:Method: GET 2023-10-17T18:44:12.123 DEBUG:boto:Path: /hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.123 DEBUG:boto:Data: 2023-10-17T18:44:12.124 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.124 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.124 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.125 DEBUG:boto:Params: {} 2023-10-17T18:44:12.125 DEBUG:boto:Token: None 2023-10-17T18:44:12.126 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/?versions 2023-10-17T18:44:12.126 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo= 2023-10-17T18:44:12.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo=', 'Content-Length': '0'} 2023-10-17T18:44:12.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5ddc9ba70802fa-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.130 DEBUG:boto:b'hbmuog-53999falseanulltrue2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.130 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.131 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.131 DEBUG:boto:path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.132 DEBUG:boto:auth_path=/hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.132 DEBUG:boto:Method: GET 2023-10-17T18:44:12.132 DEBUG:boto:Path: /hbmuog-53/?versions&max-keys=999 2023-10-17T18:44:12.133 DEBUG:boto:Data: 2023-10-17T18:44:12.133 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.134 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.134 DEBUG:boto:Params: {} 2023-10-17T18:44:12.135 DEBUG:boto:Token: None 2023-10-17T18:44:12.135 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/?versions 2023-10-17T18:44:12.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo= 2023-10-17T18:44:12.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tu66IAoy/FVZbmlK1TwpBBLSnEo=', 'Content-Length': '0'} 2023-10-17T18:44:12.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c1bb6f32ea3967f-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.139 DEBUG:boto:b'hbmuog-53999falseanulltrue2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:12.140 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.141 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.141 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.141 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.142 DEBUG:boto:Method: GET 2023-10-17T18:44:12.142 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.143 DEBUG:boto:Data: 2023-10-17T18:44:12.143 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.144 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.144 DEBUG:boto:Params: {} 2023-10-17T18:44:12.145 DEBUG:boto:Token: None 2023-10-17T18:44:12.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk= 2023-10-17T18:44:12.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk=', 'Content-Length': '0'} 2023-10-17T18:44:12.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006423bce53bff7827-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.149 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.149 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.149 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.150 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.150 DEBUG:boto:Method: GET 2023-10-17T18:44:12.151 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.151 DEBUG:boto:Data: 2023-10-17T18:44:12.151 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.152 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.153 DEBUG:boto:Params: {} 2023-10-17T18:44:12.153 DEBUG:boto:Token: None 2023-10-17T18:44:12.154 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk= 2023-10-17T18:44:12.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk=', 'Content-Length': '0'} 2023-10-17T18:44:12.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f67cd385205efc1-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.157 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.158 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.158 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.158 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.159 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.159 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.160 DEBUG:boto:Data: 2023-10-17T18:44:12.160 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.161 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.161 DEBUG:boto:Params: {} 2023-10-17T18:44:12.162 DEBUG:boto:Token: None 2023-10-17T18:44:12.162 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.162 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vbSspda4xMoO8zoGbXLCWGtUoSM= 2023-10-17T18:44:12.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vbSspda4xMoO8zoGbXLCWGtUoSM=', 'Content-Length': '0'} 2023-10-17T18:44:12.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977d46156515a9a5-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.165 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.166 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.166 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.166 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.167 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.167 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.168 DEBUG:boto:Data: 2023-10-17T18:44:12.168 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.168 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.169 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.169 DEBUG:boto:Params: {} 2023-10-17T18:44:12.170 DEBUG:boto:Token: None 2023-10-17T18:44:12.170 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vbSspda4xMoO8zoGbXLCWGtUoSM= 2023-10-17T18:44:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vbSspda4xMoO8zoGbXLCWGtUoSM=', 'Content-Length': '0'} 2023-10-17T18:44:12.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041f22f347e2accea-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:12.174 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.174 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.174 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.175 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.175 DEBUG:boto:Method: GET 2023-10-17T18:44:12.176 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.176 DEBUG:boto:Data: 2023-10-17T18:44:12.177 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.177 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.177 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.178 DEBUG:boto:Params: {} 2023-10-17T18:44:12.178 DEBUG:boto:Token: None 2023-10-17T18:44:12.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.179 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk= 2023-10-17T18:44:12.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk=', 'Content-Length': '0'} 2023-10-17T18:44:12.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba673074afb8baf-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.182 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.183 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.183 DEBUG:boto:path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.183 DEBUG:boto:auth_path=/hbmuog-53/a?versionId=null 2023-10-17T18:44:12.184 DEBUG:boto:Method: GET 2023-10-17T18:44:12.184 DEBUG:boto:Path: /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.185 DEBUG:boto:Data: 2023-10-17T18:44:12.185 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.186 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.186 DEBUG:boto:Params: {} 2023-10-17T18:44:12.187 DEBUG:boto:Token: None 2023-10-17T18:44:12.187 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a?versionId=null 2023-10-17T18:44:12.188 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk= 2023-10-17T18:44:12.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbl2F7PQRLq67QO3mXS0hcjTsk=', 'Content-Length': '0'} 2023-10-17T18:44:12.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde30e4833f45c7c-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.192 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.193 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.193 DEBUG:boto:path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.193 DEBUG:boto:auth_path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.194 DEBUG:boto:Method: GET 2023-10-17T18:44:12.194 DEBUG:boto:Path: /hbmuog-53/?prefix=a 2023-10-17T18:44:12.195 DEBUG:boto:Data: 2023-10-17T18:44:12.195 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.196 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.196 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.196 DEBUG:boto:Params: {} 2023-10-17T18:44:12.197 DEBUG:boto:Token: None 2023-10-17T18:44:12.197 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.198 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000737d5fd18a47a5c3-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.201 DEBUG:boto:b'hbmuog-53a1000falsea2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.202 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.202 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.203 DEBUG:boto:path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.203 DEBUG:boto:auth_path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.204 DEBUG:boto:Method: GET 2023-10-17T18:44:12.204 DEBUG:boto:Path: /hbmuog-53/?prefix=a 2023-10-17T18:44:12.204 DEBUG:boto:Data: 2023-10-17T18:44:12.205 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.205 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.206 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.206 DEBUG:boto:Params: {} 2023-10-17T18:44:12.207 DEBUG:boto:Token: None 2023-10-17T18:44:12.207 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.207 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1f33022c227a8d-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.254 DEBUG:boto:b'hbmuog-53a1000falsea2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.255 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.255 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.256 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.256 DEBUG:boto:Path: /hbmuog-53/a 2023-10-17T18:44:12.257 DEBUG:boto:Data: 2023-10-17T18:44:12.257 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.258 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.258 DEBUG:boto:Params: {} 2023-10-17T18:44:12.258 DEBUG:boto:Token: None 2023-10-17T18:44:12.259 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a 2023-10-17T18:44:12.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/CAXr+6B3IOMjUjJVs6XjF5aj8= 2023-10-17T18:44:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/CAXr+6B3IOMjUjJVs6XjF5aj8=', 'Content-Length': '0'} 2023-10-17T18:44:12.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce8d512205613e2f-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.262 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.263 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.263 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.264 DEBUG:boto:Path: /hbmuog-53/a 2023-10-17T18:44:12.264 DEBUG:boto:Data: 2023-10-17T18:44:12.264 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.265 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.266 DEBUG:boto:Params: {} 2023-10-17T18:44:12.266 DEBUG:boto:Token: None 2023-10-17T18:44:12.266 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a 2023-10-17T18:44:12.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/CAXr+6B3IOMjUjJVs6XjF5aj8= 2023-10-17T18:44:12.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/CAXr+6B3IOMjUjJVs6XjF5aj8=', 'Content-Length': '0'} 2023-10-17T18:44:12.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c155976eaf51609-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:12.270 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.271 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.271 DEBUG:boto:Method: GET 2023-10-17T18:44:12.272 DEBUG:boto:Path: /hbmuog-53/a 2023-10-17T18:44:12.272 DEBUG:boto:Data: 2023-10-17T18:44:12.272 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.273 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.274 DEBUG:boto:Params: {} 2023-10-17T18:44:12.274 DEBUG:boto:Token: None 2023-10-17T18:44:12.275 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a 2023-10-17T18:44:12.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:03VXhWC93Y5ebUhA91q6RhyMXQ8= 2023-10-17T18:44:12.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:03VXhWC93Y5ebUhA91q6RhyMXQ8=', 'Content-Length': '0'} 2023-10-17T18:44:12.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde3b11e877731fe-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.278 DEBUG:boto:path=/hbmuog-53/a 2023-10-17T18:44:12.278 DEBUG:boto:auth_path=/hbmuog-53/a 2023-10-17T18:44:12.279 DEBUG:boto:Method: GET 2023-10-17T18:44:12.279 DEBUG:boto:Path: /hbmuog-53/a 2023-10-17T18:44:12.280 DEBUG:boto:Data: 2023-10-17T18:44:12.280 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.281 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.281 DEBUG:boto:Params: {} 2023-10-17T18:44:12.282 DEBUG:boto:Token: None 2023-10-17T18:44:12.282 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/a 2023-10-17T18:44:12.283 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:03VXhWC93Y5ebUhA91q6RhyMXQ8= 2023-10-17T18:44:12.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:03VXhWC93Y5ebUhA91q6RhyMXQ8=', 'Content-Length': '0'} 2023-10-17T18:44:12.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750c1b91079ceb37-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.286 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.286 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.287 DEBUG:boto:path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.287 DEBUG:boto:auth_path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.288 DEBUG:boto:Method: GET 2023-10-17T18:44:12.288 DEBUG:boto:Path: /hbmuog-53/?prefix=a 2023-10-17T18:44:12.288 DEBUG:boto:Data: 2023-10-17T18:44:12.289 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.290 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.290 DEBUG:boto:Params: {} 2023-10-17T18:44:12.290 DEBUG:boto:Token: None 2023-10-17T18:44:12.291 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b300668cf43c8f-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.295 DEBUG:boto:b'hbmuog-53a1000falsea2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.296 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.296 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.296 DEBUG:boto:path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.297 DEBUG:boto:auth_path=/hbmuog-53/?prefix=a 2023-10-17T18:44:12.297 DEBUG:boto:Method: GET 2023-10-17T18:44:12.298 DEBUG:boto:Path: /hbmuog-53/?prefix=a 2023-10-17T18:44:12.298 DEBUG:boto:Data: 2023-10-17T18:44:12.298 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.299 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.299 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.300 DEBUG:boto:Params: {} 2023-10-17T18:44:12.300 DEBUG:boto:Token: None 2023-10-17T18:44:12.300 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.301 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008749d37e6292f2bb-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.304 DEBUG:boto:b'hbmuog-53a1000falsea2023-10-17T18:43:41.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:12.305 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.306 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.306 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.307 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.307 DEBUG:boto:Method: GET 2023-10-17T18:44:12.308 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.308 DEBUG:boto:Data: 2023-10-17T18:44:12.308 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.309 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.310 DEBUG:boto:Params: {} 2023-10-17T18:44:12.310 DEBUG:boto:Token: None 2023-10-17T18:44:12.311 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA= 2023-10-17T18:44:12.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA=', 'Content-Length': '0'} 2023-10-17T18:44:12.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef99f6130b7581b-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.314 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.314 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.315 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.315 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.316 DEBUG:boto:Method: GET 2023-10-17T18:44:12.316 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.316 DEBUG:boto:Data: 2023-10-17T18:44:12.317 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.318 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.318 DEBUG:boto:Params: {} 2023-10-17T18:44:12.319 DEBUG:boto:Token: None 2023-10-17T18:44:12.319 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA= 2023-10-17T18:44:12.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA=', 'Content-Length': '0'} 2023-10-17T18:44:12.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043893f84412e3406-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.322 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.323 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.323 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.324 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.324 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.324 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.325 DEBUG:boto:Data: 2023-10-17T18:44:12.325 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.326 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.326 DEBUG:boto:Params: {} 2023-10-17T18:44:12.327 DEBUG:boto:Token: None 2023-10-17T18:44:12.327 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nHuBSxMN4MIfY3zeBWT5/RKAeMA= 2023-10-17T18:44:12.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nHuBSxMN4MIfY3zeBWT5/RKAeMA=', 'Content-Length': '0'} 2023-10-17T18:44:12.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22add266f926b2a-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.331 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.331 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.331 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.332 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.332 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.333 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.333 DEBUG:boto:Data: 2023-10-17T18:44:12.333 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.334 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.334 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.335 DEBUG:boto:Params: {} 2023-10-17T18:44:12.335 DEBUG:boto:Token: None 2023-10-17T18:44:12.336 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.336 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nHuBSxMN4MIfY3zeBWT5/RKAeMA= 2023-10-17T18:44:12.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nHuBSxMN4MIfY3zeBWT5/RKAeMA=', 'Content-Length': '0'} 2023-10-17T18:44:12.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006097d3399cfb2671-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:12.340 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.340 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.340 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.341 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.341 DEBUG:boto:Method: GET 2023-10-17T18:44:12.342 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.342 DEBUG:boto:Data: 2023-10-17T18:44:12.343 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.343 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.344 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.344 DEBUG:boto:Params: {} 2023-10-17T18:44:12.344 DEBUG:boto:Token: None 2023-10-17T18:44:12.345 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.345 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA= 2023-10-17T18:44:12.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA=', 'Content-Length': '0'} 2023-10-17T18:44:12.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8067a6082b4936-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.348 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.349 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.349 DEBUG:boto:path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.350 DEBUG:boto:auth_path=/hbmuog-53/b?versionId=null 2023-10-17T18:44:12.350 DEBUG:boto:Method: GET 2023-10-17T18:44:12.351 DEBUG:boto:Path: /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.351 DEBUG:boto:Data: 2023-10-17T18:44:12.351 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.352 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.353 DEBUG:boto:Params: {} 2023-10-17T18:44:12.353 DEBUG:boto:Token: None 2023-10-17T18:44:12.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b?versionId=null 2023-10-17T18:44:12.354 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA= 2023-10-17T18:44:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XJtZZ29Svlhls/QM9lQFvlzbeA=', 'Content-Length': '0'} 2023-10-17T18:44:12.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8473078b265bec8-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.398 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.399 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.399 DEBUG:boto:path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.400 DEBUG:boto:auth_path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.400 DEBUG:boto:Method: GET 2023-10-17T18:44:12.400 DEBUG:boto:Path: /hbmuog-53/?prefix=b 2023-10-17T18:44:12.401 DEBUG:boto:Data: 2023-10-17T18:44:12.401 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.402 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.402 DEBUG:boto:Params: {} 2023-10-17T18:44:12.403 DEBUG:boto:Token: None 2023-10-17T18:44:12.403 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ab9de95d6ad009-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.407 DEBUG:boto:b'hbmuog-53b1000falseb2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.408 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.409 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.409 DEBUG:boto:path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.410 DEBUG:boto:auth_path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.410 DEBUG:boto:Method: GET 2023-10-17T18:44:12.410 DEBUG:boto:Path: /hbmuog-53/?prefix=b 2023-10-17T18:44:12.411 DEBUG:boto:Data: 2023-10-17T18:44:12.411 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.412 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.412 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.413 DEBUG:boto:Params: {} 2023-10-17T18:44:12.413 DEBUG:boto:Token: None 2023-10-17T18:44:12.414 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.414 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824ffc2b992b7848-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.417 DEBUG:boto:b'hbmuog-53b1000falseb2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.418 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.419 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.419 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.419 DEBUG:boto:Path: /hbmuog-53/b 2023-10-17T18:44:12.420 DEBUG:boto:Data: 2023-10-17T18:44:12.420 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.421 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.421 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.421 DEBUG:boto:Params: {} 2023-10-17T18:44:12.422 DEBUG:boto:Token: None 2023-10-17T18:44:12.422 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b 2023-10-17T18:44:12.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SQ4jwKQR1DVXPzLLD1cSnTFP/R0= 2023-10-17T18:44:12.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SQ4jwKQR1DVXPzLLD1cSnTFP/R0=', 'Content-Length': '0'} 2023-10-17T18:44:12.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7f3800747249da-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.425 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.426 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.426 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.427 DEBUG:boto:Path: /hbmuog-53/b 2023-10-17T18:44:12.427 DEBUG:boto:Data: 2023-10-17T18:44:12.427 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.428 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.429 DEBUG:boto:Params: {} 2023-10-17T18:44:12.429 DEBUG:boto:Token: None 2023-10-17T18:44:12.429 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b 2023-10-17T18:44:12.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SQ4jwKQR1DVXPzLLD1cSnTFP/R0= 2023-10-17T18:44:12.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SQ4jwKQR1DVXPzLLD1cSnTFP/R0=', 'Content-Length': '0'} 2023-10-17T18:44:12.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038778f4080de275c-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:12.433 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.434 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.434 DEBUG:boto:Method: GET 2023-10-17T18:44:12.435 DEBUG:boto:Path: /hbmuog-53/b 2023-10-17T18:44:12.435 DEBUG:boto:Data: 2023-10-17T18:44:12.435 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.439 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.439 DEBUG:boto:Params: {} 2023-10-17T18:44:12.440 DEBUG:boto:Token: None 2023-10-17T18:44:12.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b 2023-10-17T18:44:12.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5LV4Ex0vAFaHILFzkbvhWXO3ybU= 2023-10-17T18:44:12.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5LV4Ex0vAFaHILFzkbvhWXO3ybU=', 'Content-Length': '0'} 2023-10-17T18:44:12.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e254f64ae69b0b5-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.444 DEBUG:boto:path=/hbmuog-53/b 2023-10-17T18:44:12.444 DEBUG:boto:auth_path=/hbmuog-53/b 2023-10-17T18:44:12.445 DEBUG:boto:Method: GET 2023-10-17T18:44:12.445 DEBUG:boto:Path: /hbmuog-53/b 2023-10-17T18:44:12.445 DEBUG:boto:Data: 2023-10-17T18:44:12.446 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.447 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.447 DEBUG:boto:Params: {} 2023-10-17T18:44:12.448 DEBUG:boto:Token: None 2023-10-17T18:44:12.448 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/b 2023-10-17T18:44:12.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5LV4Ex0vAFaHILFzkbvhWXO3ybU= 2023-10-17T18:44:12.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5LV4Ex0vAFaHILFzkbvhWXO3ybU=', 'Content-Length': '0'} 2023-10-17T18:44:12.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f533cadc639bf1-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.452 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.452 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.453 DEBUG:boto:path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.453 DEBUG:boto:auth_path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.454 DEBUG:boto:Method: GET 2023-10-17T18:44:12.454 DEBUG:boto:Path: /hbmuog-53/?prefix=b 2023-10-17T18:44:12.454 DEBUG:boto:Data: 2023-10-17T18:44:12.455 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.456 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.456 DEBUG:boto:Params: {} 2023-10-17T18:44:12.456 DEBUG:boto:Token: None 2023-10-17T18:44:12.457 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097f00535be0e8b00-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.502 DEBUG:boto:b'hbmuog-53b1000falseb2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.503 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.503 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.504 DEBUG:boto:path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.504 DEBUG:boto:auth_path=/hbmuog-53/?prefix=b 2023-10-17T18:44:12.505 DEBUG:boto:Method: GET 2023-10-17T18:44:12.505 DEBUG:boto:Path: /hbmuog-53/?prefix=b 2023-10-17T18:44:12.506 DEBUG:boto:Data: 2023-10-17T18:44:12.506 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.507 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.507 DEBUG:boto:Params: {} 2023-10-17T18:44:12.508 DEBUG:boto:Token: None 2023-10-17T18:44:12.508 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec50037940ec757-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.512 DEBUG:boto:b'hbmuog-53b1000falseb2023-10-17T18:43:41.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:12.513 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.514 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.514 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.515 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.515 DEBUG:boto:Method: GET 2023-10-17T18:44:12.516 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.516 DEBUG:boto:Data: 2023-10-17T18:44:12.516 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.517 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.517 DEBUG:boto:Params: {} 2023-10-17T18:44:12.518 DEBUG:boto:Token: None 2023-10-17T18:44:12.518 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.519 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8= 2023-10-17T18:44:12.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8=', 'Content-Length': '0'} 2023-10-17T18:44:12.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096eaf4735b042a3b-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.522 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.523 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.523 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.524 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.524 DEBUG:boto:Method: GET 2023-10-17T18:44:12.524 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.525 DEBUG:boto:Data: 2023-10-17T18:44:12.525 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.526 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.526 DEBUG:boto:Params: {} 2023-10-17T18:44:12.527 DEBUG:boto:Token: None 2023-10-17T18:44:12.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8= 2023-10-17T18:44:12.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8=', 'Content-Length': '0'} 2023-10-17T18:44:12.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbf81ea84d8f895-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.531 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.531 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.531 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.532 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.532 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.532 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.533 DEBUG:boto:Data: 2023-10-17T18:44:12.533 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.533 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.534 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.534 DEBUG:boto:Params: {} 2023-10-17T18:44:12.535 DEBUG:boto:Token: None 2023-10-17T18:44:12.535 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.535 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lxYWrjBiciwVP3J8gJRhhSLQoAE= 2023-10-17T18:44:12.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lxYWrjBiciwVP3J8gJRhhSLQoAE=', 'Content-Length': '0'} 2023-10-17T18:44:12.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084eaca3d915e4228-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.538 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.539 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.539 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.539 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.540 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.540 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.541 DEBUG:boto:Data: 2023-10-17T18:44:12.541 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.542 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.542 DEBUG:boto:Params: {} 2023-10-17T18:44:12.542 DEBUG:boto:Token: None 2023-10-17T18:44:12.543 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.543 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lxYWrjBiciwVP3J8gJRhhSLQoAE= 2023-10-17T18:44:12.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lxYWrjBiciwVP3J8gJRhhSLQoAE=', 'Content-Length': '0'} 2023-10-17T18:44:12.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb9ad31054ddcfb-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:12.546 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.547 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.547 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.548 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.548 DEBUG:boto:Method: GET 2023-10-17T18:44:12.549 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.549 DEBUG:boto:Data: 2023-10-17T18:44:12.549 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.550 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.550 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.551 DEBUG:boto:Params: {} 2023-10-17T18:44:12.551 DEBUG:boto:Token: None 2023-10-17T18:44:12.552 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8= 2023-10-17T18:44:12.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8=', 'Content-Length': '0'} 2023-10-17T18:44:12.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa09f64040201b5-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.555 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.556 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.556 DEBUG:boto:path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.556 DEBUG:boto:auth_path=/hbmuog-53/c?versionId=null 2023-10-17T18:44:12.557 DEBUG:boto:Method: GET 2023-10-17T18:44:12.557 DEBUG:boto:Path: /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.557 DEBUG:boto:Data: 2023-10-17T18:44:12.558 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.559 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.559 DEBUG:boto:Params: {} 2023-10-17T18:44:12.560 DEBUG:boto:Token: None 2023-10-17T18:44:12.560 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c?versionId=null 2023-10-17T18:44:12.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8= 2023-10-17T18:44:12.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qWkUIJ0ADvjQB5ipHuNIAmGIIe8=', 'Content-Length': '0'} 2023-10-17T18:44:12.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dbbc56aae4d3214-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.563 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.564 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.564 DEBUG:boto:path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.565 DEBUG:boto:auth_path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.565 DEBUG:boto:Method: GET 2023-10-17T18:44:12.565 DEBUG:boto:Path: /hbmuog-53/?prefix=c 2023-10-17T18:44:12.566 DEBUG:boto:Data: 2023-10-17T18:44:12.566 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.567 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.568 DEBUG:boto:Params: {} 2023-10-17T18:44:12.568 DEBUG:boto:Token: None 2023-10-17T18:44:12.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d35d1005bb7d50-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.573 DEBUG:boto:b'hbmuog-53c1000falsec2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.573 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.574 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.574 DEBUG:boto:path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.574 DEBUG:boto:auth_path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.575 DEBUG:boto:Method: GET 2023-10-17T18:44:12.575 DEBUG:boto:Path: /hbmuog-53/?prefix=c 2023-10-17T18:44:12.576 DEBUG:boto:Data: 2023-10-17T18:44:12.576 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.577 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.577 DEBUG:boto:Params: {} 2023-10-17T18:44:12.578 DEBUG:boto:Token: None 2023-10-17T18:44:12.578 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de8931fd0840a088-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.622 DEBUG:boto:b'hbmuog-53c1000falsec2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.623 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.623 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.624 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.624 DEBUG:boto:Path: /hbmuog-53/c 2023-10-17T18:44:12.624 DEBUG:boto:Data: 2023-10-17T18:44:12.625 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.625 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.626 DEBUG:boto:Params: {} 2023-10-17T18:44:12.626 DEBUG:boto:Token: None 2023-10-17T18:44:12.627 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c 2023-10-17T18:44:12.627 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aAUNxjr7UXIHFCugvKjG5IJzYBM= 2023-10-17T18:44:12.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aAUNxjr7UXIHFCugvKjG5IJzYBM=', 'Content-Length': '0'} 2023-10-17T18:44:12.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b0d9086c307f25-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.630 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.630 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.631 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.631 DEBUG:boto:Path: /hbmuog-53/c 2023-10-17T18:44:12.632 DEBUG:boto:Data: 2023-10-17T18:44:12.632 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.633 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.633 DEBUG:boto:Params: {} 2023-10-17T18:44:12.633 DEBUG:boto:Token: None 2023-10-17T18:44:12.634 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c 2023-10-17T18:44:12.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aAUNxjr7UXIHFCugvKjG5IJzYBM= 2023-10-17T18:44:12.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aAUNxjr7UXIHFCugvKjG5IJzYBM=', 'Content-Length': '0'} 2023-10-17T18:44:12.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476c25abfc8b3793-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:12.638 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.638 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.638 DEBUG:boto:Method: GET 2023-10-17T18:44:12.639 DEBUG:boto:Path: /hbmuog-53/c 2023-10-17T18:44:12.639 DEBUG:boto:Data: 2023-10-17T18:44:12.640 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.640 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.641 DEBUG:boto:Params: {} 2023-10-17T18:44:12.641 DEBUG:boto:Token: None 2023-10-17T18:44:12.642 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c 2023-10-17T18:44:12.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bgj9NPBK5tHnAwpSgNX1RFLBf2k= 2023-10-17T18:44:12.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bgj9NPBK5tHnAwpSgNX1RFLBf2k=', 'Content-Length': '0'} 2023-10-17T18:44:12.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ea52c1f61847d3-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.645 DEBUG:boto:path=/hbmuog-53/c 2023-10-17T18:44:12.646 DEBUG:boto:auth_path=/hbmuog-53/c 2023-10-17T18:44:12.646 DEBUG:boto:Method: GET 2023-10-17T18:44:12.646 DEBUG:boto:Path: /hbmuog-53/c 2023-10-17T18:44:12.647 DEBUG:boto:Data: 2023-10-17T18:44:12.647 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.648 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.648 DEBUG:boto:Params: {} 2023-10-17T18:44:12.649 DEBUG:boto:Token: None 2023-10-17T18:44:12.649 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/c 2023-10-17T18:44:12.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bgj9NPBK5tHnAwpSgNX1RFLBf2k= 2023-10-17T18:44:12.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bgj9NPBK5tHnAwpSgNX1RFLBf2k=', 'Content-Length': '0'} 2023-10-17T18:44:12.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abef7870bddd1f91-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.653 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.653 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.653 DEBUG:boto:path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.654 DEBUG:boto:auth_path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.654 DEBUG:boto:Method: GET 2023-10-17T18:44:12.655 DEBUG:boto:Path: /hbmuog-53/?prefix=c 2023-10-17T18:44:12.655 DEBUG:boto:Data: 2023-10-17T18:44:12.655 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.656 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.656 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.657 DEBUG:boto:Params: {} 2023-10-17T18:44:12.657 DEBUG:boto:Token: None 2023-10-17T18:44:12.658 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000906c42c725798234-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.662 DEBUG:boto:b'hbmuog-53c1000falsec2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.662 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.663 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.663 DEBUG:boto:path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.664 DEBUG:boto:auth_path=/hbmuog-53/?prefix=c 2023-10-17T18:44:12.664 DEBUG:boto:Method: GET 2023-10-17T18:44:12.664 DEBUG:boto:Path: /hbmuog-53/?prefix=c 2023-10-17T18:44:12.665 DEBUG:boto:Data: 2023-10-17T18:44:12.665 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.666 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.666 DEBUG:boto:Params: {} 2023-10-17T18:44:12.670 DEBUG:boto:Token: None 2023-10-17T18:44:12.670 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c066ed00490ad7d7-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.674 DEBUG:boto:b'hbmuog-53c1000falsec2023-10-17T18:43:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:12.675 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.676 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.676 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.677 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.677 DEBUG:boto:Method: GET 2023-10-17T18:44:12.678 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.678 DEBUG:boto:Data: 2023-10-17T18:44:12.678 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.679 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.680 DEBUG:boto:Params: {} 2023-10-17T18:44:12.680 DEBUG:boto:Token: None 2023-10-17T18:44:12.681 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.681 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI= 2023-10-17T18:44:12.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI=', 'Content-Length': '0'} 2023-10-17T18:44:12.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cd934495c9ec78-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.684 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.685 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.685 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.685 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.686 DEBUG:boto:Method: GET 2023-10-17T18:44:12.686 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.687 DEBUG:boto:Data: 2023-10-17T18:44:12.687 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.688 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.688 DEBUG:boto:Params: {} 2023-10-17T18:44:12.689 DEBUG:boto:Token: None 2023-10-17T18:44:12.689 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI= 2023-10-17T18:44:12.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI=', 'Content-Length': '0'} 2023-10-17T18:44:12.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099a4c055380dc25b-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.693 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.693 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.693 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.694 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.694 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.695 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.695 DEBUG:boto:Data: 2023-10-17T18:44:12.695 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.696 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.697 DEBUG:boto:Params: {} 2023-10-17T18:44:12.697 DEBUG:boto:Token: None 2023-10-17T18:44:12.697 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.698 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HU38yM0L4XuQ9aQsmIzKZCS7cI4= 2023-10-17T18:44:12.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HU38yM0L4XuQ9aQsmIzKZCS7cI4=', 'Content-Length': '0'} 2023-10-17T18:44:12.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa459b6df6d5540-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.701 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.701 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.702 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.702 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.702 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.703 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.703 DEBUG:boto:Data: 2023-10-17T18:44:12.704 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.704 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.704 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.705 DEBUG:boto:Params: {} 2023-10-17T18:44:12.705 DEBUG:boto:Token: None 2023-10-17T18:44:12.706 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.706 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HU38yM0L4XuQ9aQsmIzKZCS7cI4= 2023-10-17T18:44:12.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HU38yM0L4XuQ9aQsmIzKZCS7cI4=', 'Content-Length': '0'} 2023-10-17T18:44:12.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009dedc3f652074a1-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:12.710 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.710 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.711 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.711 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.711 DEBUG:boto:Method: GET 2023-10-17T18:44:12.712 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.712 DEBUG:boto:Data: 2023-10-17T18:44:12.712 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.713 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.713 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.714 DEBUG:boto:Params: {} 2023-10-17T18:44:12.714 DEBUG:boto:Token: None 2023-10-17T18:44:12.715 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI= 2023-10-17T18:44:12.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI=', 'Content-Length': '0'} 2023-10-17T18:44:12.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007692c961e37d1e32-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.718 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.719 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.719 DEBUG:boto:path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.719 DEBUG:boto:auth_path=/hbmuog-53/d?versionId=null 2023-10-17T18:44:12.720 DEBUG:boto:Method: GET 2023-10-17T18:44:12.720 DEBUG:boto:Path: /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.721 DEBUG:boto:Data: 2023-10-17T18:44:12.721 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.722 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.722 DEBUG:boto:Params: {} 2023-10-17T18:44:12.723 DEBUG:boto:Token: None 2023-10-17T18:44:12.723 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d?versionId=null 2023-10-17T18:44:12.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI= 2023-10-17T18:44:12.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FS8vJqvETTpxw1mWvjislejENPI=', 'Content-Length': '0'} 2023-10-17T18:44:12.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebdf024b4cddf12-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.727 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.727 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.728 DEBUG:boto:path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.728 DEBUG:boto:auth_path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.729 DEBUG:boto:Method: GET 2023-10-17T18:44:12.729 DEBUG:boto:Path: /hbmuog-53/?prefix=d 2023-10-17T18:44:12.729 DEBUG:boto:Data: 2023-10-17T18:44:12.730 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.731 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.731 DEBUG:boto:Params: {} 2023-10-17T18:44:12.732 DEBUG:boto:Token: None 2023-10-17T18:44:12.732 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a5752ec4cd4d86-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.736 DEBUG:boto:b'hbmuog-53d1000falsed2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.736 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.737 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.737 DEBUG:boto:path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.738 DEBUG:boto:auth_path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.738 DEBUG:boto:Method: GET 2023-10-17T18:44:12.739 DEBUG:boto:Path: /hbmuog-53/?prefix=d 2023-10-17T18:44:12.739 DEBUG:boto:Data: 2023-10-17T18:44:12.739 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.740 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.740 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.741 DEBUG:boto:Params: {} 2023-10-17T18:44:12.741 DEBUG:boto:Token: None 2023-10-17T18:44:12.741 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.742 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001997696460b34073-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.746 DEBUG:boto:b'hbmuog-53d1000falsed2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.746 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.747 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.747 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.747 DEBUG:boto:Path: /hbmuog-53/d 2023-10-17T18:44:12.748 DEBUG:boto:Data: 2023-10-17T18:44:12.748 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.749 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.749 DEBUG:boto:Params: {} 2023-10-17T18:44:12.750 DEBUG:boto:Token: None 2023-10-17T18:44:12.750 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d 2023-10-17T18:44:12.751 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyKuR0rn9vM6qt1G/tDFf4TQyLQ= 2023-10-17T18:44:12.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyKuR0rn9vM6qt1G/tDFf4TQyLQ=', 'Content-Length': '0'} 2023-10-17T18:44:12.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f07ba67ccd4c20-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.754 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.754 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.754 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.755 DEBUG:boto:Path: /hbmuog-53/d 2023-10-17T18:44:12.755 DEBUG:boto:Data: 2023-10-17T18:44:12.756 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.756 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.757 DEBUG:boto:Params: {} 2023-10-17T18:44:12.757 DEBUG:boto:Token: None 2023-10-17T18:44:12.757 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d 2023-10-17T18:44:12.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RyKuR0rn9vM6qt1G/tDFf4TQyLQ= 2023-10-17T18:44:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RyKuR0rn9vM6qt1G/tDFf4TQyLQ=', 'Content-Length': '0'} 2023-10-17T18:44:12.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8db266314008c6b-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:12.761 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.761 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.762 DEBUG:boto:Method: GET 2023-10-17T18:44:12.762 DEBUG:boto:Path: /hbmuog-53/d 2023-10-17T18:44:12.763 DEBUG:boto:Data: 2023-10-17T18:44:12.763 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.764 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.764 DEBUG:boto:Params: {} 2023-10-17T18:44:12.765 DEBUG:boto:Token: None 2023-10-17T18:44:12.765 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d 2023-10-17T18:44:12.766 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gIeUzqWfOzXa7krPQ208RizdHbM= 2023-10-17T18:44:12.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gIeUzqWfOzXa7krPQ208RizdHbM=', 'Content-Length': '0'} 2023-10-17T18:44:12.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020188b11d642cbc6-00652ed5fc-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.810 DEBUG:boto:path=/hbmuog-53/d 2023-10-17T18:44:12.811 DEBUG:boto:auth_path=/hbmuog-53/d 2023-10-17T18:44:12.811 DEBUG:boto:Method: GET 2023-10-17T18:44:12.812 DEBUG:boto:Path: /hbmuog-53/d 2023-10-17T18:44:12.812 DEBUG:boto:Data: 2023-10-17T18:44:12.812 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.813 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.813 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.814 DEBUG:boto:Params: {} 2023-10-17T18:44:12.814 DEBUG:boto:Token: None 2023-10-17T18:44:12.814 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/d 2023-10-17T18:44:12.815 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gIeUzqWfOzXa7krPQ208RizdHbM= 2023-10-17T18:44:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gIeUzqWfOzXa7krPQ208RizdHbM=', 'Content-Length': '0'} 2023-10-17T18:44:12.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:43:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bd7f78c1052fb5-00652ed5fc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.818 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.818 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.819 DEBUG:boto:path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.819 DEBUG:boto:auth_path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.820 DEBUG:boto:Method: GET 2023-10-17T18:44:12.820 DEBUG:boto:Path: /hbmuog-53/?prefix=d 2023-10-17T18:44:12.820 DEBUG:boto:Data: 2023-10-17T18:44:12.821 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:12.822 DEBUG:boto:Port: 8001 2023-10-17T18:44:12.822 DEBUG:boto:Params: {} 2023-10-17T18:44:12.822 DEBUG:boto:Token: None 2023-10-17T18:44:12.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.823 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab6a886f10b9223-00652ed5fc-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.827 DEBUG:boto:b'hbmuog-53d1000falsed2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.827 DEBUG:boto:path=/hbmuog-53/ 2023-10-17T18:44:12.828 DEBUG:boto:auth_path=/hbmuog-53/ 2023-10-17T18:44:12.828 DEBUG:boto:path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.829 DEBUG:boto:auth_path=/hbmuog-53/?prefix=d 2023-10-17T18:44:12.829 DEBUG:boto:Method: GET 2023-10-17T18:44:12.829 DEBUG:boto:Path: /hbmuog-53/?prefix=d 2023-10-17T18:44:12.830 DEBUG:boto:Data: 2023-10-17T18:44:12.830 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.831 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.832 DEBUG:boto:Params: {} 2023-10-17T18:44:12.832 DEBUG:boto:Token: None 2023-10-17T18:44:12.833 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-53/ 2023-10-17T18:44:12.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY= 2023-10-17T18:44:12.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zcfskJ0wHxufLJ2ZLZotzrUkjdY=', 'Content-Length': '0'} 2023-10-17T18:44:12.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94df8fa36f5974c-00652ed5fc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.837 DEBUG:boto:b'hbmuog-53d1000falsed2023-10-17T18:43:41.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:12.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-53 zones={a2, a1} 2023-10-17T18:44:12.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:12.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:12.839 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:44:12.893 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:12.894 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:12.943 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-54 2023-10-17T18:44:12.943 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:12.944 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:12.944 DEBUG:boto:Method: PUT 2023-10-17T18:44:12.944 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:12.945 DEBUG:boto:Data: 2023-10-17T18:44:12.945 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.945 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.946 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.946 DEBUG:boto:Params: {} 2023-10-17T18:44:12.947 DEBUG:boto:Token: None 2023-10-17T18:44:12.947 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-54/ 2023-10-17T18:44:12.947 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AXOwkSas2cOgmIm3A8fcsbIkPqg= 2023-10-17T18:44:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AXOwkSas2cOgmIm3A8fcsbIkPqg=', 'Content-Length': '0'} 2023-10-17T18:44:12.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002cb2654ab567182d-00652ed5fc-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.964 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:12.965 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:12.965 DEBUG:boto:Method: HEAD 2023-10-17T18:44:12.966 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:12.966 DEBUG:boto:Data: 2023-10-17T18:44:12.966 DEBUG:boto:Headers: {} 2023-10-17T18:44:12.967 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.967 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.967 DEBUG:boto:Params: {} 2023-10-17T18:44:12.968 DEBUG:boto:Token: None 2023-10-17T18:44:12.968 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-54/ 2023-10-17T18:44:12.969 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:utSPv2AyKCnKMIQEC3C3MBz4dlw= 2023-10-17T18:44:12.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:utSPv2AyKCnKMIQEC3C3MBz4dlw=', 'Content-Length': '0'} 2023-10-17T18:44:12.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ead767081c00a355-00652ed5fc-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:12.972 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:12.973 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:12.973 DEBUG:boto:Method: PUT 2023-10-17T18:44:12.973 DEBUG:boto:Path: /hbmuog-54/a 2023-10-17T18:44:12.974 DEBUG:boto:Data: 2023-10-17T18:44:12.974 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:12.975 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:12.975 DEBUG:boto:Port: 8000 2023-10-17T18:44:12.975 DEBUG:boto:Params: {} 2023-10-17T18:44:12.976 DEBUG:boto:Token: None 2023-10-17T18:44:12.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:12 GMT /hbmuog-54/a 2023-10-17T18:44:12.977 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fA/YwVoaI8fNm3aEOaUc1QfvysE= 2023-10-17T18:44:12.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fA/YwVoaI8fNm3aEOaUc1QfvysE='} 2023-10-17T18:44:13.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083487380060e3534-00652ed5fc-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.023 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:13.023 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:13.024 DEBUG:boto:Method: HEAD 2023-10-17T18:44:13.024 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:13.024 DEBUG:boto:Data: 2023-10-17T18:44:13.025 DEBUG:boto:Headers: {} 2023-10-17T18:44:13.025 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.025 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.026 DEBUG:boto:Params: {} 2023-10-17T18:44:13.026 DEBUG:boto:Token: None 2023-10-17T18:44:13.026 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/ 2023-10-17T18:44:13.027 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws= 2023-10-17T18:44:13.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws=', 'Content-Length': '0'} 2023-10-17T18:44:13.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea8d74f1afbf2508-00652ed5fd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.030 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:13.031 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:13.031 DEBUG:boto:Method: PUT 2023-10-17T18:44:13.031 DEBUG:boto:Path: /hbmuog-54/b 2023-10-17T18:44:13.032 DEBUG:boto:Data: 2023-10-17T18:44:13.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:13.032 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.033 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.033 DEBUG:boto:Params: {} 2023-10-17T18:44:13.033 DEBUG:boto:Token: None 2023-10-17T18:44:13.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/b 2023-10-17T18:44:13.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VlwuVMNRuXz4VgGLIBCHFT2Egtk= 2023-10-17T18:44:13.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VlwuVMNRuXz4VgGLIBCHFT2Egtk='} 2023-10-17T18:44:13.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5a3ebdda317f227-00652ed5fd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.041 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:13.041 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:13.042 DEBUG:boto:Method: HEAD 2023-10-17T18:44:13.042 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:13.042 DEBUG:boto:Data: 2023-10-17T18:44:13.043 DEBUG:boto:Headers: {} 2023-10-17T18:44:13.043 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.043 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.044 DEBUG:boto:Params: {} 2023-10-17T18:44:13.044 DEBUG:boto:Token: None 2023-10-17T18:44:13.045 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/ 2023-10-17T18:44:13.045 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws= 2023-10-17T18:44:13.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws=', 'Content-Length': '0'} 2023-10-17T18:44:13.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000143b9355d59def43-00652ed5fd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.048 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:13.049 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:13.049 DEBUG:boto:Method: PUT 2023-10-17T18:44:13.049 DEBUG:boto:Path: /hbmuog-54/c 2023-10-17T18:44:13.050 DEBUG:boto:Data: 2023-10-17T18:44:13.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:13.051 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.051 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.051 DEBUG:boto:Params: {} 2023-10-17T18:44:13.052 DEBUG:boto:Token: None 2023-10-17T18:44:13.052 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/c 2023-10-17T18:44:13.053 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XiYA52wVzpw3l7aGwgFeiOCGDxk= 2023-10-17T18:44:13.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XiYA52wVzpw3l7aGwgFeiOCGDxk='} 2023-10-17T18:44:13.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4c51638fb5a4dad-00652ed5fd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.059 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:13.059 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:13.059 DEBUG:boto:Method: HEAD 2023-10-17T18:44:13.060 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:13.060 DEBUG:boto:Data: 2023-10-17T18:44:13.060 DEBUG:boto:Headers: {} 2023-10-17T18:44:13.061 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.061 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.061 DEBUG:boto:Params: {} 2023-10-17T18:44:13.062 DEBUG:boto:Token: None 2023-10-17T18:44:13.062 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/ 2023-10-17T18:44:13.063 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws= 2023-10-17T18:44:13.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KXBVLtV3PnwhH1d0rbmezTZN3Ws=', 'Content-Length': '0'} 2023-10-17T18:44:13.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007625a71ceb67e6d0-00652ed5fd-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.066 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:13.066 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:13.067 DEBUG:boto:Method: PUT 2023-10-17T18:44:13.067 DEBUG:boto:Path: /hbmuog-54/d 2023-10-17T18:44:13.067 DEBUG:boto:Data: 2023-10-17T18:44:13.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:13.068 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:13.069 DEBUG:boto:Port: 8000 2023-10-17T18:44:13.069 DEBUG:boto:Params: {} 2023-10-17T18:44:13.069 DEBUG:boto:Token: None 2023-10-17T18:44:13.070 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:13 GMT /hbmuog-54/d 2023-10-17T18:44:13.070 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ja/WNuScaDBI5Q12D0Wj9xZx/cY= 2023-10-17T18:44:13.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ja/WNuScaDBI5Q12D0Wj9xZx/cY='} 2023-10-17T18:44:13.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbfe061bd490e61d-00652ed5fd-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:13.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:13.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:13.116 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:13.168 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:13.168 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:13.201 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:44:13.214 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:44:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568252.962736_631.1", 2023-10-17T18:44:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:44:12.962736Z" 2023-10-17T18:44:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:44:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.216 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:44:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:13.217 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:44:13.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.218 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:44:13.219 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568252.962736_631.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:13.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:44:13.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:13.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:13.221 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:13.270 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:13.271 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:13.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:13.299+0000 7ff2d2cc8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:13.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:13.300+0000 7ff2d2cc8780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:13.313 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:44:13.313 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:44:13.314 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:44:13.314 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:44:13.314 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:44:13.314 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:44:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:44:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:44:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568221.472769_623.1", 2023-10-17T18:44:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.472769Z", 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:44:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:44:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:44:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:44:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:44:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:44:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:44:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:44:13Z" 2023-10-17T18:44:13.326 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:44:13.326 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568221.472769_623.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:43:41.472769Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:44:13Z" } 2023-10-17T18:44:13.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568221.472769_623.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:43:41.472769Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:44:13.327 DEBUG:rgw_multi.tests:log_status={0: '1_1697568252.962736_631.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:13.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568221.472769_623.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:13.328 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568252.962736_631.1 target=1_1697568221.472769_623.1 2023-10-17T18:44:18.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:18.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:18.331 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:18.388 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:18.388 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:18.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:18.421+0000 7f5a89eea780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:18.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:18.422+0000 7f5a89eea780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:18.436 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:44:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:44:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:44:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:44:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:44:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:44:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:44:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:44:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568252.962736_631.1", 2023-10-17T18:44:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:44:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:12.962736Z", 2023-10-17T18:44:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:44:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:44:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:44:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:44:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:44:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:44:18Z" 2023-10-17T18:44:18.448 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:44:18.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568252.962736_631.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:44:12.962736Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:44:18Z" } 2023-10-17T18:44:18.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568252.962736_631.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:44:12.962736Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:44:18.450 DEBUG:rgw_multi.tests:log_status={0: '1_1697568252.962736_631.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:18.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568252.962736_631.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:18.450 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:44:18.451 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-54 2023-10-17T18:44:18.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:18.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:18.452 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:44:18.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:18.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:18.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:18.546+0000 7f4b0aed3780 1 waiting to reach incremental sync.. 2023-10-17T18:44:23.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:23.547+0000 7f4b0aed3780 1 bucket sync caught up with source: 2023-10-17T18:44:23.549 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.33.6, , , , 00000000001.14.6, , , 00000000001.35.6, , 00000000001.20.6] 2023-10-17T18:44:23.549 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.33.6, , , , 00000000001.14.6, , , 00000000001.35.6, , 00000000001.20.6] 2023-10-17T18:44:23.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:23.547+0000 7f4b0aed3780 0 bucket checkpoint complete 2023-10-17T18:44:23.566 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-54 2023-10-17T18:44:23.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:23.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:23.568 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:44:23.633 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:23.633 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:23.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:23.669+0000 7fa793c99780 1 waiting to reach incremental sync.. 2023-10-17T18:44:28.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:28.669+0000 7fa793c99780 1 waiting to reach incremental sync.. 2023-10-17T18:44:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:33.671+0000 7fa793c99780 1 bucket sync caught up with source: 2023-10-17T18:44:33.673 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.256.6, , , , 00000000001.25.6, , , 00000000001.255.6, , 00000000001.305.6] 2023-10-17T18:44:33.673 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.256.6, , , , 00000000001.25.6, , , 00000000001.255.6, , 00000000001.305.6] 2023-10-17T18:44:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:33.671+0000 7fa793c99780 0 bucket checkpoint complete 2023-10-17T18:44:33.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-54 zones={a2, a1} 2023-10-17T18:44:33.690 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.691 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.691 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.691 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:33.692 DEBUG:boto:Data: 2023-10-17T18:44:33.692 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.693 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.693 DEBUG:boto:Params: {} 2023-10-17T18:44:33.694 DEBUG:boto:Token: None 2023-10-17T18:44:33.695 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.695 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qfVPqdU2rbdaUxczrczNxkzFJ5k= 2023-10-17T18:44:33.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qfVPqdU2rbdaUxczrczNxkzFJ5k=', 'Content-Length': '0'} 2023-10-17T18:44:33.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f151df4aa701de3-00652ed611-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.699 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.699 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.700 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.700 DEBUG:boto:Path: /hbmuog-54/ 2023-10-17T18:44:33.700 DEBUG:boto:Data: 2023-10-17T18:44:33.701 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.702 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.702 DEBUG:boto:Params: {} 2023-10-17T18:44:33.703 DEBUG:boto:Token: None 2023-10-17T18:44:33.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qfVPqdU2rbdaUxczrczNxkzFJ5k= 2023-10-17T18:44:33.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qfVPqdU2rbdaUxczrczNxkzFJ5k=', 'Content-Length': '0'} 2023-10-17T18:44:33.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e15fa5551767a80-00652ed611-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.707 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:44:33.708 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.708 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.708 DEBUG:boto:path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.709 DEBUG:boto:auth_path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.709 DEBUG:boto:Method: GET 2023-10-17T18:44:33.710 DEBUG:boto:Path: /hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.710 DEBUG:boto:Data: 2023-10-17T18:44:33.710 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.711 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.712 DEBUG:boto:Params: {} 2023-10-17T18:44:33.712 DEBUG:boto:Token: None 2023-10-17T18:44:33.712 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/?versions 2023-10-17T18:44:33.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw= 2023-10-17T18:44:33.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw=', 'Content-Length': '0'} 2023-10-17T18:44:33.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf7f55e4e3ec3ba-00652ed611-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.716 DEBUG:boto:b'hbmuog-54999falseanulltrue2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.717 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:44:33.717 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:44:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:44:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:44:33.719 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:44:33.719 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.719 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.720 DEBUG:boto:path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.720 DEBUG:boto:auth_path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.720 DEBUG:boto:Method: GET 2023-10-17T18:44:33.721 DEBUG:boto:Path: /hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.721 DEBUG:boto:Data: 2023-10-17T18:44:33.722 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.722 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.723 DEBUG:boto:Params: {} 2023-10-17T18:44:33.723 DEBUG:boto:Token: None 2023-10-17T18:44:33.724 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/?versions 2023-10-17T18:44:33.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw= 2023-10-17T18:44:33.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw=', 'Content-Length': '0'} 2023-10-17T18:44:33.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba514f43775006f-00652ed611-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.728 DEBUG:boto:b'hbmuog-54999falseanulltrue2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.729 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:44:33.729 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:44:33.729 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:44:33.730 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:44:33.730 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.730 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.731 DEBUG:boto:path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.731 DEBUG:boto:auth_path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.732 DEBUG:boto:Method: GET 2023-10-17T18:44:33.732 DEBUG:boto:Path: /hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.732 DEBUG:boto:Data: 2023-10-17T18:44:33.733 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.733 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.733 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.734 DEBUG:boto:Params: {} 2023-10-17T18:44:33.734 DEBUG:boto:Token: None 2023-10-17T18:44:33.735 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/?versions 2023-10-17T18:44:33.735 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw= 2023-10-17T18:44:33.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw=', 'Content-Length': '0'} 2023-10-17T18:44:33.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e1430bcc6df69e-00652ed611-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.741 DEBUG:boto:b'hbmuog-54999falseanulltrue2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.742 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.743 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.743 DEBUG:boto:path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.743 DEBUG:boto:auth_path=/hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.744 DEBUG:boto:Method: GET 2023-10-17T18:44:33.744 DEBUG:boto:Path: /hbmuog-54/?versions&max-keys=999 2023-10-17T18:44:33.744 DEBUG:boto:Data: 2023-10-17T18:44:33.745 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.745 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.746 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.746 DEBUG:boto:Params: {} 2023-10-17T18:44:33.747 DEBUG:boto:Token: None 2023-10-17T18:44:33.747 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/?versions 2023-10-17T18:44:33.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw= 2023-10-17T18:44:33.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2bZnQiPJ91OvhJ9CncKLsmsyyw=', 'Content-Length': '0'} 2023-10-17T18:44:33.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a6e3ae88c025d77-00652ed611-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.751 DEBUG:boto:b'hbmuog-54999falseanulltrue2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:33.752 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.753 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.753 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.753 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.754 DEBUG:boto:Method: GET 2023-10-17T18:44:33.754 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.755 DEBUG:boto:Data: 2023-10-17T18:44:33.755 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.756 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.756 DEBUG:boto:Params: {} 2023-10-17T18:44:33.757 DEBUG:boto:Token: None 2023-10-17T18:44:33.757 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ= 2023-10-17T18:44:33.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ=', 'Content-Length': '0'} 2023-10-17T18:44:33.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4e6cd97c5f8279-00652ed611-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.802 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.803 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.803 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.804 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.804 DEBUG:boto:Method: GET 2023-10-17T18:44:33.804 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.805 DEBUG:boto:Data: 2023-10-17T18:44:33.805 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.806 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.806 DEBUG:boto:Params: {} 2023-10-17T18:44:33.807 DEBUG:boto:Token: None 2023-10-17T18:44:33.807 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.808 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ= 2023-10-17T18:44:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ=', 'Content-Length': '0'} 2023-10-17T18:44:33.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e13361d066c814-00652ed611-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.811 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.811 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.812 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.812 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.813 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.813 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.813 DEBUG:boto:Data: 2023-10-17T18:44:33.814 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.814 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.814 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.815 DEBUG:boto:Params: {} 2023-10-17T18:44:33.815 DEBUG:boto:Token: None 2023-10-17T18:44:33.816 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JLdFG+roqQFiJxDz3SkyRI0omDI= 2023-10-17T18:44:33.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JLdFG+roqQFiJxDz3SkyRI0omDI=', 'Content-Length': '0'} 2023-10-17T18:44:33.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5fc8c5636d34a4e-00652ed611-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.819 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.819 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.820 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.820 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.820 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.821 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.821 DEBUG:boto:Data: 2023-10-17T18:44:33.821 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.822 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.823 DEBUG:boto:Params: {} 2023-10-17T18:44:33.823 DEBUG:boto:Token: None 2023-10-17T18:44:33.823 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JLdFG+roqQFiJxDz3SkyRI0omDI= 2023-10-17T18:44:33.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JLdFG+roqQFiJxDz3SkyRI0omDI=', 'Content-Length': '0'} 2023-10-17T18:44:33.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d83105a595aed9-00652ed611-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:33.827 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.828 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.828 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.828 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.829 DEBUG:boto:Method: GET 2023-10-17T18:44:33.829 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.830 DEBUG:boto:Data: 2023-10-17T18:44:33.830 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.830 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.831 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.831 DEBUG:boto:Params: {} 2023-10-17T18:44:33.832 DEBUG:boto:Token: None 2023-10-17T18:44:33.832 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ= 2023-10-17T18:44:33.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ=', 'Content-Length': '0'} 2023-10-17T18:44:33.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9df8796fd023215-00652ed611-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.835 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.836 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.836 DEBUG:boto:path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.837 DEBUG:boto:auth_path=/hbmuog-54/a?versionId=null 2023-10-17T18:44:33.837 DEBUG:boto:Method: GET 2023-10-17T18:44:33.837 DEBUG:boto:Path: /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.838 DEBUG:boto:Data: 2023-10-17T18:44:33.838 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.839 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.839 DEBUG:boto:Params: {} 2023-10-17T18:44:33.840 DEBUG:boto:Token: None 2023-10-17T18:44:33.840 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a?versionId=null 2023-10-17T18:44:33.841 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ= 2023-10-17T18:44:33.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4I77LTeK+7ALT3eOh36YNlL2xJQ=', 'Content-Length': '0'} 2023-10-17T18:44:33.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000797488d2bdb54ad1-00652ed611-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.886 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.887 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.887 DEBUG:boto:path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.887 DEBUG:boto:auth_path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.888 DEBUG:boto:Method: GET 2023-10-17T18:44:33.888 DEBUG:boto:Path: /hbmuog-54/?prefix=a 2023-10-17T18:44:33.889 DEBUG:boto:Data: 2023-10-17T18:44:33.889 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.890 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.890 DEBUG:boto:Params: {} 2023-10-17T18:44:33.891 DEBUG:boto:Token: None 2023-10-17T18:44:33.891 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s= 2023-10-17T18:44:33.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s=', 'Content-Length': '0'} 2023-10-17T18:44:33.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc8d30acc0f141c-00652ed611-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.895 DEBUG:boto:b'hbmuog-54a1000falsea2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.896 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.896 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.897 DEBUG:boto:path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.897 DEBUG:boto:auth_path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.898 DEBUG:boto:Method: GET 2023-10-17T18:44:33.898 DEBUG:boto:Path: /hbmuog-54/?prefix=a 2023-10-17T18:44:33.898 DEBUG:boto:Data: 2023-10-17T18:44:33.899 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.900 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.900 DEBUG:boto:Params: {} 2023-10-17T18:44:33.900 DEBUG:boto:Token: None 2023-10-17T18:44:33.901 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s= 2023-10-17T18:44:33.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s=', 'Content-Length': '0'} 2023-10-17T18:44:33.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91ecff7275743a7-00652ed611-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.905 DEBUG:boto:b'hbmuog-54a1000falsea2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.906 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.906 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.906 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.907 DEBUG:boto:Path: /hbmuog-54/a 2023-10-17T18:44:33.907 DEBUG:boto:Data: 2023-10-17T18:44:33.908 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.908 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.909 DEBUG:boto:Params: {} 2023-10-17T18:44:33.909 DEBUG:boto:Token: None 2023-10-17T18:44:33.909 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a 2023-10-17T18:44:33.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eM1oigoGQ2phSHAMicm0cpZczWk= 2023-10-17T18:44:33.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eM1oigoGQ2phSHAMicm0cpZczWk=', 'Content-Length': '0'} 2023-10-17T18:44:33.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031194690a06b0ced-00652ed611-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.913 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.913 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.914 DEBUG:boto:Method: HEAD 2023-10-17T18:44:33.914 DEBUG:boto:Path: /hbmuog-54/a 2023-10-17T18:44:33.914 DEBUG:boto:Data: 2023-10-17T18:44:33.915 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.915 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.916 DEBUG:boto:Params: {} 2023-10-17T18:44:33.916 DEBUG:boto:Token: None 2023-10-17T18:44:33.917 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a 2023-10-17T18:44:33.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eM1oigoGQ2phSHAMicm0cpZczWk= 2023-10-17T18:44:33.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eM1oigoGQ2phSHAMicm0cpZczWk=', 'Content-Length': '0'} 2023-10-17T18:44:33.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db56fe466b47055a-00652ed611-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:44:33.920 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.921 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.921 DEBUG:boto:Method: GET 2023-10-17T18:44:33.921 DEBUG:boto:Path: /hbmuog-54/a 2023-10-17T18:44:33.922 DEBUG:boto:Data: 2023-10-17T18:44:33.922 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.923 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.924 DEBUG:boto:Params: {} 2023-10-17T18:44:33.924 DEBUG:boto:Token: None 2023-10-17T18:44:33.924 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a 2023-10-17T18:44:33.925 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jse5YFFL1rrH+edR8MJrwwYJ0Mo= 2023-10-17T18:44:33.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jse5YFFL1rrH+edR8MJrwwYJ0Mo=', 'Content-Length': '0'} 2023-10-17T18:44:33.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5c25c7f7f90819-00652ed611-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.974 DEBUG:boto:path=/hbmuog-54/a 2023-10-17T18:44:33.975 DEBUG:boto:auth_path=/hbmuog-54/a 2023-10-17T18:44:33.975 DEBUG:boto:Method: GET 2023-10-17T18:44:33.975 DEBUG:boto:Path: /hbmuog-54/a 2023-10-17T18:44:33.976 DEBUG:boto:Data: 2023-10-17T18:44:33.976 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.977 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.977 DEBUG:boto:Params: {} 2023-10-17T18:44:33.978 DEBUG:boto:Token: None 2023-10-17T18:44:33.978 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/a 2023-10-17T18:44:33.979 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jse5YFFL1rrH+edR8MJrwwYJ0Mo= 2023-10-17T18:44:33.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jse5YFFL1rrH+edR8MJrwwYJ0Mo=', 'Content-Length': '0'} 2023-10-17T18:44:33.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fdd16ad98845d1a-00652ed611-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.982 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.982 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.983 DEBUG:boto:path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.983 DEBUG:boto:auth_path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.984 DEBUG:boto:Method: GET 2023-10-17T18:44:33.984 DEBUG:boto:Path: /hbmuog-54/?prefix=a 2023-10-17T18:44:33.984 DEBUG:boto:Data: 2023-10-17T18:44:33.985 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.985 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:33.986 DEBUG:boto:Port: 8001 2023-10-17T18:44:33.986 DEBUG:boto:Params: {} 2023-10-17T18:44:33.986 DEBUG:boto:Token: None 2023-10-17T18:44:33.987 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.987 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s= 2023-10-17T18:44:33.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s=', 'Content-Length': '0'} 2023-10-17T18:44:33.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a10e8a367ef089a5-00652ed611-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:33.991 DEBUG:boto:b'hbmuog-54a1000falsea2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:33.991 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:33.992 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:33.992 DEBUG:boto:path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.993 DEBUG:boto:auth_path=/hbmuog-54/?prefix=a 2023-10-17T18:44:33.993 DEBUG:boto:Method: GET 2023-10-17T18:44:33.994 DEBUG:boto:Path: /hbmuog-54/?prefix=a 2023-10-17T18:44:33.994 DEBUG:boto:Data: 2023-10-17T18:44:33.994 DEBUG:boto:Headers: {} 2023-10-17T18:44:33.995 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:33.995 DEBUG:boto:Port: 8000 2023-10-17T18:44:33.996 DEBUG:boto:Params: {} 2023-10-17T18:44:33.996 DEBUG:boto:Token: None 2023-10-17T18:44:33.997 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:33 GMT /hbmuog-54/ 2023-10-17T18:44:33.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s= 2023-10-17T18:44:33.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CNIWSa+BLCAnluk8uI7xloxyQ1s=', 'Content-Length': '0'} 2023-10-17T18:44:34.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944bf208bad0f542-00652ed611-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.001 DEBUG:boto:b'hbmuog-54a1000falsea2023-10-17T18:44:12.979Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:34.002 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.002 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.002 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.003 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.003 DEBUG:boto:Method: GET 2023-10-17T18:44:34.004 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.004 DEBUG:boto:Data: 2023-10-17T18:44:34.004 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.005 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.006 DEBUG:boto:Params: {} 2023-10-17T18:44:34.006 DEBUG:boto:Token: None 2023-10-17T18:44:34.007 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.007 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM= 2023-10-17T18:44:34.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM=', 'Content-Length': '0'} 2023-10-17T18:44:34.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02ee6baeda42b8e-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.010 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.011 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.011 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.011 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.012 DEBUG:boto:Method: GET 2023-10-17T18:44:34.012 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.013 DEBUG:boto:Data: 2023-10-17T18:44:34.013 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.013 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.014 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.014 DEBUG:boto:Params: {} 2023-10-17T18:44:34.015 DEBUG:boto:Token: None 2023-10-17T18:44:34.015 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.016 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM= 2023-10-17T18:44:34.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM=', 'Content-Length': '0'} 2023-10-17T18:44:34.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000586c3ad8f17a69-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.019 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.019 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.020 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.020 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.020 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.021 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.021 DEBUG:boto:Data: 2023-10-17T18:44:34.021 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.022 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.023 DEBUG:boto:Params: {} 2023-10-17T18:44:34.023 DEBUG:boto:Token: None 2023-10-17T18:44:34.023 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OQixM7JTmfpAmRiWP1idDioA/1Y= 2023-10-17T18:44:34.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OQixM7JTmfpAmRiWP1idDioA/1Y=', 'Content-Length': '0'} 2023-10-17T18:44:34.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac12b3eacf7bbef9-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.027 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.027 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.028 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.028 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.028 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.029 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.029 DEBUG:boto:Data: 2023-10-17T18:44:34.029 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.030 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.031 DEBUG:boto:Params: {} 2023-10-17T18:44:34.031 DEBUG:boto:Token: None 2023-10-17T18:44:34.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OQixM7JTmfpAmRiWP1idDioA/1Y= 2023-10-17T18:44:34.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OQixM7JTmfpAmRiWP1idDioA/1Y=', 'Content-Length': '0'} 2023-10-17T18:44:34.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1a6e87e435e9a4-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:34.035 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.036 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.036 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.036 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.037 DEBUG:boto:Method: GET 2023-10-17T18:44:34.037 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.038 DEBUG:boto:Data: 2023-10-17T18:44:34.038 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.039 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.039 DEBUG:boto:Params: {} 2023-10-17T18:44:34.040 DEBUG:boto:Token: None 2023-10-17T18:44:34.040 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM= 2023-10-17T18:44:34.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM=', 'Content-Length': '0'} 2023-10-17T18:44:34.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ed7de5b4158b33-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.086 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.087 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.087 DEBUG:boto:path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.087 DEBUG:boto:auth_path=/hbmuog-54/b?versionId=null 2023-10-17T18:44:34.088 DEBUG:boto:Method: GET 2023-10-17T18:44:34.088 DEBUG:boto:Path: /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.089 DEBUG:boto:Data: 2023-10-17T18:44:34.089 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.090 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.090 DEBUG:boto:Params: {} 2023-10-17T18:44:34.091 DEBUG:boto:Token: None 2023-10-17T18:44:34.091 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b?versionId=null 2023-10-17T18:44:34.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM= 2023-10-17T18:44:34.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L4aiFAkf0Eg9wNpPbpvxDBFNhoM=', 'Content-Length': '0'} 2023-10-17T18:44:34.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebf1c3f36b2c030a-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.095 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.096 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.096 DEBUG:boto:path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.096 DEBUG:boto:auth_path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.097 DEBUG:boto:Method: GET 2023-10-17T18:44:34.097 DEBUG:boto:Path: /hbmuog-54/?prefix=b 2023-10-17T18:44:34.098 DEBUG:boto:Data: 2023-10-17T18:44:34.098 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.099 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.099 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.100 DEBUG:boto:Params: {} 2023-10-17T18:44:34.100 DEBUG:boto:Token: None 2023-10-17T18:44:34.101 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.101 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034178b941b3bcd82-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.105 DEBUG:boto:b'hbmuog-54b1000falseb2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.105 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.106 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.106 DEBUG:boto:path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.107 DEBUG:boto:auth_path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.107 DEBUG:boto:Method: GET 2023-10-17T18:44:34.107 DEBUG:boto:Path: /hbmuog-54/?prefix=b 2023-10-17T18:44:34.108 DEBUG:boto:Data: 2023-10-17T18:44:34.108 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.109 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.109 DEBUG:boto:Params: {} 2023-10-17T18:44:34.110 DEBUG:boto:Token: None 2023-10-17T18:44:34.110 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784c5706569b3f04-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.154 DEBUG:boto:b'hbmuog-54b1000falseb2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.155 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.155 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.156 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.156 DEBUG:boto:Path: /hbmuog-54/b 2023-10-17T18:44:34.156 DEBUG:boto:Data: 2023-10-17T18:44:34.157 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.157 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.158 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.158 DEBUG:boto:Params: {} 2023-10-17T18:44:34.158 DEBUG:boto:Token: None 2023-10-17T18:44:34.159 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b 2023-10-17T18:44:34.159 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5IM3IUjlaScN7JGlFnqvXhWQY+o= 2023-10-17T18:44:34.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5IM3IUjlaScN7JGlFnqvXhWQY+o=', 'Content-Length': '0'} 2023-10-17T18:44:34.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f5ed482f8422b7-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.162 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.163 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.163 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.164 DEBUG:boto:Path: /hbmuog-54/b 2023-10-17T18:44:34.164 DEBUG:boto:Data: 2023-10-17T18:44:34.164 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.165 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.165 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.165 DEBUG:boto:Params: {} 2023-10-17T18:44:34.166 DEBUG:boto:Token: None 2023-10-17T18:44:34.166 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b 2023-10-17T18:44:34.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5IM3IUjlaScN7JGlFnqvXhWQY+o= 2023-10-17T18:44:34.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5IM3IUjlaScN7JGlFnqvXhWQY+o=', 'Content-Length': '0'} 2023-10-17T18:44:34.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006921510470186ce1-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:44:34.170 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.170 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.171 DEBUG:boto:Method: GET 2023-10-17T18:44:34.171 DEBUG:boto:Path: /hbmuog-54/b 2023-10-17T18:44:34.172 DEBUG:boto:Data: 2023-10-17T18:44:34.172 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.172 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.173 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.173 DEBUG:boto:Params: {} 2023-10-17T18:44:34.174 DEBUG:boto:Token: None 2023-10-17T18:44:34.174 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b 2023-10-17T18:44:34.175 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hwLMWUDb+93Vjv9OCunQrw8eCDQ= 2023-10-17T18:44:34.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hwLMWUDb+93Vjv9OCunQrw8eCDQ=', 'Content-Length': '0'} 2023-10-17T18:44:34.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4a66223dc7acd1-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.178 DEBUG:boto:path=/hbmuog-54/b 2023-10-17T18:44:34.178 DEBUG:boto:auth_path=/hbmuog-54/b 2023-10-17T18:44:34.179 DEBUG:boto:Method: GET 2023-10-17T18:44:34.179 DEBUG:boto:Path: /hbmuog-54/b 2023-10-17T18:44:34.180 DEBUG:boto:Data: 2023-10-17T18:44:34.180 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.180 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.181 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.181 DEBUG:boto:Params: {} 2023-10-17T18:44:34.182 DEBUG:boto:Token: None 2023-10-17T18:44:34.182 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/b 2023-10-17T18:44:34.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hwLMWUDb+93Vjv9OCunQrw8eCDQ= 2023-10-17T18:44:34.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hwLMWUDb+93Vjv9OCunQrw8eCDQ=', 'Content-Length': '0'} 2023-10-17T18:44:34.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5e3a9b693b3755-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.186 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.186 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.186 DEBUG:boto:path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.187 DEBUG:boto:auth_path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.187 DEBUG:boto:Method: GET 2023-10-17T18:44:34.188 DEBUG:boto:Path: /hbmuog-54/?prefix=b 2023-10-17T18:44:34.188 DEBUG:boto:Data: 2023-10-17T18:44:34.188 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.189 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.189 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.190 DEBUG:boto:Params: {} 2023-10-17T18:44:34.190 DEBUG:boto:Token: None 2023-10-17T18:44:34.191 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.191 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f8d069109211f0b-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.195 DEBUG:boto:b'hbmuog-54b1000falseb2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.195 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.196 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.196 DEBUG:boto:path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.196 DEBUG:boto:auth_path=/hbmuog-54/?prefix=b 2023-10-17T18:44:34.197 DEBUG:boto:Method: GET 2023-10-17T18:44:34.197 DEBUG:boto:Path: /hbmuog-54/?prefix=b 2023-10-17T18:44:34.198 DEBUG:boto:Data: 2023-10-17T18:44:34.198 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.199 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.199 DEBUG:boto:Params: {} 2023-10-17T18:44:34.200 DEBUG:boto:Token: None 2023-10-17T18:44:34.200 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc0ff453c6d7c7b-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.204 DEBUG:boto:b'hbmuog-54b1000falseb2023-10-17T18:44:13.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:34.205 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.206 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.206 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.206 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.207 DEBUG:boto:Method: GET 2023-10-17T18:44:34.207 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.208 DEBUG:boto:Data: 2023-10-17T18:44:34.208 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.209 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.209 DEBUG:boto:Params: {} 2023-10-17T18:44:34.210 DEBUG:boto:Token: None 2023-10-17T18:44:34.210 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc= 2023-10-17T18:44:34.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc=', 'Content-Length': '0'} 2023-10-17T18:44:34.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b93153e9c79fe2-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.213 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.214 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.214 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.215 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.215 DEBUG:boto:Method: GET 2023-10-17T18:44:34.215 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.216 DEBUG:boto:Data: 2023-10-17T18:44:34.216 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.216 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.217 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.217 DEBUG:boto:Params: {} 2023-10-17T18:44:34.218 DEBUG:boto:Token: None 2023-10-17T18:44:34.218 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc= 2023-10-17T18:44:34.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc=', 'Content-Length': '0'} 2023-10-17T18:44:34.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1661a1d46e40c4e-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.221 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.222 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.222 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.223 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.223 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.223 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.224 DEBUG:boto:Data: 2023-10-17T18:44:34.224 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.225 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.225 DEBUG:boto:Params: {} 2023-10-17T18:44:34.225 DEBUG:boto:Token: None 2023-10-17T18:44:34.226 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OEAgHhEK8q54kMAyCi5k15ZRdKQ= 2023-10-17T18:44:34.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OEAgHhEK8q54kMAyCi5k15ZRdKQ=', 'Content-Length': '0'} 2023-10-17T18:44:34.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b682af2aa16c2a2e-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.229 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.230 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.230 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.231 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.231 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.231 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.232 DEBUG:boto:Data: 2023-10-17T18:44:34.232 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.233 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.233 DEBUG:boto:Params: {} 2023-10-17T18:44:34.234 DEBUG:boto:Token: None 2023-10-17T18:44:34.234 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OEAgHhEK8q54kMAyCi5k15ZRdKQ= 2023-10-17T18:44:34.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OEAgHhEK8q54kMAyCi5k15ZRdKQ=', 'Content-Length': '0'} 2023-10-17T18:44:34.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f164339220f7a-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:34.237 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.238 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.238 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.239 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.239 DEBUG:boto:Method: GET 2023-10-17T18:44:34.239 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.240 DEBUG:boto:Data: 2023-10-17T18:44:34.240 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.240 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.241 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.241 DEBUG:boto:Params: {} 2023-10-17T18:44:34.242 DEBUG:boto:Token: None 2023-10-17T18:44:34.242 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc= 2023-10-17T18:44:34.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc=', 'Content-Length': '0'} 2023-10-17T18:44:34.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a37ddfa2af1837-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.246 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.247 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.247 DEBUG:boto:path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.247 DEBUG:boto:auth_path=/hbmuog-54/c?versionId=null 2023-10-17T18:44:34.248 DEBUG:boto:Method: GET 2023-10-17T18:44:34.248 DEBUG:boto:Path: /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.248 DEBUG:boto:Data: 2023-10-17T18:44:34.249 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.250 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.250 DEBUG:boto:Params: {} 2023-10-17T18:44:34.250 DEBUG:boto:Token: None 2023-10-17T18:44:34.251 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c?versionId=null 2023-10-17T18:44:34.251 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc= 2023-10-17T18:44:34.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6NgHUhy8MnxiswRFXn7FqIV3kXc=', 'Content-Length': '0'} 2023-10-17T18:44:34.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a892e443e67307-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.254 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.254 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.255 DEBUG:boto:path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.255 DEBUG:boto:auth_path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.256 DEBUG:boto:Method: GET 2023-10-17T18:44:34.256 DEBUG:boto:Path: /hbmuog-54/?prefix=c 2023-10-17T18:44:34.256 DEBUG:boto:Data: 2023-10-17T18:44:34.257 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.257 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.258 DEBUG:boto:Params: {} 2023-10-17T18:44:34.258 DEBUG:boto:Token: None 2023-10-17T18:44:34.259 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e43a98fef53e72f3-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.262 DEBUG:boto:b'hbmuog-54c1000falsec2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.263 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.263 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.264 DEBUG:boto:path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.264 DEBUG:boto:auth_path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.265 DEBUG:boto:Method: GET 2023-10-17T18:44:34.265 DEBUG:boto:Path: /hbmuog-54/?prefix=c 2023-10-17T18:44:34.265 DEBUG:boto:Data: 2023-10-17T18:44:34.266 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.266 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.266 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.267 DEBUG:boto:Params: {} 2023-10-17T18:44:34.267 DEBUG:boto:Token: None 2023-10-17T18:44:34.268 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.268 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a661f80aba41ca63-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.271 DEBUG:boto:b'hbmuog-54c1000falsec2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.272 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.272 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.273 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.273 DEBUG:boto:Path: /hbmuog-54/c 2023-10-17T18:44:34.273 DEBUG:boto:Data: 2023-10-17T18:44:34.274 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.274 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.274 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.275 DEBUG:boto:Params: {} 2023-10-17T18:44:34.275 DEBUG:boto:Token: None 2023-10-17T18:44:34.276 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c 2023-10-17T18:44:34.276 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2Xskgid5Iz9inrRDeFhTSstBcFY= 2023-10-17T18:44:34.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2Xskgid5Iz9inrRDeFhTSstBcFY=', 'Content-Length': '0'} 2023-10-17T18:44:34.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ba4fc788b1e4f8-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.279 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.279 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.280 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.280 DEBUG:boto:Path: /hbmuog-54/c 2023-10-17T18:44:34.280 DEBUG:boto:Data: 2023-10-17T18:44:34.281 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.281 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.282 DEBUG:boto:Params: {} 2023-10-17T18:44:34.282 DEBUG:boto:Token: None 2023-10-17T18:44:34.283 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c 2023-10-17T18:44:34.283 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2Xskgid5Iz9inrRDeFhTSstBcFY= 2023-10-17T18:44:34.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2Xskgid5Iz9inrRDeFhTSstBcFY=', 'Content-Length': '0'} 2023-10-17T18:44:34.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d2488e61ec5638-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:44:34.286 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.287 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.287 DEBUG:boto:Method: GET 2023-10-17T18:44:34.287 DEBUG:boto:Path: /hbmuog-54/c 2023-10-17T18:44:34.288 DEBUG:boto:Data: 2023-10-17T18:44:34.288 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.289 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.290 DEBUG:boto:Params: {} 2023-10-17T18:44:34.290 DEBUG:boto:Token: None 2023-10-17T18:44:34.290 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c 2023-10-17T18:44:34.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:21ShLclM7Xg8EWfV6lcdOjSfE5o= 2023-10-17T18:44:34.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:21ShLclM7Xg8EWfV6lcdOjSfE5o=', 'Content-Length': '0'} 2023-10-17T18:44:34.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008205a91dfb2f8d95-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.338 DEBUG:boto:path=/hbmuog-54/c 2023-10-17T18:44:34.339 DEBUG:boto:auth_path=/hbmuog-54/c 2023-10-17T18:44:34.339 DEBUG:boto:Method: GET 2023-10-17T18:44:34.340 DEBUG:boto:Path: /hbmuog-54/c 2023-10-17T18:44:34.340 DEBUG:boto:Data: 2023-10-17T18:44:34.341 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.341 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.342 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.342 DEBUG:boto:Params: {} 2023-10-17T18:44:34.343 DEBUG:boto:Token: None 2023-10-17T18:44:34.343 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/c 2023-10-17T18:44:34.344 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:21ShLclM7Xg8EWfV6lcdOjSfE5o= 2023-10-17T18:44:34.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:21ShLclM7Xg8EWfV6lcdOjSfE5o=', 'Content-Length': '0'} 2023-10-17T18:44:34.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001202bec51c6a4811-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.347 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.347 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.349 DEBUG:boto:path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.350 DEBUG:boto:auth_path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.350 DEBUG:boto:Method: GET 2023-10-17T18:44:34.351 DEBUG:boto:Path: /hbmuog-54/?prefix=c 2023-10-17T18:44:34.351 DEBUG:boto:Data: 2023-10-17T18:44:34.351 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.352 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.353 DEBUG:boto:Params: {} 2023-10-17T18:44:34.353 DEBUG:boto:Token: None 2023-10-17T18:44:34.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.354 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62da985c6f5feea-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.358 DEBUG:boto:b'hbmuog-54c1000falsec2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.359 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.360 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.360 DEBUG:boto:path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.360 DEBUG:boto:auth_path=/hbmuog-54/?prefix=c 2023-10-17T18:44:34.361 DEBUG:boto:Method: GET 2023-10-17T18:44:34.361 DEBUG:boto:Path: /hbmuog-54/?prefix=c 2023-10-17T18:44:34.362 DEBUG:boto:Data: 2023-10-17T18:44:34.362 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.363 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.363 DEBUG:boto:Params: {} 2023-10-17T18:44:34.364 DEBUG:boto:Token: None 2023-10-17T18:44:34.364 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306f088dd179d806-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.369 DEBUG:boto:b'hbmuog-54c1000falsec2023-10-17T18:44:13.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:34.370 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.370 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.371 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.371 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.372 DEBUG:boto:Method: GET 2023-10-17T18:44:34.372 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.373 DEBUG:boto:Data: 2023-10-17T18:44:34.373 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.374 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.374 DEBUG:boto:Params: {} 2023-10-17T18:44:34.375 DEBUG:boto:Token: None 2023-10-17T18:44:34.375 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4= 2023-10-17T18:44:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4=', 'Content-Length': '0'} 2023-10-17T18:44:34.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e5f40096551051-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.379 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.379 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.380 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.380 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.381 DEBUG:boto:Method: GET 2023-10-17T18:44:34.381 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.382 DEBUG:boto:Data: 2023-10-17T18:44:34.382 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.382 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.383 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.383 DEBUG:boto:Params: {} 2023-10-17T18:44:34.384 DEBUG:boto:Token: None 2023-10-17T18:44:34.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4= 2023-10-17T18:44:34.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4=', 'Content-Length': '0'} 2023-10-17T18:44:34.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a119c11ebf49f2e-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.388 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.389 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.389 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.389 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.390 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.390 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.391 DEBUG:boto:Data: 2023-10-17T18:44:34.391 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.392 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.392 DEBUG:boto:Params: {} 2023-10-17T18:44:34.392 DEBUG:boto:Token: None 2023-10-17T18:44:34.393 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KIISGLbH2hxhvmL7AZmaNOA7qeg= 2023-10-17T18:44:34.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KIISGLbH2hxhvmL7AZmaNOA7qeg=', 'Content-Length': '0'} 2023-10-17T18:44:34.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b8a5e6ec4585ce-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.396 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.397 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.397 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.398 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.398 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.398 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.399 DEBUG:boto:Data: 2023-10-17T18:44:34.399 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.400 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.400 DEBUG:boto:Params: {} 2023-10-17T18:44:34.401 DEBUG:boto:Token: None 2023-10-17T18:44:34.401 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KIISGLbH2hxhvmL7AZmaNOA7qeg= 2023-10-17T18:44:34.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KIISGLbH2hxhvmL7AZmaNOA7qeg=', 'Content-Length': '0'} 2023-10-17T18:44:34.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59adbdf9d49a429-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:34.405 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.405 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.406 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.406 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.407 DEBUG:boto:Method: GET 2023-10-17T18:44:34.407 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.408 DEBUG:boto:Data: 2023-10-17T18:44:34.408 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.409 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.409 DEBUG:boto:Params: {} 2023-10-17T18:44:34.410 DEBUG:boto:Token: None 2023-10-17T18:44:34.410 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4= 2023-10-17T18:44:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4=', 'Content-Length': '0'} 2023-10-17T18:44:34.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3caea17ae756d15-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.414 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.415 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.415 DEBUG:boto:path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.415 DEBUG:boto:auth_path=/hbmuog-54/d?versionId=null 2023-10-17T18:44:34.416 DEBUG:boto:Method: GET 2023-10-17T18:44:34.416 DEBUG:boto:Path: /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.417 DEBUG:boto:Data: 2023-10-17T18:44:34.417 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.418 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.419 DEBUG:boto:Params: {} 2023-10-17T18:44:34.419 DEBUG:boto:Token: None 2023-10-17T18:44:34.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d?versionId=null 2023-10-17T18:44:34.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4= 2023-10-17T18:44:34.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zKIEN+y23g6y2keIon8kckHmHI4=', 'Content-Length': '0'} 2023-10-17T18:44:34.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69765f9c0f530bf-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.424 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.424 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.424 DEBUG:boto:path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.425 DEBUG:boto:auth_path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.425 DEBUG:boto:Method: GET 2023-10-17T18:44:34.426 DEBUG:boto:Path: /hbmuog-54/?prefix=d 2023-10-17T18:44:34.426 DEBUG:boto:Data: 2023-10-17T18:44:34.427 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.428 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.428 DEBUG:boto:Params: {} 2023-10-17T18:44:34.429 DEBUG:boto:Token: None 2023-10-17T18:44:34.429 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250fdcfdd4ac4316-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.433 DEBUG:boto:b'hbmuog-54d1000falsed2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.434 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.434 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.435 DEBUG:boto:path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.435 DEBUG:boto:auth_path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.436 DEBUG:boto:Method: GET 2023-10-17T18:44:34.436 DEBUG:boto:Path: /hbmuog-54/?prefix=d 2023-10-17T18:44:34.437 DEBUG:boto:Data: 2023-10-17T18:44:34.437 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.438 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.439 DEBUG:boto:Params: {} 2023-10-17T18:44:34.439 DEBUG:boto:Token: None 2023-10-17T18:44:34.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.440 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b6a9d029b53e51-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.486 DEBUG:boto:b'hbmuog-54d1000falsed2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.487 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.488 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.488 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.489 DEBUG:boto:Path: /hbmuog-54/d 2023-10-17T18:44:34.489 DEBUG:boto:Data: 2023-10-17T18:44:34.489 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.490 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.491 DEBUG:boto:Params: {} 2023-10-17T18:44:34.491 DEBUG:boto:Token: None 2023-10-17T18:44:34.492 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d 2023-10-17T18:44:34.492 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ik+yoQkmWCqaZrMdWM4cwRFw6R8= 2023-10-17T18:44:34.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ik+yoQkmWCqaZrMdWM4cwRFw6R8=', 'Content-Length': '0'} 2023-10-17T18:44:34.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed521a7a1d4bdad4-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.496 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.496 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.496 DEBUG:boto:Method: HEAD 2023-10-17T18:44:34.497 DEBUG:boto:Path: /hbmuog-54/d 2023-10-17T18:44:34.497 DEBUG:boto:Data: 2023-10-17T18:44:34.498 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.499 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.499 DEBUG:boto:Params: {} 2023-10-17T18:44:34.499 DEBUG:boto:Token: None 2023-10-17T18:44:34.500 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d 2023-10-17T18:44:34.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ik+yoQkmWCqaZrMdWM4cwRFw6R8= 2023-10-17T18:44:34.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ik+yoQkmWCqaZrMdWM4cwRFw6R8=', 'Content-Length': '0'} 2023-10-17T18:44:34.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c3aa0fb895cc4b-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:44:34.504 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.504 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.505 DEBUG:boto:Method: GET 2023-10-17T18:44:34.505 DEBUG:boto:Path: /hbmuog-54/d 2023-10-17T18:44:34.506 DEBUG:boto:Data: 2023-10-17T18:44:34.506 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.507 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.508 DEBUG:boto:Params: {} 2023-10-17T18:44:34.508 DEBUG:boto:Token: None 2023-10-17T18:44:34.509 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d 2023-10-17T18:44:34.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:De6jRd6k8TBByAgzyer2dJsDofg= 2023-10-17T18:44:34.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:De6jRd6k8TBByAgzyer2dJsDofg=', 'Content-Length': '0'} 2023-10-17T18:44:34.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-54:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d585c21f2c9eb9-00652ed612-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.554 DEBUG:boto:path=/hbmuog-54/d 2023-10-17T18:44:34.555 DEBUG:boto:auth_path=/hbmuog-54/d 2023-10-17T18:44:34.555 DEBUG:boto:Method: GET 2023-10-17T18:44:34.556 DEBUG:boto:Path: /hbmuog-54/d 2023-10-17T18:44:34.556 DEBUG:boto:Data: 2023-10-17T18:44:34.557 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.558 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.558 DEBUG:boto:Params: {} 2023-10-17T18:44:34.559 DEBUG:boto:Token: None 2023-10-17T18:44:34.559 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/d 2023-10-17T18:44:34.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:De6jRd6k8TBByAgzyer2dJsDofg= 2023-10-17T18:44:34.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:De6jRd6k8TBByAgzyer2dJsDofg=', 'Content-Length': '0'} 2023-10-17T18:44:34.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006618ae6fa4672bb2-00652ed612-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.563 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.563 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.564 DEBUG:boto:path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.564 DEBUG:boto:auth_path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.565 DEBUG:boto:Method: GET 2023-10-17T18:44:34.565 DEBUG:boto:Path: /hbmuog-54/?prefix=d 2023-10-17T18:44:34.566 DEBUG:boto:Data: 2023-10-17T18:44:34.566 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:44:34.567 DEBUG:boto:Port: 8001 2023-10-17T18:44:34.568 DEBUG:boto:Params: {} 2023-10-17T18:44:34.568 DEBUG:boto:Token: None 2023-10-17T18:44:34.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026056b9ec3a3e19c-00652ed612-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.573 DEBUG:boto:b'hbmuog-54d1000falsed2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.574 DEBUG:boto:path=/hbmuog-54/ 2023-10-17T18:44:34.574 DEBUG:boto:auth_path=/hbmuog-54/ 2023-10-17T18:44:34.575 DEBUG:boto:path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.575 DEBUG:boto:auth_path=/hbmuog-54/?prefix=d 2023-10-17T18:44:34.576 DEBUG:boto:Method: GET 2023-10-17T18:44:34.576 DEBUG:boto:Path: /hbmuog-54/?prefix=d 2023-10-17T18:44:34.577 DEBUG:boto:Data: 2023-10-17T18:44:34.577 DEBUG:boto:Headers: {} 2023-10-17T18:44:34.578 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:34.578 DEBUG:boto:Port: 8000 2023-10-17T18:44:34.578 DEBUG:boto:Params: {} 2023-10-17T18:44:34.579 DEBUG:boto:Token: None 2023-10-17T18:44:34.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:44:34 GMT /hbmuog-54/ 2023-10-17T18:44:34.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g= 2023-10-17T18:44:34.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qAN/rzm/gRW4ppvOASBkEI+fS0g=', 'Content-Length': '0'} 2023-10-17T18:44:34.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a26d8146fcd4fe6-00652ed612-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:44:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:34.584 DEBUG:boto:b'hbmuog-54d1000falsed2023-10-17T18:44:13.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:44:34.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-54 zones={a2, a1} 2023-10-17T18:44:34.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:34.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:34.586 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:44:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:34.643 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:34.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:34.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:34.702 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:34.751 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:34.751 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:34.792 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:44:34.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:34.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:34.795 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:34.845 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:34.845 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:34.885 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:44:34.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.886 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "1#00000000001.20.6", 2023-10-17T18:44:34.886 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.5747307249183264980", 2023-10-17T18:44:34.887 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:44:34.887 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "a", 2023-10-17T18:44:34.887 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:44:34.887 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:44:34.888 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:44:34.888 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.491105821Z", 2023-10-17T18:44:34.888 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:44:34.888 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:44:34.888 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 11 2023-10-17T18:44:34.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.889 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:44:34.889 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:44:34.889 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:44:34.889 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:44:34.890 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:44:34.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.890 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18" 2023-10-17T18:44:34.890 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:34.890 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:34.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.891 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "5#00000000001.27.6", 2023-10-17T18:44:34.891 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.909186991962247100", 2023-10-17T18:44:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:44:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "d", 2023-10-17T18:44:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:44:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:44:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:44:34.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.595040440Z", 2023-10-17T18:44:34.893 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:44:34.893 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:44:34.893 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:44:34.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.894 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:44:34.894 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:44:34.894 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:44:34.894 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:44:34.895 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:44:34.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.895 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18" 2023-10-17T18:44:34.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:34.895 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "8#00000000001.52.6", 2023-10-17T18:44:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.12169369283385095433", 2023-10-17T18:44:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:44:34.897 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "c", 2023-10-17T18:44:34.897 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:44:34.897 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:44:34.897 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:44:34.897 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.575582086Z", 2023-10-17T18:44:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:44:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:44:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:44:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:44:34.899 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:44:34.899 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:44:34.899 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:44:34.899 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:44:34.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.900 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18" 2023-10-17T18:44:34.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:34.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:34.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.901 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "10#00000000001.18.6", 2023-10-17T18:44:34.901 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.12784454509878143546", 2023-10-17T18:44:34.901 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:44:34.901 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "b", 2023-10-17T18:44:34.902 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:44:34.902 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:44:34.902 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:44:34.902 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:43:41.511344133Z", 2023-10-17T18:44:34.902 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:44:34.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:44:34.903 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 21 2023-10-17T18:44:34.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:34.903 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:44:34.904 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:44:34.904 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:44:34.904 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:44:34.904 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:44:34.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:34.905 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-53:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.18" 2023-10-17T18:44:34.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:34.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:34.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:34.906 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:44:34.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:34.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:44:34.908 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:44:34.955 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:34.956 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:34.999 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:34.998+0000 7f5ea70d5700 -1 req 17882464913304561057 0.002999917s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.000 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:34.998+0000 7f5e8b09d700 -1 req 3967499130671336395 0.002999917s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.008 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.007+0000 7f5e6985a700 -1 req 14112356935386089629 0.001999945s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.047 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.046+0000 7f5e60848700 -1 req 12813887199746059109 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.098 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.096+0000 7f5e4e824700 -1 req 4141178352225875414 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.100 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.099+0000 7f5e6f065700 -1 req 15230498448763601405 0.001999945s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.100 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.099+0000 7f5e5a03b700 -1 req 12266352506251451438 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.137 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.135+0000 7f5e57836700 -1 req 14711936134857293300 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.141 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.140+0000 7f5e337ee700 -1 req 17436322742146705565 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.143 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.142+0000 7f5e45011700 -1 req 10615034090121396904 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.144 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.143+0000 7f5e3d802700 -1 req 5144051488133798224 0.001999944s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.180 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.178+0000 7f5e2e7e4700 -1 req 9216711858407072666 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.185 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.183+0000 7f5e23fcf700 -1 req 15695765960454488350 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.186 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.185+0000 7f5e1f7c6700 -1 req 8331838660415143812 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.235 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.234+0000 7f5e07f97700 -1 req 17087034207235370699 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.275 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.274+0000 7f5ee7155700 -1 req 17571170214110092585 0.001999944s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.279 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.278+0000 7f5e0d7a2700 -1 req 7448418658799857270 0.001999945s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.314 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.313+0000 7f5e04790700 -1 req 9423291227141198077 0.001999944s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-28:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.28,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-28:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.28,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.323 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.322+0000 7f5ede143700 -1 req 9763857295969552752 0.001999945s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-21:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.21,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-21:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.21,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.325 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.323+0000 7f5ef396e700 -1 req 9684987644567735462 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-5:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.5,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-5:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.5,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.357 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.356+0000 7f5ee5151700 -1 req 18441432897064129321 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-51:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.16,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-51:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.16,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.365 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.363+0000 7f5eb08e8700 -1 req 15783226799089076682 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-20:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.20,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-20:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.20,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.401 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.399+0000 7f5ecc11f700 -1 req 3514303013235063975 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-27:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.27,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-27:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.27,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.407 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.406+0000 7f5eba8fc700 -1 req 8706470213248971646 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-26:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.26,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-26:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.26,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.451 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.450+0000 7f5e85892700 -1 req 60904427121088168 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-38:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.3,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-38:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.3,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.505 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.503+0000 7f5e84890700 -1 req 8630354870765807555 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-36:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.1,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-36:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.1,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.540 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:44:35.539+0000 7f5e66053700 -1 req 17617916930958838203 0.001999945s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-29:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.29,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-29:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.29,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:44:35.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.599 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:35.648 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:35.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:35.690 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:44:35.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-10-17T18:44:35.693 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-55 2023-10-17T18:44:35.694 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:44:35.694 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:44:35.695 DEBUG:boto:Method: PUT 2023-10-17T18:44:35.695 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:44:35.696 DEBUG:boto:Data: 2023-10-17T18:44:35.696 DEBUG:boto:Headers: {} 2023-10-17T18:44:35.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.697 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.697 DEBUG:boto:Params: {} 2023-10-17T18:44:35.698 DEBUG:boto:Token: None 2023-10-17T18:44:35.698 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/ 2023-10-17T18:44:35.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SAg9Z2hYeL/uKsL+QuawPOov3ew= 2023-10-17T18:44:35.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SAg9Z2hYeL/uKsL+QuawPOov3ew=', 'Content-Length': '0'} 2023-10-17T18:44:35.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079b6a6a4a6145fc9-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.714 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:44:35.714 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:44:35.715 DEBUG:boto:Method: HEAD 2023-10-17T18:44:35.715 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:44:35.715 DEBUG:boto:Data: 2023-10-17T18:44:35.716 DEBUG:boto:Headers: {} 2023-10-17T18:44:35.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.716 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.717 DEBUG:boto:Params: {} 2023-10-17T18:44:35.717 DEBUG:boto:Token: None 2023-10-17T18:44:35.718 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/ 2023-10-17T18:44:35.718 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T18:44:35.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T18:44:35.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4c9fe7fbfa229b2-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.722 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:44:35.723 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:44:35.723 DEBUG:boto:Method: PUT 2023-10-17T18:44:35.723 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:44:35.724 DEBUG:boto:Data: 2023-10-17T18:44:35.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:35.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.725 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.725 DEBUG:boto:Params: {} 2023-10-17T18:44:35.726 DEBUG:boto:Token: None 2023-10-17T18:44:35.726 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/a 2023-10-17T18:44:35.727 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3MFWJEUUghIs90BQAosZO0cbEy4= 2023-10-17T18:44:35.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3MFWJEUUghIs90BQAosZO0cbEy4='} 2023-10-17T18:44:35.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000012fd8e6d3d2e8fe-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.771 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:44:35.772 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:44:35.772 DEBUG:boto:Method: HEAD 2023-10-17T18:44:35.772 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:44:35.773 DEBUG:boto:Data: 2023-10-17T18:44:35.773 DEBUG:boto:Headers: {} 2023-10-17T18:44:35.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.774 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.775 DEBUG:boto:Params: {} 2023-10-17T18:44:35.775 DEBUG:boto:Token: None 2023-10-17T18:44:35.776 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/ 2023-10-17T18:44:35.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T18:44:35.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T18:44:35.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000321b320fe67b5f51-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.779 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:44:35.781 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:44:35.781 DEBUG:boto:Method: PUT 2023-10-17T18:44:35.782 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:44:35.782 DEBUG:boto:Data: 2023-10-17T18:44:35.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:35.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.783 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.784 DEBUG:boto:Params: {} 2023-10-17T18:44:35.785 DEBUG:boto:Token: None 2023-10-17T18:44:35.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/b 2023-10-17T18:44:35.786 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vCfwuNlmU5H3TWgeqEgQfHWKmtE= 2023-10-17T18:44:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vCfwuNlmU5H3TWgeqEgQfHWKmtE='} 2023-10-17T18:44:35.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068b42d41773f4d13-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.791 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:44:35.792 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:44:35.792 DEBUG:boto:Method: HEAD 2023-10-17T18:44:35.793 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:44:35.793 DEBUG:boto:Data: 2023-10-17T18:44:35.794 DEBUG:boto:Headers: {} 2023-10-17T18:44:35.794 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.794 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.795 DEBUG:boto:Params: {} 2023-10-17T18:44:35.795 DEBUG:boto:Token: None 2023-10-17T18:44:35.796 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/ 2023-10-17T18:44:35.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T18:44:35.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T18:44:35.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ede015d66e19b9d-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.799 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:44:35.800 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:44:35.800 DEBUG:boto:Method: PUT 2023-10-17T18:44:35.801 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:44:35.801 DEBUG:boto:Data: 2023-10-17T18:44:35.802 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:35.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.802 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.803 DEBUG:boto:Params: {} 2023-10-17T18:44:35.803 DEBUG:boto:Token: None 2023-10-17T18:44:35.804 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/c 2023-10-17T18:44:35.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OO/BNij3RTA9UHTP/jyiT7SMgWY= 2023-10-17T18:44:35.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OO/BNij3RTA9UHTP/jyiT7SMgWY='} 2023-10-17T18:44:35.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002077ee5fc1a66ab4-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.847 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:44:35.848 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:44:35.848 DEBUG:boto:Method: HEAD 2023-10-17T18:44:35.848 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:44:35.849 DEBUG:boto:Data: 2023-10-17T18:44:35.849 DEBUG:boto:Headers: {} 2023-10-17T18:44:35.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.850 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.851 DEBUG:boto:Params: {} 2023-10-17T18:44:35.851 DEBUG:boto:Token: None 2023-10-17T18:44:35.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/ 2023-10-17T18:44:35.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T18:44:35.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T18:44:35.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f71536734718b53-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.855 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:44:35.856 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:44:35.856 DEBUG:boto:Method: PUT 2023-10-17T18:44:35.857 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:44:35.857 DEBUG:boto:Data: 2023-10-17T18:44:35.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:44:35.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:44:35.859 DEBUG:boto:Port: 8000 2023-10-17T18:44:35.859 DEBUG:boto:Params: {} 2023-10-17T18:44:35.859 DEBUG:boto:Token: None 2023-10-17T18:44:35.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:44:35 GMT /hbmuog-55/d 2023-10-17T18:44:35.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GRdVmGpIHLRCYKudlUIftaVmzVE= 2023-10-17T18:44:35.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GRdVmGpIHLRCYKudlUIftaVmzVE='} 2023-10-17T18:44:35.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bef12cbd9b9ef9f8-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:44:35.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.867 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:35.917 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:35.917 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:35.947 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:44:35.959 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:44:35.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:35.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568275.712131_636.1", 2023-10-17T18:44:35.960 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:44:35.712131Z" 2023-10-17T18:44:35.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:35.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:35.961 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:35.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:44:35.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:35.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:35.962 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:35.962 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:44:35.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:35.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:35.962 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:35.963 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:44:35.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:35.963 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:44:35.964 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:35.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:44:35.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:35.966 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:36.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:36.044+0000 7f371ee56780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:36.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:36.045+0000 7f371ee56780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:36.059 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:44:36.059 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:44:36.059 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:44:36.059 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:44:36.059 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:44:36.060 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:44:36.060 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:36.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.060 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:44:36.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:36.061 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:44:36.061 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:36.061 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:36.061 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568275.706429_633.1", 2023-10-17T18:44:36.061 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:36.062 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:44:36.062 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:36.062 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.706429Z", 2023-10-17T18:44:36.062 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:36.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:36.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:36.063 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:44:36.063 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:36.065 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:44:36.065 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:36.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:36.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:36.066 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:44:36.066 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:36.066 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:36.066 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:36.067 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:36.067 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:36.067 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:36.067 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:36.068 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:36.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:36.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:36.068 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:44:36.069 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:36.069 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:36.069 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:36.069 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:36.070 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:36.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:36.070 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:36.070 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:36.070 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:36.071 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:36.071 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:36.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.071 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:44:36.071 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:44:36.072 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:44:36.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:36.072 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:44:36Z" 2023-10-17T18:44:36.072 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:44:36.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568275.706429_633.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:44:35.706429Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:44:36Z" } 2023-10-17T18:44:36.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568275.706429_633.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:44:35.706429Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:44:36.074 DEBUG:rgw_multi.tests:log_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:36.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568275.706429_633.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:36.075 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568275.712131_636.1 target=1_1697568275.706429_633.1 2023-10-17T18:44:41.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:41.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:44:41.077 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:44:41.136 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:41.136 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:41.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:41.167+0000 7faefb953780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:41.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:41.168+0000 7faefb953780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:44:41.183 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:44:41.183 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:44:41.184 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:44:41.184 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:44:41.184 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:44:41.184 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:44:41.184 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:41.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.185 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:44:41.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:41.185 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:44:41.185 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568275.712131_636.1", 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:41.186 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.712131Z", 2023-10-17T18:44:41.187 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:41.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:41.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:41.187 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:44:41.188 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:41.188 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:41.188 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:44:41.188 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:41.188 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:41.189 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:41.189 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:44:41.189 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:41.189 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:41.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:41.190 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:44:41.190 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:41.190 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:41.190 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:41.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:41.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:44:41.192 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:44:41.192 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:44:41.192 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:44:41.193 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:44:41.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:41.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:44:41.194 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:44:41.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.194 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:44:41.195 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:44:41.195 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:44:41.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:44:41.195 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:44:41Z" 2023-10-17T18:44:41.195 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:44:41.196 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568275.712131_636.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:44:35.712131Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:44:41Z" } 2023-10-17T18:44:41.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568275.712131_636.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:44:35.712131Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:44:41.197 DEBUG:rgw_multi.tests:log_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:41.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:44:41.198 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:44:41.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T18:44:41.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:41.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:44:41.199 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:44:41.259 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:44:41.259 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:44:41.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:41.292+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T18:44:46.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:46.294+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T18:44:51.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:51.294+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T18:44:56.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:44:56.295+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T18:45:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:01.295+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T18:45:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:06.297+0000 7f6fbb07e780 1 bucket sync caught up with source: 2023-10-17T18:45:06.299 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.26.6, , , , 00000000001.18.6, , , 00000000001.49.6, , 00000000001.110.6] 2023-10-17T18:45:06.300 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.26.6, , , , 00000000001.18.6, , , 00000000001.49.6, , 00000000001.110.6] 2023-10-17T18:45:06.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:06.297+0000 7f6fbb07e780 0 bucket checkpoint complete 2023-10-17T18:45:06.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T18:45:06.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:45:06.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:45:06.318 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:45:06.381 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:06.382 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:06.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:06.420+0000 7f40b8b39780 1 bucket sync caught up with source: 2023-10-17T18:45:06.422 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.146.6, , , , 00000000001.50.6, , , 00000000001.221.6, , 00000000001.406.6] 2023-10-17T18:45:06.422 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.146.6, , , , 00000000001.50.6, , , 00000000001.221.6, , 00000000001.406.6] 2023-10-17T18:45:06.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:06.420+0000 7f40b8b39780 0 bucket checkpoint complete 2023-10-17T18:45:06.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:45:06.436 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.436 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.437 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.437 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:06.437 DEBUG:boto:Data: 2023-10-17T18:45:06.438 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.439 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.439 DEBUG:boto:Params: {} 2023-10-17T18:45:06.440 DEBUG:boto:Token: None 2023-10-17T18:45:06.440 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE= 2023-10-17T18:45:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE=', 'Content-Length': '0'} 2023-10-17T18:45:06.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000796b1091626f0358-00652ed632-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.444 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.444 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.445 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.445 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:06.446 DEBUG:boto:Data: 2023-10-17T18:45:06.446 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.447 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.447 DEBUG:boto:Params: {} 2023-10-17T18:45:06.448 DEBUG:boto:Token: None 2023-10-17T18:45:06.448 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE= 2023-10-17T18:45:06.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE=', 'Content-Length': '0'} 2023-10-17T18:45:06.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000369086b13ab47f13-00652ed632-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:45:06.452 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.453 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.453 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.453 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.454 DEBUG:boto:Method: GET 2023-10-17T18:45:06.454 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.455 DEBUG:boto:Data: 2023-10-17T18:45:06.455 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.456 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.456 DEBUG:boto:Params: {} 2023-10-17T18:45:06.457 DEBUG:boto:Token: None 2023-10-17T18:45:06.457 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/?versions 2023-10-17T18:45:06.458 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T18:45:06.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T18:45:06.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006493f162474a9bbc-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.461 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.462 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:45:06.462 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:45:06.463 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:45:06.463 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:45:06.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:45:06.465 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.466 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.466 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.466 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.467 DEBUG:boto:Method: GET 2023-10-17T18:45:06.467 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.468 DEBUG:boto:Data: 2023-10-17T18:45:06.468 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.469 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.469 DEBUG:boto:Params: {} 2023-10-17T18:45:06.470 DEBUG:boto:Token: None 2023-10-17T18:45:06.470 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/?versions 2023-10-17T18:45:06.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T18:45:06.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T18:45:06.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2fbc2a85165e71c-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.474 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:45:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:45:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:45:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:45:06.476 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.477 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.477 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.478 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.478 DEBUG:boto:Method: GET 2023-10-17T18:45:06.478 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.479 DEBUG:boto:Data: 2023-10-17T18:45:06.479 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.479 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.480 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.480 DEBUG:boto:Params: {} 2023-10-17T18:45:06.481 DEBUG:boto:Token: None 2023-10-17T18:45:06.481 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/?versions 2023-10-17T18:45:06.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T18:45:06.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T18:45:06.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf482fe93b76d92-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.485 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.485 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.486 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.486 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.487 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.487 DEBUG:boto:Method: GET 2023-10-17T18:45:06.488 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:45:06.488 DEBUG:boto:Data: 2023-10-17T18:45:06.488 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.489 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.490 DEBUG:boto:Params: {} 2023-10-17T18:45:06.490 DEBUG:boto:Token: None 2023-10-17T18:45:06.491 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/?versions 2023-10-17T18:45:06.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T18:45:06.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T18:45:06.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c594de954af1f139-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.494 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:45:06.495 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.496 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.496 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.497 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.497 DEBUG:boto:Method: GET 2023-10-17T18:45:06.497 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.498 DEBUG:boto:Data: 2023-10-17T18:45:06.498 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.499 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.500 DEBUG:boto:Params: {} 2023-10-17T18:45:06.500 DEBUG:boto:Token: None 2023-10-17T18:45:06.500 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T18:45:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T18:45:06.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919d40f60fb50a03-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.504 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.504 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.505 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.505 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.506 DEBUG:boto:Method: GET 2023-10-17T18:45:06.506 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.507 DEBUG:boto:Data: 2023-10-17T18:45:06.507 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.508 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.508 DEBUG:boto:Params: {} 2023-10-17T18:45:06.509 DEBUG:boto:Token: None 2023-10-17T18:45:06.509 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T18:45:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T18:45:06.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001448386bd1bfe9ac-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.554 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.555 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.555 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.555 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.556 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.556 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.557 DEBUG:boto:Data: 2023-10-17T18:45:06.557 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.558 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.558 DEBUG:boto:Params: {} 2023-10-17T18:45:06.558 DEBUG:boto:Token: None 2023-10-17T18:45:06.559 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.559 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4= 2023-10-17T18:45:06.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4=', 'Content-Length': '0'} 2023-10-17T18:45:06.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413b5994ba002bee-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.562 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.562 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.563 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.563 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.564 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.564 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.564 DEBUG:boto:Data: 2023-10-17T18:45:06.565 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.565 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.565 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.566 DEBUG:boto:Params: {} 2023-10-17T18:45:06.566 DEBUG:boto:Token: None 2023-10-17T18:45:06.567 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.567 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4= 2023-10-17T18:45:06.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4=', 'Content-Length': '0'} 2023-10-17T18:45:06.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c874687b8320c44d-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:45:06.570 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.571 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.571 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.572 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.572 DEBUG:boto:Method: GET 2023-10-17T18:45:06.572 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.573 DEBUG:boto:Data: 2023-10-17T18:45:06.573 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.574 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.574 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.574 DEBUG:boto:Params: {} 2023-10-17T18:45:06.575 DEBUG:boto:Token: None 2023-10-17T18:45:06.575 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.576 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T18:45:06.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T18:45:06.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69f251439fcfdf0-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.578 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.579 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.579 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.580 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:45:06.580 DEBUG:boto:Method: GET 2023-10-17T18:45:06.581 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.581 DEBUG:boto:Data: 2023-10-17T18:45:06.581 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.582 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.583 DEBUG:boto:Params: {} 2023-10-17T18:45:06.583 DEBUG:boto:Token: None 2023-10-17T18:45:06.584 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:45:06.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T18:45:06.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T18:45:06.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdad742a36987599-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.587 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.587 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.588 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.588 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.588 DEBUG:boto:Method: GET 2023-10-17T18:45:06.589 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:45:06.589 DEBUG:boto:Data: 2023-10-17T18:45:06.589 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.590 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.591 DEBUG:boto:Params: {} 2023-10-17T18:45:06.591 DEBUG:boto:Token: None 2023-10-17T18:45:06.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034713855188e80a8-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.595 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.596 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.596 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.597 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.597 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.597 DEBUG:boto:Method: GET 2023-10-17T18:45:06.598 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:45:06.598 DEBUG:boto:Data: 2023-10-17T18:45:06.599 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.599 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.600 DEBUG:boto:Params: {} 2023-10-17T18:45:06.600 DEBUG:boto:Token: None 2023-10-17T18:45:06.601 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc613644b31e10b-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.604 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.605 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.605 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.606 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.606 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:45:06.606 DEBUG:boto:Data: 2023-10-17T18:45:06.607 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.608 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.608 DEBUG:boto:Params: {} 2023-10-17T18:45:06.608 DEBUG:boto:Token: None 2023-10-17T18:45:06.609 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a 2023-10-17T18:45:06.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM= 2023-10-17T18:45:06.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM=', 'Content-Length': '0'} 2023-10-17T18:45:06.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfe92f410cd1cc9-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.612 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.612 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.613 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.613 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:45:06.613 DEBUG:boto:Data: 2023-10-17T18:45:06.614 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.614 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.615 DEBUG:boto:Params: {} 2023-10-17T18:45:06.615 DEBUG:boto:Token: None 2023-10-17T18:45:06.616 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a 2023-10-17T18:45:06.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM= 2023-10-17T18:45:06.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM=', 'Content-Length': '0'} 2023-10-17T18:45:06.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003734017191c63b3a-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:45:06.619 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.619 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.620 DEBUG:boto:Method: GET 2023-10-17T18:45:06.620 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:45:06.621 DEBUG:boto:Data: 2023-10-17T18:45:06.621 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.622 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.622 DEBUG:boto:Params: {} 2023-10-17T18:45:06.623 DEBUG:boto:Token: None 2023-10-17T18:45:06.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a 2023-10-17T18:45:06.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM= 2023-10-17T18:45:06.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM=', 'Content-Length': '0'} 2023-10-17T18:45:06.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8572085dedffe5-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.666 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:45:06.667 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:45:06.667 DEBUG:boto:Method: GET 2023-10-17T18:45:06.668 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:45:06.668 DEBUG:boto:Data: 2023-10-17T18:45:06.668 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.669 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.669 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.670 DEBUG:boto:Params: {} 2023-10-17T18:45:06.670 DEBUG:boto:Token: None 2023-10-17T18:45:06.671 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/a 2023-10-17T18:45:06.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM= 2023-10-17T18:45:06.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM=', 'Content-Length': '0'} 2023-10-17T18:45:06.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b1d0ba41394495-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.674 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.675 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.675 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.675 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.676 DEBUG:boto:Method: GET 2023-10-17T18:45:06.676 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:45:06.677 DEBUG:boto:Data: 2023-10-17T18:45:06.677 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.678 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.678 DEBUG:boto:Params: {} 2023-10-17T18:45:06.679 DEBUG:boto:Token: None 2023-10-17T18:45:06.679 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.680 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d551f0209795c50-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.683 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.684 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.684 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.684 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.685 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:45:06.685 DEBUG:boto:Method: GET 2023-10-17T18:45:06.686 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:45:06.686 DEBUG:boto:Data: 2023-10-17T18:45:06.686 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.687 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.688 DEBUG:boto:Params: {} 2023-10-17T18:45:06.688 DEBUG:boto:Token: None 2023-10-17T18:45:06.689 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.689 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d5fadd9f3a03563-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.692 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:45:06.693 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.694 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.694 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.695 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.695 DEBUG:boto:Method: GET 2023-10-17T18:45:06.695 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.696 DEBUG:boto:Data: 2023-10-17T18:45:06.696 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.696 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.697 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.697 DEBUG:boto:Params: {} 2023-10-17T18:45:06.698 DEBUG:boto:Token: None 2023-10-17T18:45:06.698 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T18:45:06.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T18:45:06.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7b714a1caa4a10-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.701 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.702 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.702 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.703 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.703 DEBUG:boto:Method: GET 2023-10-17T18:45:06.703 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.704 DEBUG:boto:Data: 2023-10-17T18:45:06.704 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.705 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.705 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.705 DEBUG:boto:Params: {} 2023-10-17T18:45:06.706 DEBUG:boto:Token: None 2023-10-17T18:45:06.706 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T18:45:06.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T18:45:06.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e7a04edbff2924-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.710 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.710 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.711 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.711 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.711 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.712 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.712 DEBUG:boto:Data: 2023-10-17T18:45:06.712 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.713 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.713 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.713 DEBUG:boto:Params: {} 2023-10-17T18:45:06.714 DEBUG:boto:Token: None 2023-10-17T18:45:06.714 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI= 2023-10-17T18:45:06.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI=', 'Content-Length': '0'} 2023-10-17T18:45:06.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5784e0b84ca25f-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.717 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.718 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.718 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.719 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.719 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.719 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.720 DEBUG:boto:Data: 2023-10-17T18:45:06.720 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.721 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.721 DEBUG:boto:Params: {} 2023-10-17T18:45:06.722 DEBUG:boto:Token: None 2023-10-17T18:45:06.722 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI= 2023-10-17T18:45:06.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI=', 'Content-Length': '0'} 2023-10-17T18:45:06.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1edf61cd3bc4c3-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:45:06.726 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.726 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.727 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.727 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.727 DEBUG:boto:Method: GET 2023-10-17T18:45:06.728 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.728 DEBUG:boto:Data: 2023-10-17T18:45:06.728 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.729 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.730 DEBUG:boto:Params: {} 2023-10-17T18:45:06.730 DEBUG:boto:Token: None 2023-10-17T18:45:06.731 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T18:45:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T18:45:06.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952381b147a3b654-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.734 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.735 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.735 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.735 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:45:06.736 DEBUG:boto:Method: GET 2023-10-17T18:45:06.736 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.736 DEBUG:boto:Data: 2023-10-17T18:45:06.737 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.738 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.738 DEBUG:boto:Params: {} 2023-10-17T18:45:06.739 DEBUG:boto:Token: None 2023-10-17T18:45:06.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:45:06.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T18:45:06.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T18:45:06.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c039ba492fb68-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.742 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.743 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.743 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.744 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.744 DEBUG:boto:Method: GET 2023-10-17T18:45:06.745 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:45:06.745 DEBUG:boto:Data: 2023-10-17T18:45:06.745 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.746 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.747 DEBUG:boto:Params: {} 2023-10-17T18:45:06.747 DEBUG:boto:Token: None 2023-10-17T18:45:06.748 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb5d7b04df101da-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.751 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.752 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.753 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.753 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.753 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.754 DEBUG:boto:Method: GET 2023-10-17T18:45:06.754 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:45:06.755 DEBUG:boto:Data: 2023-10-17T18:45:06.755 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.756 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.756 DEBUG:boto:Params: {} 2023-10-17T18:45:06.757 DEBUG:boto:Token: None 2023-10-17T18:45:06.757 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef05ec451fb49de-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.761 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.762 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.762 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.762 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.763 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:45:06.763 DEBUG:boto:Data: 2023-10-17T18:45:06.763 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.764 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.764 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.764 DEBUG:boto:Params: {} 2023-10-17T18:45:06.765 DEBUG:boto:Token: None 2023-10-17T18:45:06.765 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b 2023-10-17T18:45:06.766 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo= 2023-10-17T18:45:06.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo=', 'Content-Length': '0'} 2023-10-17T18:45:06.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd06e845ad135dd-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.769 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.769 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.769 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.770 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:45:06.770 DEBUG:boto:Data: 2023-10-17T18:45:06.771 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.771 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.772 DEBUG:boto:Params: {} 2023-10-17T18:45:06.772 DEBUG:boto:Token: None 2023-10-17T18:45:06.773 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b 2023-10-17T18:45:06.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo= 2023-10-17T18:45:06.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo=', 'Content-Length': '0'} 2023-10-17T18:45:06.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b25f09d8319ec96-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:45:06.776 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.776 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.777 DEBUG:boto:Method: GET 2023-10-17T18:45:06.777 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:45:06.778 DEBUG:boto:Data: 2023-10-17T18:45:06.778 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.779 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.779 DEBUG:boto:Params: {} 2023-10-17T18:45:06.780 DEBUG:boto:Token: None 2023-10-17T18:45:06.780 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b 2023-10-17T18:45:06.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik= 2023-10-17T18:45:06.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik=', 'Content-Length': '0'} 2023-10-17T18:45:06.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dbe62eb65cfeec-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.826 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:45:06.827 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:45:06.827 DEBUG:boto:Method: GET 2023-10-17T18:45:06.827 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:45:06.828 DEBUG:boto:Data: 2023-10-17T18:45:06.828 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.829 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.829 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.829 DEBUG:boto:Params: {} 2023-10-17T18:45:06.830 DEBUG:boto:Token: None 2023-10-17T18:45:06.830 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/b 2023-10-17T18:45:06.831 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik= 2023-10-17T18:45:06.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik=', 'Content-Length': '0'} 2023-10-17T18:45:06.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e23e9f4771a0e54-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.834 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.834 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.834 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.835 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.835 DEBUG:boto:Method: GET 2023-10-17T18:45:06.836 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:45:06.836 DEBUG:boto:Data: 2023-10-17T18:45:06.836 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.837 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.838 DEBUG:boto:Params: {} 2023-10-17T18:45:06.838 DEBUG:boto:Token: None 2023-10-17T18:45:06.839 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a8d2d7d2b54681-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.842 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.843 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.844 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.844 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.844 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:45:06.845 DEBUG:boto:Method: GET 2023-10-17T18:45:06.845 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:45:06.846 DEBUG:boto:Data: 2023-10-17T18:45:06.846 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.846 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.847 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.847 DEBUG:boto:Params: {} 2023-10-17T18:45:06.848 DEBUG:boto:Token: None 2023-10-17T18:45:06.848 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee403291b8787d5-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.852 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:45:06.853 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.853 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.854 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.854 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.855 DEBUG:boto:Method: GET 2023-10-17T18:45:06.855 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.855 DEBUG:boto:Data: 2023-10-17T18:45:06.856 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.857 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.857 DEBUG:boto:Params: {} 2023-10-17T18:45:06.857 DEBUG:boto:Token: None 2023-10-17T18:45:06.858 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T18:45:06.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T18:45:06.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c80e87a14756e2-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.861 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.861 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.862 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.862 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.863 DEBUG:boto:Method: GET 2023-10-17T18:45:06.863 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.864 DEBUG:boto:Data: 2023-10-17T18:45:06.864 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.864 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.865 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.865 DEBUG:boto:Params: {} 2023-10-17T18:45:06.865 DEBUG:boto:Token: None 2023-10-17T18:45:06.866 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.866 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T18:45:06.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T18:45:06.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b60be710b80135-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.910 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.911 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.911 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.911 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.912 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.912 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.913 DEBUG:boto:Data: 2023-10-17T18:45:06.913 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.914 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.914 DEBUG:boto:Params: {} 2023-10-17T18:45:06.915 DEBUG:boto:Token: None 2023-10-17T18:45:06.915 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk= 2023-10-17T18:45:06.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk=', 'Content-Length': '0'} 2023-10-17T18:45:06.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c402c304929b0658-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.919 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.919 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.919 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.920 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.920 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.921 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.921 DEBUG:boto:Data: 2023-10-17T18:45:06.921 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.922 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.922 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.922 DEBUG:boto:Params: {} 2023-10-17T18:45:06.923 DEBUG:boto:Token: None 2023-10-17T18:45:06.923 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.924 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk= 2023-10-17T18:45:06.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk=', 'Content-Length': '0'} 2023-10-17T18:45:06.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095835a57ef358a57-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:45:06.927 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.927 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.928 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.928 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.929 DEBUG:boto:Method: GET 2023-10-17T18:45:06.929 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.929 DEBUG:boto:Data: 2023-10-17T18:45:06.930 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.931 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.931 DEBUG:boto:Params: {} 2023-10-17T18:45:06.931 DEBUG:boto:Token: None 2023-10-17T18:45:06.932 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T18:45:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T18:45:06.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f6bfbf1375f79d-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.935 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.935 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.936 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.936 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:45:06.937 DEBUG:boto:Method: GET 2023-10-17T18:45:06.937 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.937 DEBUG:boto:Data: 2023-10-17T18:45:06.938 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.939 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.939 DEBUG:boto:Params: {} 2023-10-17T18:45:06.939 DEBUG:boto:Token: None 2023-10-17T18:45:06.940 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:45:06.940 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T18:45:06.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T18:45:06.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e7f7f8e0f26cfe-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.943 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.943 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.944 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:45:06.944 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:45:06.944 DEBUG:boto:Method: GET 2023-10-17T18:45:06.945 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:45:06.945 DEBUG:boto:Data: 2023-10-17T18:45:06.946 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.946 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.947 DEBUG:boto:Params: {} 2023-10-17T18:45:06.947 DEBUG:boto:Token: None 2023-10-17T18:45:06.948 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9bc04136672fa6f-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.951 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.952 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:06.952 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:06.953 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:45:06.953 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:45:06.954 DEBUG:boto:Method: GET 2023-10-17T18:45:06.954 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:45:06.954 DEBUG:boto:Data: 2023-10-17T18:45:06.955 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.955 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.956 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.956 DEBUG:boto:Params: {} 2023-10-17T18:45:06.957 DEBUG:boto:Token: None 2023-10-17T18:45:06.957 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/ 2023-10-17T18:45:06.957 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T18:45:06.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T18:45:06.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f684a54b230c680-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.961 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:06.961 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.962 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.962 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.962 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:45:06.963 DEBUG:boto:Data: 2023-10-17T18:45:06.963 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.963 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.964 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.964 DEBUG:boto:Params: {} 2023-10-17T18:45:06.965 DEBUG:boto:Token: None 2023-10-17T18:45:06.965 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c 2023-10-17T18:45:06.965 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8= 2023-10-17T18:45:06.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8=', 'Content-Length': '0'} 2023-10-17T18:45:06.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068887a8e8e2c0c28-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.968 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.969 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.969 DEBUG:boto:Method: HEAD 2023-10-17T18:45:06.969 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:45:06.970 DEBUG:boto:Data: 2023-10-17T18:45:06.970 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:06.971 DEBUG:boto:Port: 8000 2023-10-17T18:45:06.971 DEBUG:boto:Params: {} 2023-10-17T18:45:06.972 DEBUG:boto:Token: None 2023-10-17T18:45:06.972 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c 2023-10-17T18:45:06.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8= 2023-10-17T18:45:06.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8=', 'Content-Length': '0'} 2023-10-17T18:45:06.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ff2223e03c17d8-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:45:06.976 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:06.976 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:06.977 DEBUG:boto:Method: GET 2023-10-17T18:45:06.977 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:45:06.977 DEBUG:boto:Data: 2023-10-17T18:45:06.978 DEBUG:boto:Headers: {} 2023-10-17T18:45:06.978 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:06.979 DEBUG:boto:Port: 8001 2023-10-17T18:45:06.979 DEBUG:boto:Params: {} 2023-10-17T18:45:06.979 DEBUG:boto:Token: None 2023-10-17T18:45:06.980 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:06 GMT /hbmuog-55/c 2023-10-17T18:45:06.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3ovWD/7xYZULXZwonzRhROxsxtg= 2023-10-17T18:45:06.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3ovWD/7xYZULXZwonzRhROxsxtg=', 'Content-Length': '0'} 2023-10-17T18:45:06.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b22db588e66406-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.026 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:45:07.027 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:45:07.027 DEBUG:boto:Method: GET 2023-10-17T18:45:07.027 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:45:07.028 DEBUG:boto:Data: 2023-10-17T18:45:07.028 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.029 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.029 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.029 DEBUG:boto:Params: {} 2023-10-17T18:45:07.030 DEBUG:boto:Token: None 2023-10-17T18:45:07.030 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/c 2023-10-17T18:45:07.031 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SkHs6NR6JRnepy7phJQMWSdCdbI= 2023-10-17T18:45:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SkHs6NR6JRnepy7phJQMWSdCdbI=', 'Content-Length': '0'} 2023-10-17T18:45:07.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce73fe7f090069c-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.034 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.034 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.035 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:45:07.035 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:45:07.035 DEBUG:boto:Method: GET 2023-10-17T18:45:07.036 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:45:07.036 DEBUG:boto:Data: 2023-10-17T18:45:07.037 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.037 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.038 DEBUG:boto:Params: {} 2023-10-17T18:45:07.038 DEBUG:boto:Token: None 2023-10-17T18:45:07.039 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.039 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a216d18b07d566a1-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.042 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.043 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.043 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.044 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:45:07.044 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:45:07.044 DEBUG:boto:Method: GET 2023-10-17T18:45:07.045 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:45:07.045 DEBUG:boto:Data: 2023-10-17T18:45:07.045 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.046 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.047 DEBUG:boto:Params: {} 2023-10-17T18:45:07.047 DEBUG:boto:Token: None 2023-10-17T18:45:07.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006807472f4b895388-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.051 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:45:07.052 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.053 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.053 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.053 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.054 DEBUG:boto:Method: GET 2023-10-17T18:45:07.054 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.055 DEBUG:boto:Data: 2023-10-17T18:45:07.055 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.056 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.056 DEBUG:boto:Params: {} 2023-10-17T18:45:07.057 DEBUG:boto:Token: None 2023-10-17T18:45:07.057 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T18:45:07.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T18:45:07.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2a72dc670fdf0e-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.061 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.061 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.061 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.062 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.062 DEBUG:boto:Method: GET 2023-10-17T18:45:07.063 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.063 DEBUG:boto:Data: 2023-10-17T18:45:07.063 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.064 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.064 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.065 DEBUG:boto:Params: {} 2023-10-17T18:45:07.065 DEBUG:boto:Token: None 2023-10-17T18:45:07.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T18:45:07.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T18:45:07.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f06436aec5001df-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.110 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.111 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.111 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.111 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.112 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.112 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.113 DEBUG:boto:Data: 2023-10-17T18:45:07.113 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.114 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.114 DEBUG:boto:Params: {} 2023-10-17T18:45:07.114 DEBUG:boto:Token: None 2023-10-17T18:45:07.115 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c= 2023-10-17T18:45:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c=', 'Content-Length': '0'} 2023-10-17T18:45:07.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45b2a9b292b5e2c-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.119 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.119 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.119 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.120 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.120 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.121 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.121 DEBUG:boto:Data: 2023-10-17T18:45:07.121 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.122 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.122 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.122 DEBUG:boto:Params: {} 2023-10-17T18:45:07.123 DEBUG:boto:Token: None 2023-10-17T18:45:07.123 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.124 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c= 2023-10-17T18:45:07.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c=', 'Content-Length': '0'} 2023-10-17T18:45:07.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417eff1f6b43e5bc-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:45:07.127 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.127 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.128 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.128 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.129 DEBUG:boto:Method: GET 2023-10-17T18:45:07.129 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.129 DEBUG:boto:Data: 2023-10-17T18:45:07.130 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.131 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.131 DEBUG:boto:Params: {} 2023-10-17T18:45:07.131 DEBUG:boto:Token: None 2023-10-17T18:45:07.132 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.132 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T18:45:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T18:45:07.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541da552fc2c9fde-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.135 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.135 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.136 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.136 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:45:07.137 DEBUG:boto:Method: GET 2023-10-17T18:45:07.137 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.137 DEBUG:boto:Data: 2023-10-17T18:45:07.138 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.139 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.139 DEBUG:boto:Params: {} 2023-10-17T18:45:07.139 DEBUG:boto:Token: None 2023-10-17T18:45:07.140 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:45:07.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T18:45:07.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T18:45:07.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ebabc8f10a99e1-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.143 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.143 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.144 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.144 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.145 DEBUG:boto:Method: GET 2023-10-17T18:45:07.145 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:45:07.145 DEBUG:boto:Data: 2023-10-17T18:45:07.146 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.147 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.147 DEBUG:boto:Params: {} 2023-10-17T18:45:07.147 DEBUG:boto:Token: None 2023-10-17T18:45:07.148 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2c302d9570478e-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.151 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.152 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.152 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.153 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.153 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.154 DEBUG:boto:Method: GET 2023-10-17T18:45:07.154 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:45:07.154 DEBUG:boto:Data: 2023-10-17T18:45:07.155 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.156 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.156 DEBUG:boto:Params: {} 2023-10-17T18:45:07.156 DEBUG:boto:Token: None 2023-10-17T18:45:07.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa30345ffcc01683-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.161 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.161 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.162 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.162 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.163 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:45:07.163 DEBUG:boto:Data: 2023-10-17T18:45:07.163 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.164 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.164 DEBUG:boto:Params: {} 2023-10-17T18:45:07.165 DEBUG:boto:Token: None 2023-10-17T18:45:07.165 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d 2023-10-17T18:45:07.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI= 2023-10-17T18:45:07.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI=', 'Content-Length': '0'} 2023-10-17T18:45:07.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc73d99441d6b0eb-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.168 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.169 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.169 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.169 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:45:07.170 DEBUG:boto:Data: 2023-10-17T18:45:07.170 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.171 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.171 DEBUG:boto:Params: {} 2023-10-17T18:45:07.172 DEBUG:boto:Token: None 2023-10-17T18:45:07.172 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d 2023-10-17T18:45:07.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI= 2023-10-17T18:45:07.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI=', 'Content-Length': '0'} 2023-10-17T18:45:07.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002137673f73d9ba93-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:45:07.176 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.176 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.177 DEBUG:boto:Method: GET 2023-10-17T18:45:07.177 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:45:07.178 DEBUG:boto:Data: 2023-10-17T18:45:07.178 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.179 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.179 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.179 DEBUG:boto:Params: {} 2023-10-17T18:45:07.180 DEBUG:boto:Token: None 2023-10-17T18:45:07.180 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d 2023-10-17T18:45:07.181 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY= 2023-10-17T18:45:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY=', 'Content-Length': '0'} 2023-10-17T18:45:07.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006473954e2ed1c1d3-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.226 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:45:07.227 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:45:07.227 DEBUG:boto:Method: GET 2023-10-17T18:45:07.227 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:45:07.228 DEBUG:boto:Data: 2023-10-17T18:45:07.228 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.229 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.229 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.229 DEBUG:boto:Params: {} 2023-10-17T18:45:07.230 DEBUG:boto:Token: None 2023-10-17T18:45:07.230 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/d 2023-10-17T18:45:07.231 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY= 2023-10-17T18:45:07.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY=', 'Content-Length': '0'} 2023-10-17T18:45:07.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f5b25df14e119c-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.234 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.234 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.235 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.235 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.236 DEBUG:boto:Method: GET 2023-10-17T18:45:07.236 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:45:07.236 DEBUG:boto:Data: 2023-10-17T18:45:07.237 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.237 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:45:07.237 DEBUG:boto:Port: 8001 2023-10-17T18:45:07.238 DEBUG:boto:Params: {} 2023-10-17T18:45:07.238 DEBUG:boto:Token: None 2023-10-17T18:45:07.239 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2926edb0d2836a-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.242 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.243 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.243 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.244 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.244 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:45:07.245 DEBUG:boto:Method: GET 2023-10-17T18:45:07.245 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:45:07.245 DEBUG:boto:Data: 2023-10-17T18:45:07.246 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.247 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.247 DEBUG:boto:Params: {} 2023-10-17T18:45:07.247 DEBUG:boto:Token: None 2023-10-17T18:45:07.248 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T18:45:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T18:45:07.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f621f5ac89cd02-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.252 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:45:07.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:45:07.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.254 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:45:07.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.345 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:45:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding": "None", 2023-10-17T18:45:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "current_index": { 2023-10-17T18:45:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:45:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "Normal", 2023-10-17T18:45:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "normal": { 2023-10-17T18:45:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 11, 2023-10-17T18:45:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:45:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "logs": [ 2023-10-17T18:45:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:45:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 11, 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:45:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:45:07.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:45:07.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:45:07.352 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:45:07.403 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.403 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.446 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:45:07.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.446 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "1#00000000001.26.6", 2023-10-17T18:45:07.447 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.85525403851221246", 2023-10-17T18:45:07.447 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:45:07.447 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "a", 2023-10-17T18:45:07.447 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:45:07.447 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:45:07.448 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:45:07.448 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.729680197Z", 2023-10-17T18:45:07.448 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:45:07.448 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:45:07.448 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:45:07.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.449 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:45:07.449 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:45:07.449 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:45:07.449 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:45:07.450 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:45:07.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.450 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T18:45:07.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.450 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "5#00000000001.18.6", 2023-10-17T18:45:07.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.13758827529464904184", 2023-10-17T18:45:07.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "d", 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.863031096Z", 2023-10-17T18:45:07.452 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:45:07.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:45:07.453 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 1 2023-10-17T18:45:07.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.453 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:45:07.453 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:45:07.454 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:45:07.454 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:45:07.454 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:45:07.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.454 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T18:45:07.455 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.455 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.455 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "8#00000000001.49.6", 2023-10-17T18:45:07.456 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2339600626480474804", 2023-10-17T18:45:07.456 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:45:07.456 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "c", 2023-10-17T18:45:07.456 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:45:07.456 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.806992640Z", 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:45:07.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.458 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:45:07.458 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:45:07.458 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:45:07.458 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.460 INFO:teuthology.orchestra.run.smithi049.stdout: "op_id": "10#00000000001.110.6", 2023-10-17T18:45:07.460 INFO:teuthology.orchestra.run.smithi049.stdout: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.7544705034949184787", 2023-10-17T18:45:07.460 INFO:teuthology.orchestra.run.smithi049.stdout: "op": "write", 2023-10-17T18:45:07.460 INFO:teuthology.orchestra.run.smithi049.stdout: "object": "b", 2023-10-17T18:45:07.461 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:45:07.461 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "complete", 2023-10-17T18:45:07.461 INFO:teuthology.orchestra.run.smithi049.stdout: "index_ver": 1, 2023-10-17T18:45:07.461 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:44:35.788417331Z", 2023-10-17T18:45:07.461 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:45:07.462 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:45:07.462 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:45:07.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.462 INFO:teuthology.orchestra.run.smithi049.stdout: "bilog_flags": 0, 2023-10-17T18:45:07.462 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned": false, 2023-10-17T18:45:07.463 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "", 2023-10-17T18:45:07.463 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "", 2023-10-17T18:45:07.463 INFO:teuthology.orchestra.run.smithi049.stdout: "zones_trace": [ 2023-10-17T18:45:07.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.463 INFO:teuthology.orchestra.run.smithi049.stdout: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T18:45:07.464 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.464 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.464 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.464 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:45:07.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.466 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:45:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.518 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.556 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:45:07.556 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-55 2023-10-17T18:45:07.560 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 4 2023-10-17T18:45:07.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:45:07.597 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:45:07.644 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.644 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.686 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:45:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.686 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding": "None", 2023-10-17T18:45:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "current_index": { 2023-10-17T18:45:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:45:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "Normal", 2023-10-17T18:45:07.687 INFO:teuthology.orchestra.run.smithi049.stdout: "normal": { 2023-10-17T18:45:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:45:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:45:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.688 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "logs": [ 2023-10-17T18:45:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.689 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:45:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:45:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:45:07.690 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:45:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 11, 2023-10-17T18:45:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:45:07.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:45:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:45:07.692 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:45:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:45:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:45:07.693 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:45:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:45:07.694 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:45:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:45:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:45:07.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:45:07.696 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:45:07.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:45:07.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:45:07.699 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:45:07.751 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.751 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.794 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2023-10-17T18:45:07.796 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.796 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.797 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.797 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:07.798 DEBUG:boto:Data: 2023-10-17T18:45:07.798 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.799 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.799 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.799 DEBUG:boto:Params: {} 2023-10-17T18:45:07.800 DEBUG:boto:Token: None 2023-10-17T18:45:07.800 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.801 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T18:45:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T18:45:07.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000812648bb6c4b0711-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.804 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:45:07.805 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:45:07.805 DEBUG:boto:Method: PUT 2023-10-17T18:45:07.805 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:45:07.806 DEBUG:boto:Data: 2023-10-17T18:45:07.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:45:07.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.807 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.807 DEBUG:boto:Params: {} 2023-10-17T18:45:07.808 DEBUG:boto:Token: None 2023-10-17T18:45:07.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/e 2023-10-17T18:45:07.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gIWC+zvbjVA2FD5bpzcSuZqfiRU= 2023-10-17T18:45:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gIWC+zvbjVA2FD5bpzcSuZqfiRU='} 2023-10-17T18:45:07.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000492895b46a57f550-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.815 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.815 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.816 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.816 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:07.817 DEBUG:boto:Data: 2023-10-17T18:45:07.817 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.818 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.818 DEBUG:boto:Params: {} 2023-10-17T18:45:07.819 DEBUG:boto:Token: None 2023-10-17T18:45:07.819 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T18:45:07.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T18:45:07.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d93c3e409f49b7b-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.823 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:45:07.823 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:45:07.824 DEBUG:boto:Method: PUT 2023-10-17T18:45:07.824 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:45:07.825 DEBUG:boto:Data: 2023-10-17T18:45:07.825 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:45:07.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.827 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.827 DEBUG:boto:Params: {} 2023-10-17T18:45:07.828 DEBUG:boto:Token: None 2023-10-17T18:45:07.828 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/f 2023-10-17T18:45:07.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BL7UlQ0y5x/sUMnEem7F4jaWFh8= 2023-10-17T18:45:07.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BL7UlQ0y5x/sUMnEem7F4jaWFh8='} 2023-10-17T18:45:07.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eac6ee2f2966a98a-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.871 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.871 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.872 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.872 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:07.872 DEBUG:boto:Data: 2023-10-17T18:45:07.873 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.873 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.873 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.874 DEBUG:boto:Params: {} 2023-10-17T18:45:07.874 DEBUG:boto:Token: None 2023-10-17T18:45:07.875 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T18:45:07.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T18:45:07.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008464733dac03985-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.878 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:45:07.878 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:45:07.879 DEBUG:boto:Method: PUT 2023-10-17T18:45:07.879 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:45:07.880 DEBUG:boto:Data: 2023-10-17T18:45:07.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:45:07.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.881 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.881 DEBUG:boto:Params: {} 2023-10-17T18:45:07.881 DEBUG:boto:Token: None 2023-10-17T18:45:07.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/g 2023-10-17T18:45:07.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9zTdUFeZlwPCQ84Jhxf1k64xfDI= 2023-10-17T18:45:07.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9zTdUFeZlwPCQ84Jhxf1k64xfDI='} 2023-10-17T18:45:07.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de8bbcc255ff8c7c-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.888 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:45:07.888 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:45:07.889 DEBUG:boto:Method: HEAD 2023-10-17T18:45:07.889 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:45:07.889 DEBUG:boto:Data: 2023-10-17T18:45:07.890 DEBUG:boto:Headers: {} 2023-10-17T18:45:07.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.890 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.891 DEBUG:boto:Params: {} 2023-10-17T18:45:07.891 DEBUG:boto:Token: None 2023-10-17T18:45:07.892 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/ 2023-10-17T18:45:07.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T18:45:07.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T18:45:07.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000415d6192aded757d-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.895 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:45:07.895 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:45:07.895 DEBUG:boto:Method: PUT 2023-10-17T18:45:07.896 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:45:07.896 DEBUG:boto:Data: 2023-10-17T18:45:07.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:45:07.897 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:45:07.897 DEBUG:boto:Port: 8000 2023-10-17T18:45:07.897 DEBUG:boto:Params: {} 2023-10-17T18:45:07.898 DEBUG:boto:Token: None 2023-10-17T18:45:07.898 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:45:07 GMT /hbmuog-55/h 2023-10-17T18:45:07.899 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lkG1puv499lpV+VoKJQCZp24aSg= 2023-10-17T18:45:07.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lkG1puv499lpV+VoKJQCZp24aSg='} 2023-10-17T18:45:07.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026b748dbfb444e9a-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:45:07.903 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T18:45:07.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:45:07.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:45:07.904 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:45:07.962 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:45:07.962 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:45:07.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:07.995+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:45:12.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:12.996+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:45:17.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:17.997+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:45:22.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:22.999+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:22.999 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:23.000 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:28.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:27.999+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:28.000 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:28.000 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:33.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:33.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:33.001 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:33.001 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:38.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:38.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:38.002 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:38.002 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:43.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:43.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:43.003 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:43.003 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:48.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:48.001+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:48.003 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:48.003 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:53.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:53.002+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:53.004 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:53.004 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:45:58.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:45:58.004+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:45:58.005 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , ] 2023-10-17T18:45:58.005 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:03.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:03.005+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:08.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:08.007 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:08.007 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:13.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:13.007 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:13.008 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:18.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:18.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:18.008 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:18.008 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:23.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:23.007+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:23.009 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:23.010 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:28.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:28.008+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:28.010 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:28.011 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:33.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:33.009+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:33.011 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:33.011 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:38.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:38.011+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:38.012 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:38.012 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:43.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:43.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:43.013 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:43.013 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:48.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:48.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:48.014 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:48.014 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:53.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:53.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:53.014 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:53.015 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:46:58.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:46:58.014 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:46:58.014 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:03.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:03.013+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:03.015 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:03.015 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:08.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:08.014+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:08.016 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:08.016 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:13.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:13.016+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:13.017 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:13.017 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:18.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:18.017+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:18.018 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:18.018 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:23.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:23.018+0000 7f6a2dc95780 1 bucket sync caught up with source: 2023-10-17T18:47:23.019 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:23.019 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T18:47:23.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:23.018+0000 7f6a2dc95780 0 bucket checkpoint complete 2023-10-17T18:47:23.036 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T18:47:23.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:23.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:23.038 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:47:23.103 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:23.103 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:23.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:23.140+0000 7feb5d21b780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:23.143 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000001.406.6] 2023-10-17T18:47:23.143 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:28.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:28.142+0000 7feb5d21b780 1 waiting for incremental sync to catch up: 2023-10-17T18:47:28.144 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000001.406.6] 2023-10-17T18:47:28.144 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:33.143+0000 7feb5d21b780 1 bucket sync caught up with source: 2023-10-17T18:47:33.145 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:33.145 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:33.143+0000 7feb5d21b780 0 bucket checkpoint complete 2023-10-17T18:47:33.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:47:33.163 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.163 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.163 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.164 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:47:33.164 DEBUG:boto:Data: 2023-10-17T18:47:33.165 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.165 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.165 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.166 DEBUG:boto:Params: {} 2023-10-17T18:47:33.166 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:47:33.167 DEBUG:boto:Token: None 2023-10-17T18:47:33.167 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.168 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg= 2023-10-17T18:47:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg=', 'Content-Length': '0'} 2023-10-17T18:47:33.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6bdfc182aee8348-00652ed6c5-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.175 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.176 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.176 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.176 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:47:33.177 DEBUG:boto:Data: 2023-10-17T18:47:33.177 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.177 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.178 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.178 DEBUG:boto:Params: {} 2023-10-17T18:47:33.179 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:47:33.179 DEBUG:boto:Token: None 2023-10-17T18:47:33.180 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg= 2023-10-17T18:47:33.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg=', 'Content-Length': '0'} 2023-10-17T18:47:33.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c855944fb3c8909-00652ed6c5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:47:33.187 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.187 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.188 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.188 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.189 DEBUG:boto:Method: GET 2023-10-17T18:47:33.189 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.189 DEBUG:boto:Data: 2023-10-17T18:47:33.190 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.191 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.191 DEBUG:boto:Params: {} 2023-10-17T18:47:33.192 DEBUG:boto:Token: None 2023-10-17T18:47:33.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/?versions 2023-10-17T18:47:33.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T18:47:33.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T18:47:33.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f18734ba029419-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.197 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.198 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:47:33.198 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:47:33.199 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:47:33.199 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:47:33.200 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:47:33.200 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:47:33.200 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:47:33.201 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:47:33.201 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:47:33.202 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.202 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.203 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.203 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.203 DEBUG:boto:Method: GET 2023-10-17T18:47:33.204 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.204 DEBUG:boto:Data: 2023-10-17T18:47:33.205 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.205 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.206 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.206 DEBUG:boto:Params: {} 2023-10-17T18:47:33.207 DEBUG:boto:Token: None 2023-10-17T18:47:33.207 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/?versions 2023-10-17T18:47:33.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T18:47:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T18:47:33.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0acc9a7dd25b2ea-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.211 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.212 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:47:33.212 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:47:33.213 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:47:33.213 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:47:33.213 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:47:33.214 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:47:33.214 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:47:33.214 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:47:33.215 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.215 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.216 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.216 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.217 DEBUG:boto:Method: GET 2023-10-17T18:47:33.218 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.218 DEBUG:boto:Data: 2023-10-17T18:47:33.219 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.220 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.220 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.221 DEBUG:boto:Params: {} 2023-10-17T18:47:33.222 DEBUG:boto:Token: None 2023-10-17T18:47:33.222 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/?versions 2023-10-17T18:47:33.223 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T18:47:33.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T18:47:33.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047855ffaecac7f83-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.227 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.228 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.228 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.229 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.229 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.230 DEBUG:boto:Method: GET 2023-10-17T18:47:33.230 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:33.231 DEBUG:boto:Data: 2023-10-17T18:47:33.231 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.232 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.232 DEBUG:boto:Params: {} 2023-10-17T18:47:33.233 DEBUG:boto:Token: None 2023-10-17T18:47:33.234 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/?versions 2023-10-17T18:47:33.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T18:47:33.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T18:47:33.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1755ba2da10f34-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.237 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:33.239 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.239 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.240 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.240 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.240 DEBUG:boto:Method: GET 2023-10-17T18:47:33.241 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.241 DEBUG:boto:Data: 2023-10-17T18:47:33.242 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.243 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.243 DEBUG:boto:Params: {} 2023-10-17T18:47:33.243 DEBUG:boto:Token: None 2023-10-17T18:47:33.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T18:47:33.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T18:47:33.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076eb605c525af276-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.247 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.248 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.248 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.249 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.249 DEBUG:boto:Method: GET 2023-10-17T18:47:33.249 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.250 DEBUG:boto:Data: 2023-10-17T18:47:33.250 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.251 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.252 DEBUG:boto:Params: {} 2023-10-17T18:47:33.252 DEBUG:boto:Token: None 2023-10-17T18:47:33.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T18:47:33.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T18:47:33.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b175c7a49137a644-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.256 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.257 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.257 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.258 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.258 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.258 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.259 DEBUG:boto:Data: 2023-10-17T18:47:33.259 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.260 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.260 DEBUG:boto:Params: {} 2023-10-17T18:47:33.261 DEBUG:boto:Token: None 2023-10-17T18:47:33.261 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.262 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8= 2023-10-17T18:47:33.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8=', 'Content-Length': '0'} 2023-10-17T18:47:33.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d0b45db2a3c5b3-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.265 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.265 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.265 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.266 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.266 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.267 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.267 DEBUG:boto:Data: 2023-10-17T18:47:33.267 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.268 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.268 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.268 DEBUG:boto:Params: {} 2023-10-17T18:47:33.269 DEBUG:boto:Token: None 2023-10-17T18:47:33.269 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.270 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8= 2023-10-17T18:47:33.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8=', 'Content-Length': '0'} 2023-10-17T18:47:33.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c437ae2d36565e9-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:33.273 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.274 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.274 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.274 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.275 DEBUG:boto:Method: GET 2023-10-17T18:47:33.275 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.276 DEBUG:boto:Data: 2023-10-17T18:47:33.276 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.276 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.277 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.277 DEBUG:boto:Params: {} 2023-10-17T18:47:33.278 DEBUG:boto:Token: None 2023-10-17T18:47:33.278 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.279 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T18:47:33.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T18:47:33.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac510b6248432e44-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.322 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.323 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.323 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.323 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:33.324 DEBUG:boto:Method: GET 2023-10-17T18:47:33.324 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.325 DEBUG:boto:Data: 2023-10-17T18:47:33.325 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.326 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.326 DEBUG:boto:Params: {} 2023-10-17T18:47:33.327 DEBUG:boto:Token: None 2023-10-17T18:47:33.327 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:33.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T18:47:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T18:47:33.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b707310a962f5576-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.333 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.333 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.334 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.334 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.335 DEBUG:boto:Method: GET 2023-10-17T18:47:33.335 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:33.335 DEBUG:boto:Data: 2023-10-17T18:47:33.336 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.337 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.337 DEBUG:boto:Params: {} 2023-10-17T18:47:33.337 DEBUG:boto:Token: None 2023-10-17T18:47:33.338 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d0f9941e213595-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.342 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.342 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.343 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.343 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.344 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.344 DEBUG:boto:Method: GET 2023-10-17T18:47:33.345 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:33.345 DEBUG:boto:Data: 2023-10-17T18:47:33.345 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.346 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.346 DEBUG:boto:Params: {} 2023-10-17T18:47:33.347 DEBUG:boto:Token: None 2023-10-17T18:47:33.347 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011de529f6e8bd495-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.351 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.351 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.352 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.352 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.353 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:33.353 DEBUG:boto:Data: 2023-10-17T18:47:33.353 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.354 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.354 DEBUG:boto:Params: {} 2023-10-17T18:47:33.355 DEBUG:boto:Token: None 2023-10-17T18:47:33.355 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a 2023-10-17T18:47:33.356 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU= 2023-10-17T18:47:33.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU=', 'Content-Length': '0'} 2023-10-17T18:47:33.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d1f4c3c0f42355-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.358 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.359 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.359 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.360 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:33.360 DEBUG:boto:Data: 2023-10-17T18:47:33.360 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.361 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.361 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.361 DEBUG:boto:Params: {} 2023-10-17T18:47:33.362 DEBUG:boto:Token: None 2023-10-17T18:47:33.362 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a 2023-10-17T18:47:33.363 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU= 2023-10-17T18:47:33.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU=', 'Content-Length': '0'} 2023-10-17T18:47:33.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb5d539acac5fdb7-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:33.366 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.366 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.367 DEBUG:boto:Method: GET 2023-10-17T18:47:33.367 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:33.368 DEBUG:boto:Data: 2023-10-17T18:47:33.368 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.368 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.369 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.369 DEBUG:boto:Params: {} 2023-10-17T18:47:33.370 DEBUG:boto:Token: None 2023-10-17T18:47:33.370 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a 2023-10-17T18:47:33.371 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM= 2023-10-17T18:47:33.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM=', 'Content-Length': '0'} 2023-10-17T18:47:33.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1e4574344cc9c5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.373 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:33.374 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:33.374 DEBUG:boto:Method: GET 2023-10-17T18:47:33.375 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:33.375 DEBUG:boto:Data: 2023-10-17T18:47:33.375 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.376 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.376 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.377 DEBUG:boto:Params: {} 2023-10-17T18:47:33.377 DEBUG:boto:Token: None 2023-10-17T18:47:33.377 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/a 2023-10-17T18:47:33.378 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM= 2023-10-17T18:47:33.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM=', 'Content-Length': '0'} 2023-10-17T18:47:33.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091084b3aaae70eb7-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.426 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.427 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.427 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.428 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.428 DEBUG:boto:Method: GET 2023-10-17T18:47:33.428 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:33.429 DEBUG:boto:Data: 2023-10-17T18:47:33.429 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.430 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.430 DEBUG:boto:Params: {} 2023-10-17T18:47:33.431 DEBUG:boto:Token: None 2023-10-17T18:47:33.431 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.432 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53847c0aeff47dd-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.435 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.435 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.436 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.436 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.437 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:33.437 DEBUG:boto:Method: GET 2023-10-17T18:47:33.438 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:33.438 DEBUG:boto:Data: 2023-10-17T18:47:33.438 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.439 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.440 DEBUG:boto:Params: {} 2023-10-17T18:47:33.441 DEBUG:boto:Token: None 2023-10-17T18:47:33.441 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085dd4d9b70b1564d-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.446 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:33.447 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.448 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.448 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.448 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.449 DEBUG:boto:Method: GET 2023-10-17T18:47:33.449 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.450 DEBUG:boto:Data: 2023-10-17T18:47:33.450 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.451 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.452 DEBUG:boto:Params: {} 2023-10-17T18:47:33.452 DEBUG:boto:Token: None 2023-10-17T18:47:33.453 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T18:47:33.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T18:47:33.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2370ff5913d61b-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.457 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.457 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.458 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.458 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.459 DEBUG:boto:Method: GET 2023-10-17T18:47:33.459 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.460 DEBUG:boto:Data: 2023-10-17T18:47:33.460 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.461 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.461 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.462 DEBUG:boto:Params: {} 2023-10-17T18:47:33.462 DEBUG:boto:Token: None 2023-10-17T18:47:33.463 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.463 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T18:47:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T18:47:33.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9274b1000ea7d8-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.467 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.467 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.468 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.468 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.469 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.469 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.470 DEBUG:boto:Data: 2023-10-17T18:47:33.470 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.471 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.471 DEBUG:boto:Params: {} 2023-10-17T18:47:33.472 DEBUG:boto:Token: None 2023-10-17T18:47:33.472 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU= 2023-10-17T18:47:33.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU=', 'Content-Length': '0'} 2023-10-17T18:47:33.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27f7678727afead-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.476 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.477 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.477 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.477 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.478 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.478 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.479 DEBUG:boto:Data: 2023-10-17T18:47:33.479 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.481 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.481 DEBUG:boto:Params: {} 2023-10-17T18:47:33.482 DEBUG:boto:Token: None 2023-10-17T18:47:33.483 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.484 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU= 2023-10-17T18:47:33.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU=', 'Content-Length': '0'} 2023-10-17T18:47:33.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8aaefa95a26e27-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:33.488 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.489 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.489 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.490 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.490 DEBUG:boto:Method: GET 2023-10-17T18:47:33.491 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.491 DEBUG:boto:Data: 2023-10-17T18:47:33.492 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.493 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.494 DEBUG:boto:Params: {} 2023-10-17T18:47:33.494 DEBUG:boto:Token: None 2023-10-17T18:47:33.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.495 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T18:47:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T18:47:33.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bebf81f522e35e5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.498 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.499 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.499 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.500 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:33.500 DEBUG:boto:Method: GET 2023-10-17T18:47:33.501 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.501 DEBUG:boto:Data: 2023-10-17T18:47:33.502 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.503 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.503 DEBUG:boto:Params: {} 2023-10-17T18:47:33.504 DEBUG:boto:Token: None 2023-10-17T18:47:33.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:33.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T18:47:33.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T18:47:33.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015888f9a9af5e9e5-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.508 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.509 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.509 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.510 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.510 DEBUG:boto:Method: GET 2023-10-17T18:47:33.511 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:33.511 DEBUG:boto:Data: 2023-10-17T18:47:33.511 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.512 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.513 DEBUG:boto:Params: {} 2023-10-17T18:47:33.513 DEBUG:boto:Token: None 2023-10-17T18:47:33.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf19f9a728460431-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.518 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.519 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.519 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.520 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.520 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.520 DEBUG:boto:Method: GET 2023-10-17T18:47:33.521 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:33.521 DEBUG:boto:Data: 2023-10-17T18:47:33.522 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.523 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.523 DEBUG:boto:Params: {} 2023-10-17T18:47:33.523 DEBUG:boto:Token: None 2023-10-17T18:47:33.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e918e60c9eeea658-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.570 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.571 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.572 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.572 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.573 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:33.573 DEBUG:boto:Data: 2023-10-17T18:47:33.573 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.574 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.574 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.575 DEBUG:boto:Params: {} 2023-10-17T18:47:33.575 DEBUG:boto:Token: None 2023-10-17T18:47:33.576 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b 2023-10-17T18:47:33.576 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08= 2023-10-17T18:47:33.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08=', 'Content-Length': '0'} 2023-10-17T18:47:33.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943570052fc795a8-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.579 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.580 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.580 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.581 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:33.581 DEBUG:boto:Data: 2023-10-17T18:47:33.582 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.582 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.583 DEBUG:boto:Params: {} 2023-10-17T18:47:33.583 DEBUG:boto:Token: None 2023-10-17T18:47:33.584 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b 2023-10-17T18:47:33.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08= 2023-10-17T18:47:33.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08=', 'Content-Length': '0'} 2023-10-17T18:47:33.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267296fec0648bcc-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:33.588 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.588 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.589 DEBUG:boto:Method: GET 2023-10-17T18:47:33.590 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:33.590 DEBUG:boto:Data: 2023-10-17T18:47:33.590 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.591 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.592 DEBUG:boto:Params: {} 2023-10-17T18:47:33.592 DEBUG:boto:Token: None 2023-10-17T18:47:33.593 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b 2023-10-17T18:47:33.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g= 2023-10-17T18:47:33.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g=', 'Content-Length': '0'} 2023-10-17T18:47:33.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b576a554aae910e5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.638 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:33.639 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:33.639 DEBUG:boto:Method: GET 2023-10-17T18:47:33.640 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:33.640 DEBUG:boto:Data: 2023-10-17T18:47:33.641 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.642 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.642 DEBUG:boto:Params: {} 2023-10-17T18:47:33.643 DEBUG:boto:Token: None 2023-10-17T18:47:33.643 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/b 2023-10-17T18:47:33.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g= 2023-10-17T18:47:33.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g=', 'Content-Length': '0'} 2023-10-17T18:47:33.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd70d2889e5c182c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.647 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.648 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.648 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.649 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.649 DEBUG:boto:Method: GET 2023-10-17T18:47:33.650 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:33.650 DEBUG:boto:Data: 2023-10-17T18:47:33.651 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.652 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.652 DEBUG:boto:Params: {} 2023-10-17T18:47:33.653 DEBUG:boto:Token: None 2023-10-17T18:47:33.653 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.654 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c68dd638f9a273a1-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.658 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.658 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.659 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.659 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.660 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:33.660 DEBUG:boto:Method: GET 2023-10-17T18:47:33.660 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:33.661 DEBUG:boto:Data: 2023-10-17T18:47:33.662 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.663 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.663 DEBUG:boto:Params: {} 2023-10-17T18:47:33.664 DEBUG:boto:Token: None 2023-10-17T18:47:33.664 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.665 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72178a3106b1126-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.668 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:33.669 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.670 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.670 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.671 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.671 DEBUG:boto:Method: GET 2023-10-17T18:47:33.672 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.672 DEBUG:boto:Data: 2023-10-17T18:47:33.673 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.674 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.674 DEBUG:boto:Params: {} 2023-10-17T18:47:33.675 DEBUG:boto:Token: None 2023-10-17T18:47:33.675 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T18:47:33.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T18:47:33.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf007bf915b97f6-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.680 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.680 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.681 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.682 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.682 DEBUG:boto:Method: GET 2023-10-17T18:47:33.683 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.684 DEBUG:boto:Data: 2023-10-17T18:47:33.685 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.685 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.686 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.687 DEBUG:boto:Params: {} 2023-10-17T18:47:33.688 DEBUG:boto:Token: None 2023-10-17T18:47:33.688 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.689 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T18:47:33.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T18:47:33.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8962e3cb529a16-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.734 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.735 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.735 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.736 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.736 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.737 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.737 DEBUG:boto:Data: 2023-10-17T18:47:33.737 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.738 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.739 DEBUG:boto:Params: {} 2023-10-17T18:47:33.739 DEBUG:boto:Token: None 2023-10-17T18:47:33.740 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY= 2023-10-17T18:47:33.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY=', 'Content-Length': '0'} 2023-10-17T18:47:33.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f08c74f7687acf6-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.747 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.747 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.748 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.748 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.749 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.749 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.750 DEBUG:boto:Data: 2023-10-17T18:47:33.750 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.751 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.751 DEBUG:boto:Params: {} 2023-10-17T18:47:33.752 DEBUG:boto:Token: None 2023-10-17T18:47:33.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY= 2023-10-17T18:47:33.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY=', 'Content-Length': '0'} 2023-10-17T18:47:33.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090426f443226552b-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:33.756 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.757 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.757 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.757 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.758 DEBUG:boto:Method: GET 2023-10-17T18:47:33.758 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.759 DEBUG:boto:Data: 2023-10-17T18:47:33.759 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.760 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.760 DEBUG:boto:Params: {} 2023-10-17T18:47:33.761 DEBUG:boto:Token: None 2023-10-17T18:47:33.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T18:47:33.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T18:47:33.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eede1366b02c119-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.765 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.765 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.766 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.766 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:33.767 DEBUG:boto:Method: GET 2023-10-17T18:47:33.767 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.768 DEBUG:boto:Data: 2023-10-17T18:47:33.768 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.769 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.769 DEBUG:boto:Params: {} 2023-10-17T18:47:33.770 DEBUG:boto:Token: None 2023-10-17T18:47:33.770 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:33.771 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T18:47:33.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T18:47:33.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8b3feb5ff7652c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.774 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.774 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.775 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.775 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.776 DEBUG:boto:Method: GET 2023-10-17T18:47:33.776 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:33.777 DEBUG:boto:Data: 2023-10-17T18:47:33.777 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.777 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.778 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.778 DEBUG:boto:Params: {} 2023-10-17T18:47:33.779 DEBUG:boto:Token: None 2023-10-17T18:47:33.779 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.780 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4a61375342e18e-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.783 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.784 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.784 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.785 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.785 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.786 DEBUG:boto:Method: GET 2023-10-17T18:47:33.786 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:33.786 DEBUG:boto:Data: 2023-10-17T18:47:33.787 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.787 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.788 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.788 DEBUG:boto:Params: {} 2023-10-17T18:47:33.789 DEBUG:boto:Token: None 2023-10-17T18:47:33.789 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001738618c10372fef-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.793 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.794 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.794 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.794 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.795 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:33.795 DEBUG:boto:Data: 2023-10-17T18:47:33.796 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.796 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.797 DEBUG:boto:Params: {} 2023-10-17T18:47:33.797 DEBUG:boto:Token: None 2023-10-17T18:47:33.798 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c 2023-10-17T18:47:33.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE= 2023-10-17T18:47:33.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE=', 'Content-Length': '0'} 2023-10-17T18:47:33.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d325ad86d0b3a718-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.801 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.802 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.802 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.803 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:33.803 DEBUG:boto:Data: 2023-10-17T18:47:33.804 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.804 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.804 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.805 DEBUG:boto:Params: {} 2023-10-17T18:47:33.805 DEBUG:boto:Token: None 2023-10-17T18:47:33.806 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c 2023-10-17T18:47:33.806 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE= 2023-10-17T18:47:33.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE=', 'Content-Length': '0'} 2023-10-17T18:47:33.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670b639f1d46dae6-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:33.810 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.810 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.810 DEBUG:boto:Method: GET 2023-10-17T18:47:33.811 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:33.811 DEBUG:boto:Data: 2023-10-17T18:47:33.812 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.813 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.813 DEBUG:boto:Params: {} 2023-10-17T18:47:33.813 DEBUG:boto:Token: None 2023-10-17T18:47:33.814 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c 2023-10-17T18:47:33.814 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo= 2023-10-17T18:47:33.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo=', 'Content-Length': '0'} 2023-10-17T18:47:33.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024861dca3003a326-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.858 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:33.859 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:33.860 DEBUG:boto:Method: GET 2023-10-17T18:47:33.860 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:33.860 DEBUG:boto:Data: 2023-10-17T18:47:33.861 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.861 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.862 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.862 DEBUG:boto:Params: {} 2023-10-17T18:47:33.863 DEBUG:boto:Token: None 2023-10-17T18:47:33.863 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/c 2023-10-17T18:47:33.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo= 2023-10-17T18:47:33.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo=', 'Content-Length': '0'} 2023-10-17T18:47:33.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537f119fa8c112d8-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.867 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.868 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.868 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.869 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.869 DEBUG:boto:Method: GET 2023-10-17T18:47:33.869 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:33.870 DEBUG:boto:Data: 2023-10-17T18:47:33.870 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.871 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.872 DEBUG:boto:Params: {} 2023-10-17T18:47:33.872 DEBUG:boto:Token: None 2023-10-17T18:47:33.873 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbcd0b3fd60a6209-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.877 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.878 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.878 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.879 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.879 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:33.879 DEBUG:boto:Method: GET 2023-10-17T18:47:33.880 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:33.880 DEBUG:boto:Data: 2023-10-17T18:47:33.881 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.881 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.882 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.882 DEBUG:boto:Params: {} 2023-10-17T18:47:33.883 DEBUG:boto:Token: None 2023-10-17T18:47:33.883 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000552fc3d011067be-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.930 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:33.932 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.932 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.933 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.933 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.933 DEBUG:boto:Method: GET 2023-10-17T18:47:33.934 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.934 DEBUG:boto:Data: 2023-10-17T18:47:33.935 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.935 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.936 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.936 DEBUG:boto:Params: {} 2023-10-17T18:47:33.937 DEBUG:boto:Token: None 2023-10-17T18:47:33.937 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.938 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T18:47:33.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T18:47:33.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f78a1cc33cd9c5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.941 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.942 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.942 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.942 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.943 DEBUG:boto:Method: GET 2023-10-17T18:47:33.943 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.944 DEBUG:boto:Data: 2023-10-17T18:47:33.944 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.945 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.945 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.945 DEBUG:boto:Params: {} 2023-10-17T18:47:33.946 DEBUG:boto:Token: None 2023-10-17T18:47:33.947 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T18:47:33.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T18:47:33.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee5a0a86f970b8c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.953 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.953 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.954 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.954 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.954 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.955 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.955 DEBUG:boto:Data: 2023-10-17T18:47:33.956 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.956 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.956 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.957 DEBUG:boto:Params: {} 2023-10-17T18:47:33.957 DEBUG:boto:Token: None 2023-10-17T18:47:33.958 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.958 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o= 2023-10-17T18:47:33.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o=', 'Content-Length': '0'} 2023-10-17T18:47:33.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3e258a1530917c-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.961 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.961 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.962 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.962 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.963 DEBUG:boto:Method: HEAD 2023-10-17T18:47:33.963 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.963 DEBUG:boto:Data: 2023-10-17T18:47:33.964 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.964 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.965 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.965 DEBUG:boto:Params: {} 2023-10-17T18:47:33.965 DEBUG:boto:Token: None 2023-10-17T18:47:33.966 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.966 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o= 2023-10-17T18:47:33.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o=', 'Content-Length': '0'} 2023-10-17T18:47:33.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17526dd1ec40c47-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:33.970 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.970 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.971 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.971 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.971 DEBUG:boto:Method: GET 2023-10-17T18:47:33.972 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.972 DEBUG:boto:Data: 2023-10-17T18:47:33.973 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.973 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.974 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.974 DEBUG:boto:Params: {} 2023-10-17T18:47:33.974 DEBUG:boto:Token: None 2023-10-17T18:47:33.975 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.975 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T18:47:33.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T18:47:33.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b53d185aa743b8b-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.979 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:33.979 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:33.980 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.980 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:33.980 DEBUG:boto:Method: GET 2023-10-17T18:47:33.981 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.981 DEBUG:boto:Data: 2023-10-17T18:47:33.982 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:33.983 DEBUG:boto:Port: 8000 2023-10-17T18:47:33.983 DEBUG:boto:Params: {} 2023-10-17T18:47:33.984 DEBUG:boto:Token: None 2023-10-17T18:47:33.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:33.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T18:47:33.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T18:47:33.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe57aa79e1978c4a-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.988 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.988 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.989 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:33.989 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:33.989 DEBUG:boto:Method: GET 2023-10-17T18:47:33.990 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:33.990 DEBUG:boto:Data: 2023-10-17T18:47:33.991 DEBUG:boto:Headers: {} 2023-10-17T18:47:33.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:33.992 DEBUG:boto:Port: 8001 2023-10-17T18:47:33.992 DEBUG:boto:Params: {} 2023-10-17T18:47:33.993 DEBUG:boto:Token: None 2023-10-17T18:47:33.993 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:33 GMT /hbmuog-55/ 2023-10-17T18:47:33.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T18:47:33.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T18:47:33.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbbef0839a84755-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:33.997 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:33.998 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:33.999 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:33.999 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:33.999 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:34.000 DEBUG:boto:Method: GET 2023-10-17T18:47:34.000 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:34.001 DEBUG:boto:Data: 2023-10-17T18:47:34.001 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.002 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.002 DEBUG:boto:Params: {} 2023-10-17T18:47:34.003 DEBUG:boto:Token: None 2023-10-17T18:47:34.003 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b72d3a0d17cfdc1-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.007 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.007 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:34.008 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:34.008 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.009 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:34.009 DEBUG:boto:Data: 2023-10-17T18:47:34.010 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.010 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.010 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.011 DEBUG:boto:Params: {} 2023-10-17T18:47:34.011 DEBUG:boto:Token: None 2023-10-17T18:47:34.012 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/d 2023-10-17T18:47:34.012 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk= 2023-10-17T18:47:34.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk=', 'Content-Length': '0'} 2023-10-17T18:47:34.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba205cc85cf0942-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.015 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:34.015 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:34.016 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.016 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:34.017 DEBUG:boto:Data: 2023-10-17T18:47:34.017 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.018 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.018 DEBUG:boto:Params: {} 2023-10-17T18:47:34.019 DEBUG:boto:Token: None 2023-10-17T18:47:34.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/d 2023-10-17T18:47:34.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk= 2023-10-17T18:47:34.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk=', 'Content-Length': '0'} 2023-10-17T18:47:34.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a0718a1c7b16b4-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:34.023 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:34.024 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:34.024 DEBUG:boto:Method: GET 2023-10-17T18:47:34.025 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:34.025 DEBUG:boto:Data: 2023-10-17T18:47:34.026 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.027 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.027 DEBUG:boto:Params: {} 2023-10-17T18:47:34.027 DEBUG:boto:Token: None 2023-10-17T18:47:34.028 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/d 2023-10-17T18:47:34.028 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc= 2023-10-17T18:47:34.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc=', 'Content-Length': '0'} 2023-10-17T18:47:34.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180fe22a15533878-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.031 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:34.032 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:34.032 DEBUG:boto:Method: GET 2023-10-17T18:47:34.033 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:34.033 DEBUG:boto:Data: 2023-10-17T18:47:34.034 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.034 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.035 DEBUG:boto:Params: {} 2023-10-17T18:47:34.035 DEBUG:boto:Token: None 2023-10-17T18:47:34.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/d 2023-10-17T18:47:34.036 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc= 2023-10-17T18:47:34.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc=', 'Content-Length': '0'} 2023-10-17T18:47:34.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bad0e4eae859d1-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.082 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.083 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.083 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:34.084 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:34.084 DEBUG:boto:Method: GET 2023-10-17T18:47:34.085 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:34.085 DEBUG:boto:Data: 2023-10-17T18:47:34.086 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.086 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.087 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.087 DEBUG:boto:Params: {} 2023-10-17T18:47:34.088 DEBUG:boto:Token: None 2023-10-17T18:47:34.088 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.089 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eee05fecddf5c8ab-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.092 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.093 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.093 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.094 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:34.094 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:34.095 DEBUG:boto:Method: GET 2023-10-17T18:47:34.095 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:34.095 DEBUG:boto:Data: 2023-10-17T18:47:34.096 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.097 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.097 DEBUG:boto:Params: {} 2023-10-17T18:47:34.098 DEBUG:boto:Token: None 2023-10-17T18:47:34.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a714135991b71cf0-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.102 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:34.103 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.104 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.104 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.105 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.105 DEBUG:boto:Method: GET 2023-10-17T18:47:34.105 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.106 DEBUG:boto:Data: 2023-10-17T18:47:34.106 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.107 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.107 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.108 DEBUG:boto:Params: {} 2023-10-17T18:47:34.108 DEBUG:boto:Token: None 2023-10-17T18:47:34.109 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T18:47:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T18:47:34.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004722b81b5441e24-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.154 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.155 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.155 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.156 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.156 DEBUG:boto:Method: GET 2023-10-17T18:47:34.157 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.157 DEBUG:boto:Data: 2023-10-17T18:47:34.158 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.158 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.158 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.159 DEBUG:boto:Params: {} 2023-10-17T18:47:34.159 DEBUG:boto:Token: None 2023-10-17T18:47:34.160 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.161 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T18:47:34.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T18:47:34.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223709468da86845-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.164 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.164 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.165 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.165 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.166 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.166 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.167 DEBUG:boto:Data: 2023-10-17T18:47:34.167 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.167 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.168 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.168 DEBUG:boto:Params: {} 2023-10-17T18:47:34.169 DEBUG:boto:Token: None 2023-10-17T18:47:34.169 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.170 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs= 2023-10-17T18:47:34.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs=', 'Content-Length': '0'} 2023-10-17T18:47:34.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb5fdf4f7939e32-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.173 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.173 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.174 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.174 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.175 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.175 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.175 DEBUG:boto:Data: 2023-10-17T18:47:34.176 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.176 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.177 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.177 DEBUG:boto:Params: {} 2023-10-17T18:47:34.177 DEBUG:boto:Token: None 2023-10-17T18:47:34.178 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.178 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs= 2023-10-17T18:47:34.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs=', 'Content-Length': '0'} 2023-10-17T18:47:34.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d89c71beaea7b5-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:34.182 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.183 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.183 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.184 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.184 DEBUG:boto:Method: GET 2023-10-17T18:47:34.184 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.185 DEBUG:boto:Data: 2023-10-17T18:47:34.185 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.186 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.186 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.187 DEBUG:boto:Params: {} 2023-10-17T18:47:34.187 DEBUG:boto:Token: None 2023-10-17T18:47:34.188 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.188 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T18:47:34.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T18:47:34.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786ec2077d4fb071-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.191 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.192 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.192 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.193 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:34.193 DEBUG:boto:Method: GET 2023-10-17T18:47:34.194 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.194 DEBUG:boto:Data: 2023-10-17T18:47:34.194 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.195 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.195 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.196 DEBUG:boto:Params: {} 2023-10-17T18:47:34.196 DEBUG:boto:Token: None 2023-10-17T18:47:34.197 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:34.197 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T18:47:34.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T18:47:34.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001396714b2576766b-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.201 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.201 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.202 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.202 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.202 DEBUG:boto:Method: GET 2023-10-17T18:47:34.203 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:34.203 DEBUG:boto:Data: 2023-10-17T18:47:34.204 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.204 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.205 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.205 DEBUG:boto:Params: {} 2023-10-17T18:47:34.205 DEBUG:boto:Token: None 2023-10-17T18:47:34.206 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.206 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db2a44c5fb7f9e8-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.210 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.211 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.211 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.212 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.212 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.213 DEBUG:boto:Method: GET 2023-10-17T18:47:34.213 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:34.214 DEBUG:boto:Data: 2023-10-17T18:47:34.214 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.215 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.215 DEBUG:boto:Params: {} 2023-10-17T18:47:34.216 DEBUG:boto:Token: None 2023-10-17T18:47:34.217 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.217 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5b356d6ab62f7d-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.221 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.221 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.222 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.222 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.223 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:34.223 DEBUG:boto:Data: 2023-10-17T18:47:34.223 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.224 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.225 DEBUG:boto:Params: {} 2023-10-17T18:47:34.225 DEBUG:boto:Token: None 2023-10-17T18:47:34.226 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e 2023-10-17T18:47:34.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k= 2023-10-17T18:47:34.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k=', 'Content-Length': '0'} 2023-10-17T18:47:34.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab333afdb4018292-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.229 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.230 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.230 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.230 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:34.231 DEBUG:boto:Data: 2023-10-17T18:47:34.231 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.232 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.232 DEBUG:boto:Params: {} 2023-10-17T18:47:34.233 DEBUG:boto:Token: None 2023-10-17T18:47:34.233 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e 2023-10-17T18:47:34.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k= 2023-10-17T18:47:34.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k=', 'Content-Length': '0'} 2023-10-17T18:47:34.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157b387ec387a965-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:34.238 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.238 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.238 DEBUG:boto:Method: GET 2023-10-17T18:47:34.239 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:34.239 DEBUG:boto:Data: 2023-10-17T18:47:34.240 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.240 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.241 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.241 DEBUG:boto:Params: {} 2023-10-17T18:47:34.242 DEBUG:boto:Token: None 2023-10-17T18:47:34.242 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e 2023-10-17T18:47:34.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo= 2023-10-17T18:47:34.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo=', 'Content-Length': '0'} 2023-10-17T18:47:34.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5e2d9a308f40976-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.246 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:34.246 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:34.247 DEBUG:boto:Method: GET 2023-10-17T18:47:34.247 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:34.248 DEBUG:boto:Data: 2023-10-17T18:47:34.248 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.248 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.249 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.249 DEBUG:boto:Params: {} 2023-10-17T18:47:34.250 DEBUG:boto:Token: None 2023-10-17T18:47:34.250 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/e 2023-10-17T18:47:34.251 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo= 2023-10-17T18:47:34.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo=', 'Content-Length': '0'} 2023-10-17T18:47:34.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a97cfdd5f46f30-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.254 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.254 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.255 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.255 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.256 DEBUG:boto:Method: GET 2023-10-17T18:47:34.256 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:34.257 DEBUG:boto:Data: 2023-10-17T18:47:34.257 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.258 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.258 DEBUG:boto:Params: {} 2023-10-17T18:47:34.259 DEBUG:boto:Token: None 2023-10-17T18:47:34.259 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.260 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d4c055b9ec0143-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.264 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.265 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.265 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.266 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.266 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:34.267 DEBUG:boto:Method: GET 2023-10-17T18:47:34.267 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:34.267 DEBUG:boto:Data: 2023-10-17T18:47:34.268 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.268 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.269 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.269 DEBUG:boto:Params: {} 2023-10-17T18:47:34.270 DEBUG:boto:Token: None 2023-10-17T18:47:34.270 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba0be29fdc73699-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.276 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:34.278 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.278 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.279 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.279 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.279 DEBUG:boto:Method: GET 2023-10-17T18:47:34.280 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.280 DEBUG:boto:Data: 2023-10-17T18:47:34.281 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.282 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.282 DEBUG:boto:Params: {} 2023-10-17T18:47:34.283 DEBUG:boto:Token: None 2023-10-17T18:47:34.283 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.284 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T18:47:34.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T18:47:34.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382e1269d0be4d90-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.334 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.335 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.335 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.336 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.336 DEBUG:boto:Method: GET 2023-10-17T18:47:34.337 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.337 DEBUG:boto:Data: 2023-10-17T18:47:34.338 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.339 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.339 DEBUG:boto:Params: {} 2023-10-17T18:47:34.340 DEBUG:boto:Token: None 2023-10-17T18:47:34.340 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T18:47:34.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T18:47:34.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8f7b158b794c2a-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.344 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.345 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.345 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.346 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.347 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.347 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.348 DEBUG:boto:Data: 2023-10-17T18:47:34.348 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.349 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.349 DEBUG:boto:Params: {} 2023-10-17T18:47:34.350 DEBUG:boto:Token: None 2023-10-17T18:47:34.350 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.351 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g= 2023-10-17T18:47:34.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g=', 'Content-Length': '0'} 2023-10-17T18:47:34.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b87ea3c62392d8-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.354 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.355 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.355 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.356 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.356 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.357 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.357 DEBUG:boto:Data: 2023-10-17T18:47:34.358 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.359 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.359 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.360 DEBUG:boto:Params: {} 2023-10-17T18:47:34.361 DEBUG:boto:Token: None 2023-10-17T18:47:34.362 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.362 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g= 2023-10-17T18:47:34.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g=', 'Content-Length': '0'} 2023-10-17T18:47:34.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031909b04d3211495-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:34.367 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.368 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.368 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.369 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.369 DEBUG:boto:Method: GET 2023-10-17T18:47:34.370 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.370 DEBUG:boto:Data: 2023-10-17T18:47:34.371 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.371 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.371 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.372 DEBUG:boto:Params: {} 2023-10-17T18:47:34.372 DEBUG:boto:Token: None 2023-10-17T18:47:34.373 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T18:47:34.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T18:47:34.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7dbd41b2323f80-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.377 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.378 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.378 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.379 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:34.379 DEBUG:boto:Method: GET 2023-10-17T18:47:34.379 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.380 DEBUG:boto:Data: 2023-10-17T18:47:34.380 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.381 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.382 DEBUG:boto:Params: {} 2023-10-17T18:47:34.382 DEBUG:boto:Token: None 2023-10-17T18:47:34.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:34.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T18:47:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T18:47:34.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58cd1e4ea1f9232-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.430 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.431 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.431 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.432 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.432 DEBUG:boto:Method: GET 2023-10-17T18:47:34.433 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:34.433 DEBUG:boto:Data: 2023-10-17T18:47:34.434 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.434 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.435 DEBUG:boto:Params: {} 2023-10-17T18:47:34.435 DEBUG:boto:Token: None 2023-10-17T18:47:34.436 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.436 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155af481b7698b22-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.440 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.441 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.441 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.442 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.442 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.443 DEBUG:boto:Method: GET 2023-10-17T18:47:34.443 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:34.444 DEBUG:boto:Data: 2023-10-17T18:47:34.444 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.445 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.445 DEBUG:boto:Params: {} 2023-10-17T18:47:34.446 DEBUG:boto:Token: None 2023-10-17T18:47:34.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81fd388e27600d6-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.450 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.450 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.451 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.451 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.452 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:34.452 DEBUG:boto:Data: 2023-10-17T18:47:34.452 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.453 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.454 DEBUG:boto:Params: {} 2023-10-17T18:47:34.454 DEBUG:boto:Token: None 2023-10-17T18:47:34.455 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f 2023-10-17T18:47:34.455 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4= 2023-10-17T18:47:34.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4=', 'Content-Length': '0'} 2023-10-17T18:47:34.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2e7b2c047a7f9e-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.458 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.459 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.459 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.460 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:34.460 DEBUG:boto:Data: 2023-10-17T18:47:34.461 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.461 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.461 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.462 DEBUG:boto:Params: {} 2023-10-17T18:47:34.462 DEBUG:boto:Token: None 2023-10-17T18:47:34.463 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f 2023-10-17T18:47:34.463 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4= 2023-10-17T18:47:34.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4=', 'Content-Length': '0'} 2023-10-17T18:47:34.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7084b4106eabc4f-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:34.468 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.469 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.469 DEBUG:boto:Method: GET 2023-10-17T18:47:34.470 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:34.470 DEBUG:boto:Data: 2023-10-17T18:47:34.471 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.472 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.472 DEBUG:boto:Params: {} 2023-10-17T18:47:34.473 DEBUG:boto:Token: None 2023-10-17T18:47:34.473 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f 2023-10-17T18:47:34.474 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps= 2023-10-17T18:47:34.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps=', 'Content-Length': '0'} 2023-10-17T18:47:34.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deceb78d2fb1ec6f-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.477 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:34.478 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:34.478 DEBUG:boto:Method: GET 2023-10-17T18:47:34.479 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:34.479 DEBUG:boto:Data: 2023-10-17T18:47:34.480 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.481 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.481 DEBUG:boto:Params: {} 2023-10-17T18:47:34.481 DEBUG:boto:Token: None 2023-10-17T18:47:34.482 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/f 2023-10-17T18:47:34.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps= 2023-10-17T18:47:34.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps=', 'Content-Length': '0'} 2023-10-17T18:47:34.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d3050a6095e6ac-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.486 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.486 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.487 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.487 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.487 DEBUG:boto:Method: GET 2023-10-17T18:47:34.488 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:34.488 DEBUG:boto:Data: 2023-10-17T18:47:34.489 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.490 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.490 DEBUG:boto:Params: {} 2023-10-17T18:47:34.491 DEBUG:boto:Token: None 2023-10-17T18:47:34.492 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.492 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9386bf4c9900cf0-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.497 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.498 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.498 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.499 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.499 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:34.500 DEBUG:boto:Method: GET 2023-10-17T18:47:34.500 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:34.501 DEBUG:boto:Data: 2023-10-17T18:47:34.501 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.502 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.502 DEBUG:boto:Params: {} 2023-10-17T18:47:34.503 DEBUG:boto:Token: None 2023-10-17T18:47:34.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7ad71c21989349-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.507 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:34.508 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.509 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.509 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.510 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.510 DEBUG:boto:Method: GET 2023-10-17T18:47:34.511 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.511 DEBUG:boto:Data: 2023-10-17T18:47:34.511 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.512 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.513 DEBUG:boto:Params: {} 2023-10-17T18:47:34.513 DEBUG:boto:Token: None 2023-10-17T18:47:34.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T18:47:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T18:47:34.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6f220d768d0ece-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.518 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.518 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.519 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.519 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.520 DEBUG:boto:Method: GET 2023-10-17T18:47:34.520 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.520 DEBUG:boto:Data: 2023-10-17T18:47:34.521 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.522 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.522 DEBUG:boto:Params: {} 2023-10-17T18:47:34.523 DEBUG:boto:Token: None 2023-10-17T18:47:34.523 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T18:47:34.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T18:47:34.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1d816a0dc2b1e4-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.527 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.527 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.528 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.528 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.528 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.529 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.529 DEBUG:boto:Data: 2023-10-17T18:47:34.530 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.530 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.531 DEBUG:boto:Params: {} 2023-10-17T18:47:34.531 DEBUG:boto:Token: None 2023-10-17T18:47:34.532 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY= 2023-10-17T18:47:34.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY=', 'Content-Length': '0'} 2023-10-17T18:47:34.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6cadaa244877539-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.535 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.536 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.537 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.537 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.537 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.538 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.538 DEBUG:boto:Data: 2023-10-17T18:47:34.539 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.539 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.540 DEBUG:boto:Params: {} 2023-10-17T18:47:34.540 DEBUG:boto:Token: None 2023-10-17T18:47:34.541 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY= 2023-10-17T18:47:34.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY=', 'Content-Length': '0'} 2023-10-17T18:47:34.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c33a047b1f2a265-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:34.545 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.545 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.546 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.546 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.547 DEBUG:boto:Method: GET 2023-10-17T18:47:34.547 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.548 DEBUG:boto:Data: 2023-10-17T18:47:34.548 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.548 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.549 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.549 DEBUG:boto:Params: {} 2023-10-17T18:47:34.550 DEBUG:boto:Token: None 2023-10-17T18:47:34.550 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.551 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T18:47:34.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T18:47:34.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139689e13212956d-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.594 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.595 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.596 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.596 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:34.596 DEBUG:boto:Method: GET 2023-10-17T18:47:34.597 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.597 DEBUG:boto:Data: 2023-10-17T18:47:34.598 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.599 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.600 DEBUG:boto:Params: {} 2023-10-17T18:47:34.601 DEBUG:boto:Token: None 2023-10-17T18:47:34.601 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:34.602 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T18:47:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T18:47:34.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013bf83f635e39ea9-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.606 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.607 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.607 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.608 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.608 DEBUG:boto:Method: GET 2023-10-17T18:47:34.608 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:34.609 DEBUG:boto:Data: 2023-10-17T18:47:34.609 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.610 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.611 DEBUG:boto:Params: {} 2023-10-17T18:47:34.611 DEBUG:boto:Token: None 2023-10-17T18:47:34.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ec26f7d63487ed-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.615 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.616 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.617 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.617 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.617 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.618 DEBUG:boto:Method: GET 2023-10-17T18:47:34.618 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:34.619 DEBUG:boto:Data: 2023-10-17T18:47:34.619 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.620 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.620 DEBUG:boto:Params: {} 2023-10-17T18:47:34.621 DEBUG:boto:Token: None 2023-10-17T18:47:34.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de056c84f440c87-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.626 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.626 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.627 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.627 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.628 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:34.628 DEBUG:boto:Data: 2023-10-17T18:47:34.628 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.629 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.629 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.630 DEBUG:boto:Params: {} 2023-10-17T18:47:34.630 DEBUG:boto:Token: None 2023-10-17T18:47:34.630 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g 2023-10-17T18:47:34.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI= 2023-10-17T18:47:34.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI=', 'Content-Length': '0'} 2023-10-17T18:47:34.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d21227c40784d2-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.634 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.634 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.635 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.635 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:34.636 DEBUG:boto:Data: 2023-10-17T18:47:34.636 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.636 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.637 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.637 DEBUG:boto:Params: {} 2023-10-17T18:47:34.638 DEBUG:boto:Token: None 2023-10-17T18:47:34.638 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g 2023-10-17T18:47:34.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI= 2023-10-17T18:47:34.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI=', 'Content-Length': '0'} 2023-10-17T18:47:34.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078256ee6e2940741-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:34.643 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.643 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.644 DEBUG:boto:Method: GET 2023-10-17T18:47:34.644 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:34.645 DEBUG:boto:Data: 2023-10-17T18:47:34.646 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.647 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.647 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.648 DEBUG:boto:Params: {} 2023-10-17T18:47:34.648 DEBUG:boto:Token: None 2023-10-17T18:47:34.649 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g 2023-10-17T18:47:34.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo= 2023-10-17T18:47:34.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo=', 'Content-Length': '0'} 2023-10-17T18:47:34.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003effad1d8b7f2897-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.654 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:34.655 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:34.655 DEBUG:boto:Method: GET 2023-10-17T18:47:34.656 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:34.657 DEBUG:boto:Data: 2023-10-17T18:47:34.658 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.659 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.659 DEBUG:boto:Params: {} 2023-10-17T18:47:34.660 DEBUG:boto:Token: None 2023-10-17T18:47:34.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/g 2023-10-17T18:47:34.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo= 2023-10-17T18:47:34.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo=', 'Content-Length': '0'} 2023-10-17T18:47:34.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c6794bd3746c0b-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.664 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.665 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.665 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.665 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.666 DEBUG:boto:Method: GET 2023-10-17T18:47:34.666 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:34.667 DEBUG:boto:Data: 2023-10-17T18:47:34.668 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.669 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.670 DEBUG:boto:Params: {} 2023-10-17T18:47:34.670 DEBUG:boto:Token: None 2023-10-17T18:47:34.671 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d086f3f155ebad2-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.676 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.677 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.677 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.677 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.678 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:34.678 DEBUG:boto:Method: GET 2023-10-17T18:47:34.679 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:34.679 DEBUG:boto:Data: 2023-10-17T18:47:34.680 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.681 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.681 DEBUG:boto:Params: {} 2023-10-17T18:47:34.681 DEBUG:boto:Token: None 2023-10-17T18:47:34.682 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faadae5ed672d7ba-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.686 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:34.687 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.687 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.688 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.688 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.689 DEBUG:boto:Method: GET 2023-10-17T18:47:34.689 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.690 DEBUG:boto:Data: 2023-10-17T18:47:34.690 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.691 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.691 DEBUG:boto:Params: {} 2023-10-17T18:47:34.692 DEBUG:boto:Token: None 2023-10-17T18:47:34.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T18:47:34.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T18:47:34.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033814b698cf7811f-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.738 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.739 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.739 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.740 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.740 DEBUG:boto:Method: GET 2023-10-17T18:47:34.741 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.741 DEBUG:boto:Data: 2023-10-17T18:47:34.742 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.742 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.743 DEBUG:boto:Params: {} 2023-10-17T18:47:34.743 DEBUG:boto:Token: None 2023-10-17T18:47:34.744 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T18:47:34.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T18:47:34.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a06001c78093a3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.749 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.750 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.750 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.751 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.751 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.752 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.752 DEBUG:boto:Data: 2023-10-17T18:47:34.752 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.753 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.753 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.754 DEBUG:boto:Params: {} 2023-10-17T18:47:34.754 DEBUG:boto:Token: None 2023-10-17T18:47:34.755 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.755 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28= 2023-10-17T18:47:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28=', 'Content-Length': '0'} 2023-10-17T18:47:34.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a891d502da2a9451-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.758 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.759 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.759 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.760 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.760 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.760 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.761 DEBUG:boto:Data: 2023-10-17T18:47:34.761 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.762 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.762 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.763 DEBUG:boto:Params: {} 2023-10-17T18:47:34.766 DEBUG:boto:Token: None 2023-10-17T18:47:34.767 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28= 2023-10-17T18:47:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28=', 'Content-Length': '0'} 2023-10-17T18:47:34.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa1fd45e9716db3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:34.772 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.772 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.773 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.773 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.774 DEBUG:boto:Method: GET 2023-10-17T18:47:34.774 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.774 DEBUG:boto:Data: 2023-10-17T18:47:34.775 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.776 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.776 DEBUG:boto:Params: {} 2023-10-17T18:47:34.777 DEBUG:boto:Token: None 2023-10-17T18:47:34.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T18:47:34.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T18:47:34.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0291277b055e30c-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.781 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.781 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.782 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.782 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:34.783 DEBUG:boto:Method: GET 2023-10-17T18:47:34.783 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.783 DEBUG:boto:Data: 2023-10-17T18:47:34.784 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.785 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.785 DEBUG:boto:Params: {} 2023-10-17T18:47:34.786 DEBUG:boto:Token: None 2023-10-17T18:47:34.786 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:34.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T18:47:34.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T18:47:34.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed564413bfdf7b7-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.790 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.790 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.791 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.791 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.791 DEBUG:boto:Method: GET 2023-10-17T18:47:34.792 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:34.792 DEBUG:boto:Data: 2023-10-17T18:47:34.793 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.793 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.794 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.794 DEBUG:boto:Params: {} 2023-10-17T18:47:34.794 DEBUG:boto:Token: None 2023-10-17T18:47:34.795 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.795 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a24f7f0d1901146f-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.799 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.800 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.800 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.801 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.801 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.802 DEBUG:boto:Method: GET 2023-10-17T18:47:34.802 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:34.803 DEBUG:boto:Data: 2023-10-17T18:47:34.803 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.803 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.804 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.804 DEBUG:boto:Params: {} 2023-10-17T18:47:34.805 DEBUG:boto:Token: None 2023-10-17T18:47:34.805 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.806 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dff311ce203747-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.809 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.809 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.810 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.810 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.811 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:34.811 DEBUG:boto:Data: 2023-10-17T18:47:34.811 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.812 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.813 DEBUG:boto:Params: {} 2023-10-17T18:47:34.813 DEBUG:boto:Token: None 2023-10-17T18:47:34.814 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h 2023-10-17T18:47:34.814 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74= 2023-10-17T18:47:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74=', 'Content-Length': '0'} 2023-10-17T18:47:34.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b088521ac9b599-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.817 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.817 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.818 DEBUG:boto:Method: HEAD 2023-10-17T18:47:34.818 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:34.818 DEBUG:boto:Data: 2023-10-17T18:47:34.819 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.819 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.820 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.820 DEBUG:boto:Params: {} 2023-10-17T18:47:34.821 DEBUG:boto:Token: None 2023-10-17T18:47:34.821 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h 2023-10-17T18:47:34.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74= 2023-10-17T18:47:34.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74=', 'Content-Length': '0'} 2023-10-17T18:47:34.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006750a63c2263388-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:34.825 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.825 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.826 DEBUG:boto:Method: GET 2023-10-17T18:47:34.826 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:34.826 DEBUG:boto:Data: 2023-10-17T18:47:34.827 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.828 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.828 DEBUG:boto:Params: {} 2023-10-17T18:47:34.829 DEBUG:boto:Token: None 2023-10-17T18:47:34.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h 2023-10-17T18:47:34.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8= 2023-10-17T18:47:34.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8=', 'Content-Length': '0'} 2023-10-17T18:47:34.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e0a95a679c07e4-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.833 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:34.833 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:34.834 DEBUG:boto:Method: GET 2023-10-17T18:47:34.834 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:34.834 DEBUG:boto:Data: 2023-10-17T18:47:34.835 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.836 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.836 DEBUG:boto:Params: {} 2023-10-17T18:47:34.837 DEBUG:boto:Token: None 2023-10-17T18:47:34.837 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/h 2023-10-17T18:47:34.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8= 2023-10-17T18:47:34.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8=', 'Content-Length': '0'} 2023-10-17T18:47:34.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8fd6715252dee3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.840 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.841 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.841 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.842 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.842 DEBUG:boto:Method: GET 2023-10-17T18:47:34.843 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:34.843 DEBUG:boto:Data: 2023-10-17T18:47:34.844 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.844 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:34.844 DEBUG:boto:Port: 8001 2023-10-17T18:47:34.845 DEBUG:boto:Params: {} 2023-10-17T18:47:34.845 DEBUG:boto:Token: None 2023-10-17T18:47:34.846 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.846 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c439e3775824ff29-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.850 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.851 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:34.851 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:34.851 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.852 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:34.852 DEBUG:boto:Method: GET 2023-10-17T18:47:34.853 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:34.853 DEBUG:boto:Data: 2023-10-17T18:47:34.854 DEBUG:boto:Headers: {} 2023-10-17T18:47:34.854 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:34.854 DEBUG:boto:Port: 8000 2023-10-17T18:47:34.855 DEBUG:boto:Params: {} 2023-10-17T18:47:34.855 DEBUG:boto:Token: None 2023-10-17T18:47:34.856 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:34 GMT /hbmuog-55/ 2023-10-17T18:47:34.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T18:47:34.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T18:47:34.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76b0f7e6de3931b-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:34.860 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:34.860 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:47:34.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:34.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:34.862 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:34.919 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:34.920 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:47:34.962 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-55 2023-10-17T18:47:34.963 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 8 2023-10-17T18:47:34.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:34.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:34.992 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:35.040 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:35.040 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:35.081 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:47:35.081 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.081 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding": "None", 2023-10-17T18:47:35.082 INFO:teuthology.orchestra.run.smithi049.stdout: "current_index": { 2023-10-17T18:47:35.082 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:35.082 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "Normal", 2023-10-17T18:47:35.083 INFO:teuthology.orchestra.run.smithi049.stdout: "normal": { 2023-10-17T18:47:35.083 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:35.083 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:35.083 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:35.084 INFO:teuthology.orchestra.run.smithi049.stdout: "logs": [ 2023-10-17T18:47:35.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:35.084 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:47:35.085 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:35.085 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:35.086 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 0, 2023-10-17T18:47:35.086 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.086 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 11, 2023-10-17T18:47:35.086 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:35.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:35.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:35.088 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:35.088 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:35.089 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:35.089 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:35.089 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.089 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:35.090 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:35.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:35.092 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:35.092 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:35.092 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:35.092 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:35.092 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:35.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:47:35.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:35.094 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:47:35.095 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T18:47:35.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:35.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:35.097 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:47:35.156 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:35.157 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:35.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:35.190+0000 7f4fce5df780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:47:40.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:40.191+0000 7f4fce5df780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:47:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:45.191+0000 7f4fce5df780 1 bucket sync caught up with empty source 2023-10-17T18:47:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:45.191+0000 7f4fce5df780 0 bucket checkpoint complete 2023-10-17T18:47:45.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T18:47:45.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:45.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:45.213 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:47:45.280 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:45.280 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:45.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:45.318+0000 7fa6f3413780 1 bucket sync caught up with source: 2023-10-17T18:47:45.320 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:45.320 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T18:47:45.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:45.318+0000 7fa6f3413780 0 bucket checkpoint complete 2023-10-17T18:47:45.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:47:45.334 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.335 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.335 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.336 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:47:45.336 DEBUG:boto:Data: 2023-10-17T18:47:45.337 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.338 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.338 DEBUG:boto:Params: {} 2023-10-17T18:47:45.339 DEBUG:boto:Token: None 2023-10-17T18:47:45.339 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8= 2023-10-17T18:47:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8=', 'Content-Length': '0'} 2023-10-17T18:47:45.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b2106f5be232f1-00652ed6d1-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.344 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.344 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.344 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.345 DEBUG:boto:Path: /hbmuog-55/ 2023-10-17T18:47:45.345 DEBUG:boto:Data: 2023-10-17T18:47:45.346 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.347 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.347 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.348 DEBUG:boto:Params: {} 2023-10-17T18:47:45.348 DEBUG:boto:Token: None 2023-10-17T18:47:45.349 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8= 2023-10-17T18:47:45.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8=', 'Content-Length': '0'} 2023-10-17T18:47:45.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046f2d888c701c053-00652ed6d1-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:47:45.354 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.355 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.355 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.356 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.356 DEBUG:boto:Method: GET 2023-10-17T18:47:45.357 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.357 DEBUG:boto:Data: 2023-10-17T18:47:45.358 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.358 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.359 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.359 DEBUG:boto:Params: {} 2023-10-17T18:47:45.360 DEBUG:boto:Token: None 2023-10-17T18:47:45.360 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/?versions 2023-10-17T18:47:45.361 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T18:47:45.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T18:47:45.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da90932d2938ae1-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.364 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.365 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:47:45.366 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:47:45.366 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:47:45.367 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:47:45.367 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:47:45.368 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:47:45.368 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:47:45.369 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:47:45.369 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:47:45.370 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.370 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.371 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.371 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.371 DEBUG:boto:Method: GET 2023-10-17T18:47:45.372 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.372 DEBUG:boto:Data: 2023-10-17T18:47:45.373 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.373 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.374 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.374 DEBUG:boto:Params: {} 2023-10-17T18:47:45.374 DEBUG:boto:Token: None 2023-10-17T18:47:45.375 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/?versions 2023-10-17T18:47:45.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T18:47:45.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T18:47:45.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad0a81d6659045d-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.378 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.379 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:47:45.380 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:47:45.380 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:47:45.380 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:47:45.381 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:47:45.381 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:47:45.382 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:47:45.383 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:47:45.383 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.383 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.384 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.385 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.385 DEBUG:boto:Method: GET 2023-10-17T18:47:45.386 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.386 DEBUG:boto:Data: 2023-10-17T18:47:45.387 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.388 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.388 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.389 DEBUG:boto:Params: {} 2023-10-17T18:47:45.390 DEBUG:boto:Token: None 2023-10-17T18:47:45.390 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/?versions 2023-10-17T18:47:45.391 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T18:47:45.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T18:47:45.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15b59dba366a2c5-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.394 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.395 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.396 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.396 DEBUG:boto:path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.397 DEBUG:boto:auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.397 DEBUG:boto:Method: GET 2023-10-17T18:47:45.398 DEBUG:boto:Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T18:47:45.398 DEBUG:boto:Data: 2023-10-17T18:47:45.399 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.399 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.400 DEBUG:boto:Params: {} 2023-10-17T18:47:45.400 DEBUG:boto:Token: None 2023-10-17T18:47:45.401 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/?versions 2023-10-17T18:47:45.401 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T18:47:45.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T18:47:45.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c956598b62f1562-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.404 DEBUG:boto:b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:45.405 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.406 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.406 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.407 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.407 DEBUG:boto:Method: GET 2023-10-17T18:47:45.408 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.408 DEBUG:boto:Data: 2023-10-17T18:47:45.409 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.410 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.410 DEBUG:boto:Params: {} 2023-10-17T18:47:45.411 DEBUG:boto:Token: None 2023-10-17T18:47:45.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T18:47:45.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T18:47:45.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526525169338971d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.454 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.455 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.456 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.456 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.457 DEBUG:boto:Method: GET 2023-10-17T18:47:45.457 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.458 DEBUG:boto:Data: 2023-10-17T18:47:45.458 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.459 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.459 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.459 DEBUG:boto:Params: {} 2023-10-17T18:47:45.460 DEBUG:boto:Token: None 2023-10-17T18:47:45.461 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.461 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T18:47:45.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T18:47:45.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042119550987ea522-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.465 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.466 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.467 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.468 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.468 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.469 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.470 DEBUG:boto:Data: 2023-10-17T18:47:45.471 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.472 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.473 DEBUG:boto:Params: {} 2023-10-17T18:47:45.473 DEBUG:boto:Token: None 2023-10-17T18:47:45.474 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.475 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA= 2023-10-17T18:47:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA=', 'Content-Length': '0'} 2023-10-17T18:47:45.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8167fe159fba5e-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.479 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.480 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.480 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.481 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.482 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.483 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.484 DEBUG:boto:Data: 2023-10-17T18:47:45.484 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.485 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.486 DEBUG:boto:Params: {} 2023-10-17T18:47:45.486 DEBUG:boto:Token: None 2023-10-17T18:47:45.487 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA= 2023-10-17T18:47:45.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA=', 'Content-Length': '0'} 2023-10-17T18:47:45.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005971789b6b85d1ca-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:45.491 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.491 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.492 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.492 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.492 DEBUG:boto:Method: GET 2023-10-17T18:47:45.493 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.493 DEBUG:boto:Data: 2023-10-17T18:47:45.494 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.494 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.495 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.495 DEBUG:boto:Params: {} 2023-10-17T18:47:45.496 DEBUG:boto:Token: None 2023-10-17T18:47:45.496 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T18:47:45.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T18:47:45.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103cbce04635ccb3-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.501 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.501 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.502 DEBUG:boto:path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.503 DEBUG:boto:auth_path=/hbmuog-55/a?versionId=null 2023-10-17T18:47:45.504 DEBUG:boto:Method: GET 2023-10-17T18:47:45.504 DEBUG:boto:Path: /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.505 DEBUG:boto:Data: 2023-10-17T18:47:45.506 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.507 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.507 DEBUG:boto:Params: {} 2023-10-17T18:47:45.508 DEBUG:boto:Token: None 2023-10-17T18:47:45.508 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a?versionId=null 2023-10-17T18:47:45.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T18:47:45.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T18:47:45.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2f0b120445c27c-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.513 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.513 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.514 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.515 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.515 DEBUG:boto:Method: GET 2023-10-17T18:47:45.516 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:45.517 DEBUG:boto:Data: 2023-10-17T18:47:45.517 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.518 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.518 DEBUG:boto:Params: {} 2023-10-17T18:47:45.519 DEBUG:boto:Token: None 2023-10-17T18:47:45.519 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb224465411f255-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.523 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.524 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.525 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.525 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.525 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.526 DEBUG:boto:Method: GET 2023-10-17T18:47:45.526 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:45.527 DEBUG:boto:Data: 2023-10-17T18:47:45.527 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.528 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.529 DEBUG:boto:Params: {} 2023-10-17T18:47:45.529 DEBUG:boto:Token: None 2023-10-17T18:47:45.530 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.530 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2327f02599ac8ca-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.533 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.534 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.534 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.535 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.535 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:45.536 DEBUG:boto:Data: 2023-10-17T18:47:45.536 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.536 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.537 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.537 DEBUG:boto:Params: {} 2023-10-17T18:47:45.538 DEBUG:boto:Token: None 2023-10-17T18:47:45.538 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a 2023-10-17T18:47:45.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4= 2023-10-17T18:47:45.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4=', 'Content-Length': '0'} 2023-10-17T18:47:45.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300124897607a0c8-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.542 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.542 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.542 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.543 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:45.543 DEBUG:boto:Data: 2023-10-17T18:47:45.544 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.544 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.544 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.545 DEBUG:boto:Params: {} 2023-10-17T18:47:45.545 DEBUG:boto:Token: None 2023-10-17T18:47:45.546 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a 2023-10-17T18:47:45.546 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4= 2023-10-17T18:47:45.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4=', 'Content-Length': '0'} 2023-10-17T18:47:45.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b6c45b4150a375-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:47:45.549 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.550 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.550 DEBUG:boto:Method: GET 2023-10-17T18:47:45.551 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:45.551 DEBUG:boto:Data: 2023-10-17T18:47:45.551 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.552 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.552 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.553 DEBUG:boto:Params: {} 2023-10-17T18:47:45.553 DEBUG:boto:Token: None 2023-10-17T18:47:45.554 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a 2023-10-17T18:47:45.554 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8= 2023-10-17T18:47:45.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8=', 'Content-Length': '0'} 2023-10-17T18:47:45.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b45d3143d33f60e-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.557 DEBUG:boto:path=/hbmuog-55/a 2023-10-17T18:47:45.558 DEBUG:boto:auth_path=/hbmuog-55/a 2023-10-17T18:47:45.558 DEBUG:boto:Method: GET 2023-10-17T18:47:45.559 DEBUG:boto:Path: /hbmuog-55/a 2023-10-17T18:47:45.559 DEBUG:boto:Data: 2023-10-17T18:47:45.559 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.560 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.561 DEBUG:boto:Params: {} 2023-10-17T18:47:45.561 DEBUG:boto:Token: None 2023-10-17T18:47:45.562 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/a 2023-10-17T18:47:45.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8= 2023-10-17T18:47:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8=', 'Content-Length': '0'} 2023-10-17T18:47:45.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47686cdb141dc61-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.565 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.566 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.566 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.566 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.567 DEBUG:boto:Method: GET 2023-10-17T18:47:45.567 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:45.568 DEBUG:boto:Data: 2023-10-17T18:47:45.568 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.569 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.569 DEBUG:boto:Params: {} 2023-10-17T18:47:45.570 DEBUG:boto:Token: None 2023-10-17T18:47:45.570 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b883eff959f2d513-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.574 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.575 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.575 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.576 DEBUG:boto:path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.576 DEBUG:boto:auth_path=/hbmuog-55/?prefix=a 2023-10-17T18:47:45.577 DEBUG:boto:Method: GET 2023-10-17T18:47:45.577 DEBUG:boto:Path: /hbmuog-55/?prefix=a 2023-10-17T18:47:45.578 DEBUG:boto:Data: 2023-10-17T18:47:45.578 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.579 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.579 DEBUG:boto:Params: {} 2023-10-17T18:47:45.580 DEBUG:boto:Token: None 2023-10-17T18:47:45.580 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca165f99dd66d04b-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.584 DEBUG:boto:b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:45.585 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.585 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.586 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.586 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.587 DEBUG:boto:Method: GET 2023-10-17T18:47:45.587 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.587 DEBUG:boto:Data: 2023-10-17T18:47:45.588 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.589 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.589 DEBUG:boto:Params: {} 2023-10-17T18:47:45.590 DEBUG:boto:Token: None 2023-10-17T18:47:45.590 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T18:47:45.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T18:47:45.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21b4146ae4b59e1-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.594 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.594 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.595 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.595 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.595 DEBUG:boto:Method: GET 2023-10-17T18:47:45.596 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.596 DEBUG:boto:Data: 2023-10-17T18:47:45.597 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.598 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.598 DEBUG:boto:Params: {} 2023-10-17T18:47:45.599 DEBUG:boto:Token: None 2023-10-17T18:47:45.599 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T18:47:45.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T18:47:45.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043fc6b0dd1c0686-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.602 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.603 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.603 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.604 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.604 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.605 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.605 DEBUG:boto:Data: 2023-10-17T18:47:45.605 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.606 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.607 DEBUG:boto:Params: {} 2023-10-17T18:47:45.607 DEBUG:boto:Token: None 2023-10-17T18:47:45.608 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.608 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE= 2023-10-17T18:47:45.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE=', 'Content-Length': '0'} 2023-10-17T18:47:45.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ccb1940c0d7c92-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.611 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.611 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.611 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.612 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.612 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.613 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.613 DEBUG:boto:Data: 2023-10-17T18:47:45.614 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.614 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.615 DEBUG:boto:Params: {} 2023-10-17T18:47:45.615 DEBUG:boto:Token: None 2023-10-17T18:47:45.616 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE= 2023-10-17T18:47:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE=', 'Content-Length': '0'} 2023-10-17T18:47:45.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b586543a35f10fcc-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:45.619 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.620 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.620 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.621 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.621 DEBUG:boto:Method: GET 2023-10-17T18:47:45.622 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.622 DEBUG:boto:Data: 2023-10-17T18:47:45.623 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.624 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.624 DEBUG:boto:Params: {} 2023-10-17T18:47:45.624 DEBUG:boto:Token: None 2023-10-17T18:47:45.625 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.626 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T18:47:45.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T18:47:45.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cedf5e7299e94948-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.629 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.629 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.629 DEBUG:boto:path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.630 DEBUG:boto:auth_path=/hbmuog-55/b?versionId=null 2023-10-17T18:47:45.630 DEBUG:boto:Method: GET 2023-10-17T18:47:45.631 DEBUG:boto:Path: /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.631 DEBUG:boto:Data: 2023-10-17T18:47:45.632 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.632 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.633 DEBUG:boto:Params: {} 2023-10-17T18:47:45.633 DEBUG:boto:Token: None 2023-10-17T18:47:45.634 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b?versionId=null 2023-10-17T18:47:45.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T18:47:45.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T18:47:45.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a1194a5883d191-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.678 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.679 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.679 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.680 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.680 DEBUG:boto:Method: GET 2023-10-17T18:47:45.680 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:45.681 DEBUG:boto:Data: 2023-10-17T18:47:45.681 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.682 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.683 DEBUG:boto:Params: {} 2023-10-17T18:47:45.683 DEBUG:boto:Token: None 2023-10-17T18:47:45.684 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b183cdc7c33bde-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.688 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.688 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.689 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.689 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.690 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.690 DEBUG:boto:Method: GET 2023-10-17T18:47:45.691 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:45.691 DEBUG:boto:Data: 2023-10-17T18:47:45.691 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.692 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.693 DEBUG:boto:Params: {} 2023-10-17T18:47:45.693 DEBUG:boto:Token: None 2023-10-17T18:47:45.693 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755f76514725a8b2-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.697 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.698 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.698 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.698 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.699 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:45.699 DEBUG:boto:Data: 2023-10-17T18:47:45.700 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.700 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.700 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.701 DEBUG:boto:Params: {} 2023-10-17T18:47:45.701 DEBUG:boto:Token: None 2023-10-17T18:47:45.701 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b 2023-10-17T18:47:45.702 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU= 2023-10-17T18:47:45.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU=', 'Content-Length': '0'} 2023-10-17T18:47:45.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030994c0f7e3ca020-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.705 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.705 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.705 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.706 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:45.706 DEBUG:boto:Data: 2023-10-17T18:47:45.707 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.708 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.708 DEBUG:boto:Params: {} 2023-10-17T18:47:45.709 DEBUG:boto:Token: None 2023-10-17T18:47:45.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b 2023-10-17T18:47:45.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU= 2023-10-17T18:47:45.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU=', 'Content-Length': '0'} 2023-10-17T18:47:45.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50354faa7761a7b-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:47:45.713 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.713 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.713 DEBUG:boto:Method: GET 2023-10-17T18:47:45.714 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:45.714 DEBUG:boto:Data: 2023-10-17T18:47:45.715 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.716 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.716 DEBUG:boto:Params: {} 2023-10-17T18:47:45.717 DEBUG:boto:Token: None 2023-10-17T18:47:45.717 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b 2023-10-17T18:47:45.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw= 2023-10-17T18:47:45.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw=', 'Content-Length': '0'} 2023-10-17T18:47:45.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274b4693e0e08cbc-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.762 DEBUG:boto:path=/hbmuog-55/b 2023-10-17T18:47:45.763 DEBUG:boto:auth_path=/hbmuog-55/b 2023-10-17T18:47:45.763 DEBUG:boto:Method: GET 2023-10-17T18:47:45.764 DEBUG:boto:Path: /hbmuog-55/b 2023-10-17T18:47:45.764 DEBUG:boto:Data: 2023-10-17T18:47:45.764 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.765 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.766 DEBUG:boto:Params: {} 2023-10-17T18:47:45.766 DEBUG:boto:Token: None 2023-10-17T18:47:45.766 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/b 2023-10-17T18:47:45.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw= 2023-10-17T18:47:45.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw=', 'Content-Length': '0'} 2023-10-17T18:47:45.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001965f80a85676e3c-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.770 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.770 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.771 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.771 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.771 DEBUG:boto:Method: GET 2023-10-17T18:47:45.772 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:45.772 DEBUG:boto:Data: 2023-10-17T18:47:45.773 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.773 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.773 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.774 DEBUG:boto:Params: {} 2023-10-17T18:47:45.774 DEBUG:boto:Token: None 2023-10-17T18:47:45.775 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.775 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007077fab4e7fad2ed-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.778 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.779 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.780 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.780 DEBUG:boto:path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.780 DEBUG:boto:auth_path=/hbmuog-55/?prefix=b 2023-10-17T18:47:45.781 DEBUG:boto:Method: GET 2023-10-17T18:47:45.781 DEBUG:boto:Path: /hbmuog-55/?prefix=b 2023-10-17T18:47:45.782 DEBUG:boto:Data: 2023-10-17T18:47:45.782 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.783 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.783 DEBUG:boto:Params: {} 2023-10-17T18:47:45.784 DEBUG:boto:Token: None 2023-10-17T18:47:45.784 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6db61aede6a3676-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.788 DEBUG:boto:b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:45.789 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.789 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.790 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.790 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.791 DEBUG:boto:Method: GET 2023-10-17T18:47:45.791 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.791 DEBUG:boto:Data: 2023-10-17T18:47:45.792 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.793 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.793 DEBUG:boto:Params: {} 2023-10-17T18:47:45.793 DEBUG:boto:Token: None 2023-10-17T18:47:45.794 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T18:47:45.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T18:47:45.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092122c5305f1aa5d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.797 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.798 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.798 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.799 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.799 DEBUG:boto:Method: GET 2023-10-17T18:47:45.800 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.800 DEBUG:boto:Data: 2023-10-17T18:47:45.800 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.801 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.802 DEBUG:boto:Params: {} 2023-10-17T18:47:45.802 DEBUG:boto:Token: None 2023-10-17T18:47:45.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T18:47:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T18:47:45.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e3de75bab4abe54-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.806 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.807 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.807 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.807 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.808 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.808 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.809 DEBUG:boto:Data: 2023-10-17T18:47:45.809 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.810 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.810 DEBUG:boto:Params: {} 2023-10-17T18:47:45.811 DEBUG:boto:Token: None 2023-10-17T18:47:45.811 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc= 2023-10-17T18:47:45.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc=', 'Content-Length': '0'} 2023-10-17T18:47:45.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee4f70e189c3708-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.815 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.815 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.815 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.816 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.816 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.817 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.817 DEBUG:boto:Data: 2023-10-17T18:47:45.817 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.818 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.819 DEBUG:boto:Params: {} 2023-10-17T18:47:45.819 DEBUG:boto:Token: None 2023-10-17T18:47:45.820 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.820 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc= 2023-10-17T18:47:45.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc=', 'Content-Length': '0'} 2023-10-17T18:47:45.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037fde36f1ac647d-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:45.823 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.824 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.824 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.824 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.825 DEBUG:boto:Method: GET 2023-10-17T18:47:45.825 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.826 DEBUG:boto:Data: 2023-10-17T18:47:45.826 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.827 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.827 DEBUG:boto:Params: {} 2023-10-17T18:47:45.829 DEBUG:boto:Token: None 2023-10-17T18:47:45.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T18:47:45.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T18:47:45.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3fe020d0286c9c-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.833 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.833 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.834 DEBUG:boto:path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.834 DEBUG:boto:auth_path=/hbmuog-55/c?versionId=null 2023-10-17T18:47:45.835 DEBUG:boto:Method: GET 2023-10-17T18:47:45.835 DEBUG:boto:Path: /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.836 DEBUG:boto:Data: 2023-10-17T18:47:45.836 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.837 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.838 DEBUG:boto:Params: {} 2023-10-17T18:47:45.838 DEBUG:boto:Token: None 2023-10-17T18:47:45.839 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c?versionId=null 2023-10-17T18:47:45.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T18:47:45.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T18:47:45.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099a30dd2d938261-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.842 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.843 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.843 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.844 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.844 DEBUG:boto:Method: GET 2023-10-17T18:47:45.844 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:45.845 DEBUG:boto:Data: 2023-10-17T18:47:45.845 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.846 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.846 DEBUG:boto:Params: {} 2023-10-17T18:47:45.847 DEBUG:boto:Token: None 2023-10-17T18:47:45.847 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.848 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bf7c0e2fd529b03-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.851 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.852 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.852 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.853 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.853 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.854 DEBUG:boto:Method: GET 2023-10-17T18:47:45.854 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:45.854 DEBUG:boto:Data: 2023-10-17T18:47:45.855 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.855 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.856 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.856 DEBUG:boto:Params: {} 2023-10-17T18:47:45.857 DEBUG:boto:Token: None 2023-10-17T18:47:45.857 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ee874b3c302c73-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.861 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.861 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.862 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.862 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.863 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:45.863 DEBUG:boto:Data: 2023-10-17T18:47:45.863 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.864 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.864 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.865 DEBUG:boto:Params: {} 2023-10-17T18:47:45.865 DEBUG:boto:Token: None 2023-10-17T18:47:45.865 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c 2023-10-17T18:47:45.866 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ= 2023-10-17T18:47:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ=', 'Content-Length': '0'} 2023-10-17T18:47:45.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebef9aa44f68638c-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.869 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.869 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.870 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.870 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:45.870 DEBUG:boto:Data: 2023-10-17T18:47:45.871 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.872 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.872 DEBUG:boto:Params: {} 2023-10-17T18:47:45.873 DEBUG:boto:Token: None 2023-10-17T18:47:45.873 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c 2023-10-17T18:47:45.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ= 2023-10-17T18:47:45.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ=', 'Content-Length': '0'} 2023-10-17T18:47:45.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1c01e4e11117a0-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:47:45.877 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.877 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.878 DEBUG:boto:Method: GET 2023-10-17T18:47:45.878 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:45.879 DEBUG:boto:Data: 2023-10-17T18:47:45.879 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.880 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.880 DEBUG:boto:Params: {} 2023-10-17T18:47:45.881 DEBUG:boto:Token: None 2023-10-17T18:47:45.881 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c 2023-10-17T18:47:45.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY= 2023-10-17T18:47:45.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY=', 'Content-Length': '0'} 2023-10-17T18:47:45.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68d9b3e3b63aa9a-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.885 DEBUG:boto:path=/hbmuog-55/c 2023-10-17T18:47:45.885 DEBUG:boto:auth_path=/hbmuog-55/c 2023-10-17T18:47:45.886 DEBUG:boto:Method: GET 2023-10-17T18:47:45.886 DEBUG:boto:Path: /hbmuog-55/c 2023-10-17T18:47:45.887 DEBUG:boto:Data: 2023-10-17T18:47:45.887 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.888 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.889 DEBUG:boto:Params: {} 2023-10-17T18:47:45.889 DEBUG:boto:Token: None 2023-10-17T18:47:45.890 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/c 2023-10-17T18:47:45.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY= 2023-10-17T18:47:45.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY=', 'Content-Length': '0'} 2023-10-17T18:47:45.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbe036514adc240-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.893 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.893 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.894 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.894 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.895 DEBUG:boto:Method: GET 2023-10-17T18:47:45.895 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:45.896 DEBUG:boto:Data: 2023-10-17T18:47:45.896 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.896 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.897 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.897 DEBUG:boto:Params: {} 2023-10-17T18:47:45.898 DEBUG:boto:Token: None 2023-10-17T18:47:45.898 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.899 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbc129615e22c38-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.902 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.903 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.903 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.904 DEBUG:boto:path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.904 DEBUG:boto:auth_path=/hbmuog-55/?prefix=c 2023-10-17T18:47:45.905 DEBUG:boto:Method: GET 2023-10-17T18:47:45.905 DEBUG:boto:Path: /hbmuog-55/?prefix=c 2023-10-17T18:47:45.905 DEBUG:boto:Data: 2023-10-17T18:47:45.906 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.907 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.907 DEBUG:boto:Params: {} 2023-10-17T18:47:45.908 DEBUG:boto:Token: None 2023-10-17T18:47:45.908 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.909 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6e76237462ae8c-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.912 DEBUG:boto:b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:45.913 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.913 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.914 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.914 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.915 DEBUG:boto:Method: GET 2023-10-17T18:47:45.915 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.916 DEBUG:boto:Data: 2023-10-17T18:47:45.916 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.917 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.917 DEBUG:boto:Params: {} 2023-10-17T18:47:45.918 DEBUG:boto:Token: None 2023-10-17T18:47:45.918 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T18:47:45.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T18:47:45.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067467a7feb109b2d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.922 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.922 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.923 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.923 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.924 DEBUG:boto:Method: GET 2023-10-17T18:47:45.924 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.924 DEBUG:boto:Data: 2023-10-17T18:47:45.925 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.926 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.926 DEBUG:boto:Params: {} 2023-10-17T18:47:45.926 DEBUG:boto:Token: None 2023-10-17T18:47:45.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T18:47:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T18:47:45.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ce24bfcc64e261-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.931 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.931 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.931 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.932 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.932 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.933 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.933 DEBUG:boto:Data: 2023-10-17T18:47:45.933 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.934 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.935 DEBUG:boto:Params: {} 2023-10-17T18:47:45.935 DEBUG:boto:Token: None 2023-10-17T18:47:45.935 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU= 2023-10-17T18:47:45.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU=', 'Content-Length': '0'} 2023-10-17T18:47:45.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70b41858bcaf99f-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.939 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.939 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.940 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.940 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.940 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.941 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.941 DEBUG:boto:Data: 2023-10-17T18:47:45.942 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.943 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.943 DEBUG:boto:Params: {} 2023-10-17T18:47:45.944 DEBUG:boto:Token: None 2023-10-17T18:47:45.944 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU= 2023-10-17T18:47:45.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU=', 'Content-Length': '0'} 2023-10-17T18:47:45.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2743754ee007a3-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:45.948 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.948 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.949 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.949 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.950 DEBUG:boto:Method: GET 2023-10-17T18:47:45.950 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.951 DEBUG:boto:Data: 2023-10-17T18:47:45.951 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.951 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.952 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.952 DEBUG:boto:Params: {} 2023-10-17T18:47:45.953 DEBUG:boto:Token: None 2023-10-17T18:47:45.953 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T18:47:45.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T18:47:45.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a44b44f6036c4f-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.957 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.957 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.957 DEBUG:boto:path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.958 DEBUG:boto:auth_path=/hbmuog-55/d?versionId=null 2023-10-17T18:47:45.958 DEBUG:boto:Method: GET 2023-10-17T18:47:45.959 DEBUG:boto:Path: /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.959 DEBUG:boto:Data: 2023-10-17T18:47:45.960 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.960 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.961 DEBUG:boto:Params: {} 2023-10-17T18:47:45.961 DEBUG:boto:Token: None 2023-10-17T18:47:45.962 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d?versionId=null 2023-10-17T18:47:45.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T18:47:45.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T18:47:45.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e5b84201b1b33f-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.965 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.966 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.966 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:45.967 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:45.967 DEBUG:boto:Method: GET 2023-10-17T18:47:45.968 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:45.968 DEBUG:boto:Data: 2023-10-17T18:47:45.969 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.969 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.970 DEBUG:boto:Params: {} 2023-10-17T18:47:45.970 DEBUG:boto:Token: None 2023-10-17T18:47:45.971 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.971 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea471cdc1c82049-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.975 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.975 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:45.976 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:45.976 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:45.977 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:45.977 DEBUG:boto:Method: GET 2023-10-17T18:47:45.978 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:45.978 DEBUG:boto:Data: 2023-10-17T18:47:45.979 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.979 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.979 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.980 DEBUG:boto:Params: {} 2023-10-17T18:47:45.980 DEBUG:boto:Token: None 2023-10-17T18:47:45.981 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/ 2023-10-17T18:47:45.981 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T18:47:45.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T18:47:45.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a193a557426003a-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.984 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:45.985 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.985 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.986 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.986 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:45.987 DEBUG:boto:Data: 2023-10-17T18:47:45.987 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.988 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:45.988 DEBUG:boto:Port: 8001 2023-10-17T18:47:45.988 DEBUG:boto:Params: {} 2023-10-17T18:47:45.989 DEBUG:boto:Token: None 2023-10-17T18:47:45.989 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d 2023-10-17T18:47:45.990 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w= 2023-10-17T18:47:45.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w=', 'Content-Length': '0'} 2023-10-17T18:47:45.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d865a24e8f8b648-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:45.993 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:45.993 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:45.994 DEBUG:boto:Method: HEAD 2023-10-17T18:47:45.994 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:45.994 DEBUG:boto:Data: 2023-10-17T18:47:45.995 DEBUG:boto:Headers: {} 2023-10-17T18:47:45.995 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:45.996 DEBUG:boto:Port: 8000 2023-10-17T18:47:45.996 DEBUG:boto:Params: {} 2023-10-17T18:47:45.996 DEBUG:boto:Token: None 2023-10-17T18:47:45.997 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:45 GMT /hbmuog-55/d 2023-10-17T18:47:45.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w= 2023-10-17T18:47:45.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w=', 'Content-Length': '0'} 2023-10-17T18:47:46.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1657d90769aa48b-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:47:46.001 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:46.001 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:46.001 DEBUG:boto:Method: GET 2023-10-17T18:47:46.002 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:46.002 DEBUG:boto:Data: 2023-10-17T18:47:46.003 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.004 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.004 DEBUG:boto:Params: {} 2023-10-17T18:47:46.005 DEBUG:boto:Token: None 2023-10-17T18:47:46.005 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/d 2023-10-17T18:47:46.006 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM= 2023-10-17T18:47:46.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM=', 'Content-Length': '0'} 2023-10-17T18:47:46.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047cfbdd590327078-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.054 DEBUG:boto:path=/hbmuog-55/d 2023-10-17T18:47:46.055 DEBUG:boto:auth_path=/hbmuog-55/d 2023-10-17T18:47:46.055 DEBUG:boto:Method: GET 2023-10-17T18:47:46.056 DEBUG:boto:Path: /hbmuog-55/d 2023-10-17T18:47:46.056 DEBUG:boto:Data: 2023-10-17T18:47:46.057 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.058 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.058 DEBUG:boto:Params: {} 2023-10-17T18:47:46.059 DEBUG:boto:Token: None 2023-10-17T18:47:46.059 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/d 2023-10-17T18:47:46.060 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM= 2023-10-17T18:47:46.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM=', 'Content-Length': '0'} 2023-10-17T18:47:46.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2583f90960c04fb-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.063 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.063 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.064 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:46.064 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:46.065 DEBUG:boto:Method: GET 2023-10-17T18:47:46.065 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:46.066 DEBUG:boto:Data: 2023-10-17T18:47:46.066 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.067 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.067 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.068 DEBUG:boto:Params: {} 2023-10-17T18:47:46.068 DEBUG:boto:Token: None 2023-10-17T18:47:46.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.069 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d27deff34a54f0a-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.072 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.073 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.073 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.074 DEBUG:boto:path=/hbmuog-55/?prefix=d 2023-10-17T18:47:46.074 DEBUG:boto:auth_path=/hbmuog-55/?prefix=d 2023-10-17T18:47:46.075 DEBUG:boto:Method: GET 2023-10-17T18:47:46.075 DEBUG:boto:Path: /hbmuog-55/?prefix=d 2023-10-17T18:47:46.076 DEBUG:boto:Data: 2023-10-17T18:47:46.076 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.077 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.077 DEBUG:boto:Params: {} 2023-10-17T18:47:46.078 DEBUG:boto:Token: None 2023-10-17T18:47:46.078 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e57a306fdd8d81-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.122 DEBUG:boto:b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:46.124 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.124 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.124 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.125 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.125 DEBUG:boto:Method: GET 2023-10-17T18:47:46.126 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.126 DEBUG:boto:Data: 2023-10-17T18:47:46.127 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.128 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.128 DEBUG:boto:Params: {} 2023-10-17T18:47:46.128 DEBUG:boto:Token: None 2023-10-17T18:47:46.129 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.130 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T18:47:46.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T18:47:46.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abcd46ca349291cd-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.133 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.134 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.134 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.135 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.135 DEBUG:boto:Method: GET 2023-10-17T18:47:46.135 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.136 DEBUG:boto:Data: 2023-10-17T18:47:46.136 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.137 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.137 DEBUG:boto:Params: {} 2023-10-17T18:47:46.138 DEBUG:boto:Token: None 2023-10-17T18:47:46.139 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T18:47:46.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T18:47:46.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000506cc5dee8b73fa-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.142 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.143 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.143 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.143 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.144 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.144 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.145 DEBUG:boto:Data: 2023-10-17T18:47:46.145 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.145 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.146 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.146 DEBUG:boto:Params: {} 2023-10-17T18:47:46.147 DEBUG:boto:Token: None 2023-10-17T18:47:46.147 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4= 2023-10-17T18:47:46.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4=', 'Content-Length': '0'} 2023-10-17T18:47:46.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5f63ed59b61af0-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.151 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.151 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.152 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.152 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.152 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.153 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.153 DEBUG:boto:Data: 2023-10-17T18:47:46.154 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.154 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.154 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.155 DEBUG:boto:Params: {} 2023-10-17T18:47:46.155 DEBUG:boto:Token: None 2023-10-17T18:47:46.156 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.156 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4= 2023-10-17T18:47:46.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4=', 'Content-Length': '0'} 2023-10-17T18:47:46.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe49dceb4d6ed33-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:46.160 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.160 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.161 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.161 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.161 DEBUG:boto:Method: GET 2023-10-17T18:47:46.162 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.162 DEBUG:boto:Data: 2023-10-17T18:47:46.163 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.163 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.164 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.164 DEBUG:boto:Params: {} 2023-10-17T18:47:46.164 DEBUG:boto:Token: None 2023-10-17T18:47:46.165 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.165 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T18:47:46.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T18:47:46.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5d7a08a67bb9ad-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.168 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.169 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.169 DEBUG:boto:path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.170 DEBUG:boto:auth_path=/hbmuog-55/e?versionId=null 2023-10-17T18:47:46.170 DEBUG:boto:Method: GET 2023-10-17T18:47:46.171 DEBUG:boto:Path: /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.171 DEBUG:boto:Data: 2023-10-17T18:47:46.172 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.172 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.173 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.173 DEBUG:boto:Params: {} 2023-10-17T18:47:46.174 DEBUG:boto:Token: None 2023-10-17T18:47:46.174 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e?versionId=null 2023-10-17T18:47:46.175 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T18:47:46.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T18:47:46.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34f35f11d8353e6-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.178 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.178 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.179 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.179 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.179 DEBUG:boto:Method: GET 2023-10-17T18:47:46.180 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:46.180 DEBUG:boto:Data: 2023-10-17T18:47:46.181 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.181 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.182 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.182 DEBUG:boto:Params: {} 2023-10-17T18:47:46.183 DEBUG:boto:Token: None 2023-10-17T18:47:46.183 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.184 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e29aaa68a8ac21-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.187 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.188 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.188 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.189 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.189 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.190 DEBUG:boto:Method: GET 2023-10-17T18:47:46.190 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:46.191 DEBUG:boto:Data: 2023-10-17T18:47:46.191 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.191 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.192 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.192 DEBUG:boto:Params: {} 2023-10-17T18:47:46.193 DEBUG:boto:Token: None 2023-10-17T18:47:46.193 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2905cee1d55bd7-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.197 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.197 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.198 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.198 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.199 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:46.199 DEBUG:boto:Data: 2023-10-17T18:47:46.200 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.200 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.200 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.201 DEBUG:boto:Params: {} 2023-10-17T18:47:46.201 DEBUG:boto:Token: None 2023-10-17T18:47:46.202 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e 2023-10-17T18:47:46.202 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg= 2023-10-17T18:47:46.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg=', 'Content-Length': '0'} 2023-10-17T18:47:46.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e026a8cd13916a50-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.205 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.206 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.206 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.207 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:46.207 DEBUG:boto:Data: 2023-10-17T18:47:46.207 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.208 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.209 DEBUG:boto:Params: {} 2023-10-17T18:47:46.209 DEBUG:boto:Token: None 2023-10-17T18:47:46.209 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e 2023-10-17T18:47:46.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg= 2023-10-17T18:47:46.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg=', 'Content-Length': '0'} 2023-10-17T18:47:46.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f97bf7728d78f0a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:47:46.213 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.214 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.214 DEBUG:boto:Method: GET 2023-10-17T18:47:46.215 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:46.215 DEBUG:boto:Data: 2023-10-17T18:47:46.215 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.216 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.216 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.217 DEBUG:boto:Params: {} 2023-10-17T18:47:46.217 DEBUG:boto:Token: None 2023-10-17T18:47:46.217 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e 2023-10-17T18:47:46.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc= 2023-10-17T18:47:46.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc=', 'Content-Length': '0'} 2023-10-17T18:47:46.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c7b888552ba585-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.221 DEBUG:boto:path=/hbmuog-55/e 2023-10-17T18:47:46.221 DEBUG:boto:auth_path=/hbmuog-55/e 2023-10-17T18:47:46.222 DEBUG:boto:Method: GET 2023-10-17T18:47:46.222 DEBUG:boto:Path: /hbmuog-55/e 2023-10-17T18:47:46.223 DEBUG:boto:Data: 2023-10-17T18:47:46.223 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.224 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.224 DEBUG:boto:Params: {} 2023-10-17T18:47:46.225 DEBUG:boto:Token: None 2023-10-17T18:47:46.225 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/e 2023-10-17T18:47:46.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc= 2023-10-17T18:47:46.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc=', 'Content-Length': '0'} 2023-10-17T18:47:46.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e86dd4b364c619-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.229 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.229 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.230 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.230 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.231 DEBUG:boto:Method: GET 2023-10-17T18:47:46.231 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:46.231 DEBUG:boto:Data: 2023-10-17T18:47:46.232 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.233 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.233 DEBUG:boto:Params: {} 2023-10-17T18:47:46.234 DEBUG:boto:Token: None 2023-10-17T18:47:46.234 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1016a2223f25b7-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.238 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.238 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.239 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.239 DEBUG:boto:path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.240 DEBUG:boto:auth_path=/hbmuog-55/?prefix=e 2023-10-17T18:47:46.240 DEBUG:boto:Method: GET 2023-10-17T18:47:46.240 DEBUG:boto:Path: /hbmuog-55/?prefix=e 2023-10-17T18:47:46.241 DEBUG:boto:Data: 2023-10-17T18:47:46.241 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.242 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.242 DEBUG:boto:Params: {} 2023-10-17T18:47:46.243 DEBUG:boto:Token: None 2023-10-17T18:47:46.243 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003579807aa1de8d0e-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.247 DEBUG:boto:b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:46.248 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.248 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.249 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.249 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.249 DEBUG:boto:Method: GET 2023-10-17T18:47:46.250 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.250 DEBUG:boto:Data: 2023-10-17T18:47:46.251 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.252 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.252 DEBUG:boto:Params: {} 2023-10-17T18:47:46.253 DEBUG:boto:Token: None 2023-10-17T18:47:46.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.254 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T18:47:46.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T18:47:46.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f91dad0835db16-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.257 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.257 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.258 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.258 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.258 DEBUG:boto:Method: GET 2023-10-17T18:47:46.259 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.259 DEBUG:boto:Data: 2023-10-17T18:47:46.260 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.261 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.261 DEBUG:boto:Params: {} 2023-10-17T18:47:46.262 DEBUG:boto:Token: None 2023-10-17T18:47:46.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T18:47:46.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T18:47:46.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da9da4d9dcf7509-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.306 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.307 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.307 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.308 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.308 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.308 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.309 DEBUG:boto:Data: 2023-10-17T18:47:46.309 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.310 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.310 DEBUG:boto:Params: {} 2023-10-17T18:47:46.311 DEBUG:boto:Token: None 2023-10-17T18:47:46.311 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ= 2023-10-17T18:47:46.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ=', 'Content-Length': '0'} 2023-10-17T18:47:46.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae1bcb9c8d51f95-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.315 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.315 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.316 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.316 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.316 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.317 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.317 DEBUG:boto:Data: 2023-10-17T18:47:46.318 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.318 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.319 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.319 DEBUG:boto:Params: {} 2023-10-17T18:47:46.319 DEBUG:boto:Token: None 2023-10-17T18:47:46.320 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ= 2023-10-17T18:47:46.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ=', 'Content-Length': '0'} 2023-10-17T18:47:46.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9f2ccc65f2653a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:46.324 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.324 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.324 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.325 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.325 DEBUG:boto:Method: GET 2023-10-17T18:47:46.326 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.326 DEBUG:boto:Data: 2023-10-17T18:47:46.326 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.327 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.328 DEBUG:boto:Params: {} 2023-10-17T18:47:46.328 DEBUG:boto:Token: None 2023-10-17T18:47:46.329 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T18:47:46.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T18:47:46.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e616d9ea347d06-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.332 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.333 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.333 DEBUG:boto:path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.333 DEBUG:boto:auth_path=/hbmuog-55/f?versionId=null 2023-10-17T18:47:46.334 DEBUG:boto:Method: GET 2023-10-17T18:47:46.334 DEBUG:boto:Path: /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.335 DEBUG:boto:Data: 2023-10-17T18:47:46.335 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.336 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.336 DEBUG:boto:Params: {} 2023-10-17T18:47:46.337 DEBUG:boto:Token: None 2023-10-17T18:47:46.337 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f?versionId=null 2023-10-17T18:47:46.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T18:47:46.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T18:47:46.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022398fa3d924d629-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.341 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.341 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.342 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.342 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.343 DEBUG:boto:Method: GET 2023-10-17T18:47:46.343 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:46.344 DEBUG:boto:Data: 2023-10-17T18:47:46.344 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.345 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.345 DEBUG:boto:Params: {} 2023-10-17T18:47:46.346 DEBUG:boto:Token: None 2023-10-17T18:47:46.346 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6146685d25fa0d0-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.350 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.351 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.351 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.352 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.352 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.353 DEBUG:boto:Method: GET 2023-10-17T18:47:46.353 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:46.354 DEBUG:boto:Data: 2023-10-17T18:47:46.354 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.355 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.355 DEBUG:boto:Params: {} 2023-10-17T18:47:46.356 DEBUG:boto:Token: None 2023-10-17T18:47:46.356 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cbb415a00944de9-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.360 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.360 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.361 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.361 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.362 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:46.362 DEBUG:boto:Data: 2023-10-17T18:47:46.362 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.363 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.364 DEBUG:boto:Params: {} 2023-10-17T18:47:46.365 DEBUG:boto:Token: None 2023-10-17T18:47:46.366 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f 2023-10-17T18:47:46.366 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0= 2023-10-17T18:47:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0=', 'Content-Length': '0'} 2023-10-17T18:47:46.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08f11755f4d40d2-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.369 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.370 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.370 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.370 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:46.371 DEBUG:boto:Data: 2023-10-17T18:47:46.371 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.372 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.373 DEBUG:boto:Params: {} 2023-10-17T18:47:46.373 DEBUG:boto:Token: None 2023-10-17T18:47:46.373 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f 2023-10-17T18:47:46.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0= 2023-10-17T18:47:46.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0=', 'Content-Length': '0'} 2023-10-17T18:47:46.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd97810354d2e52c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:47:46.377 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.378 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.378 DEBUG:boto:Method: GET 2023-10-17T18:47:46.379 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:46.379 DEBUG:boto:Data: 2023-10-17T18:47:46.380 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.380 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.381 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.381 DEBUG:boto:Params: {} 2023-10-17T18:47:46.381 DEBUG:boto:Token: None 2023-10-17T18:47:46.382 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f 2023-10-17T18:47:46.382 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY= 2023-10-17T18:47:46.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY=', 'Content-Length': '0'} 2023-10-17T18:47:46.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf68f9960c12fa88-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.385 DEBUG:boto:path=/hbmuog-55/f 2023-10-17T18:47:46.386 DEBUG:boto:auth_path=/hbmuog-55/f 2023-10-17T18:47:46.386 DEBUG:boto:Method: GET 2023-10-17T18:47:46.387 DEBUG:boto:Path: /hbmuog-55/f 2023-10-17T18:47:46.387 DEBUG:boto:Data: 2023-10-17T18:47:46.388 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.388 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.389 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.389 DEBUG:boto:Params: {} 2023-10-17T18:47:46.390 DEBUG:boto:Token: None 2023-10-17T18:47:46.390 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/f 2023-10-17T18:47:46.391 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY= 2023-10-17T18:47:46.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY=', 'Content-Length': '0'} 2023-10-17T18:47:46.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008183b595d6004e1a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.393 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.394 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.394 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.395 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.395 DEBUG:boto:Method: GET 2023-10-17T18:47:46.396 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:46.396 DEBUG:boto:Data: 2023-10-17T18:47:46.396 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.397 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.398 DEBUG:boto:Params: {} 2023-10-17T18:47:46.398 DEBUG:boto:Token: None 2023-10-17T18:47:46.399 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b725ad6438a1559a-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.403 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.403 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.404 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.404 DEBUG:boto:path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.405 DEBUG:boto:auth_path=/hbmuog-55/?prefix=f 2023-10-17T18:47:46.405 DEBUG:boto:Method: GET 2023-10-17T18:47:46.406 DEBUG:boto:Path: /hbmuog-55/?prefix=f 2023-10-17T18:47:46.406 DEBUG:boto:Data: 2023-10-17T18:47:46.406 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.407 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.407 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.408 DEBUG:boto:Params: {} 2023-10-17T18:47:46.408 DEBUG:boto:Token: None 2023-10-17T18:47:46.409 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.409 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7ac35083727adb-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.412 DEBUG:boto:b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:46.413 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.414 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.414 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.415 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.415 DEBUG:boto:Method: GET 2023-10-17T18:47:46.416 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.416 DEBUG:boto:Data: 2023-10-17T18:47:46.416 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.417 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.418 DEBUG:boto:Params: {} 2023-10-17T18:47:46.418 DEBUG:boto:Token: None 2023-10-17T18:47:46.419 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.419 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T18:47:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T18:47:46.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091fe22073ee20df6-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.462 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.463 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.463 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.464 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.464 DEBUG:boto:Method: GET 2023-10-17T18:47:46.465 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.465 DEBUG:boto:Data: 2023-10-17T18:47:46.466 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.466 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.467 DEBUG:boto:Params: {} 2023-10-17T18:47:46.467 DEBUG:boto:Token: None 2023-10-17T18:47:46.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T18:47:46.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T18:47:46.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caf522369b6127ec-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.472 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.472 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.472 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.473 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.473 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.474 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.474 DEBUG:boto:Data: 2023-10-17T18:47:46.475 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.475 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.475 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.476 DEBUG:boto:Params: {} 2023-10-17T18:47:46.476 DEBUG:boto:Token: None 2023-10-17T18:47:46.477 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.477 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU= 2023-10-17T18:47:46.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU=', 'Content-Length': '0'} 2023-10-17T18:47:46.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb9290ec389f004-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.480 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.481 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.481 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.482 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.482 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.483 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.483 DEBUG:boto:Data: 2023-10-17T18:47:46.483 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.484 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.484 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.485 DEBUG:boto:Params: {} 2023-10-17T18:47:46.485 DEBUG:boto:Token: None 2023-10-17T18:47:46.486 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.486 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU= 2023-10-17T18:47:46.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU=', 'Content-Length': '0'} 2023-10-17T18:47:46.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a15bfd46ba6b63-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:46.489 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.490 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.490 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.491 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.491 DEBUG:boto:Method: GET 2023-10-17T18:47:46.491 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.492 DEBUG:boto:Data: 2023-10-17T18:47:46.492 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.493 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.494 DEBUG:boto:Params: {} 2023-10-17T18:47:46.494 DEBUG:boto:Token: None 2023-10-17T18:47:46.494 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.495 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T18:47:46.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T18:47:46.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ead69aa1b4c8f1-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.498 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.498 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.499 DEBUG:boto:path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.499 DEBUG:boto:auth_path=/hbmuog-55/g?versionId=null 2023-10-17T18:47:46.499 DEBUG:boto:Method: GET 2023-10-17T18:47:46.500 DEBUG:boto:Path: /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.500 DEBUG:boto:Data: 2023-10-17T18:47:46.501 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.501 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.502 DEBUG:boto:Params: {} 2023-10-17T18:47:46.502 DEBUG:boto:Token: None 2023-10-17T18:47:46.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g?versionId=null 2023-10-17T18:47:46.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T18:47:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T18:47:46.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483daeeb72586345-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.546 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.547 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.547 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.547 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.548 DEBUG:boto:Method: GET 2023-10-17T18:47:46.548 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:46.549 DEBUG:boto:Data: 2023-10-17T18:47:46.549 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.550 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.550 DEBUG:boto:Params: {} 2023-10-17T18:47:46.551 DEBUG:boto:Token: None 2023-10-17T18:47:46.551 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b6d328cce702822-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.555 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.556 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.556 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.557 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.557 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.558 DEBUG:boto:Method: GET 2023-10-17T18:47:46.558 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:46.559 DEBUG:boto:Data: 2023-10-17T18:47:46.559 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.560 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.560 DEBUG:boto:Params: {} 2023-10-17T18:47:46.561 DEBUG:boto:Token: None 2023-10-17T18:47:46.561 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf06eb6cc27ba5a4-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.565 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.565 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.566 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.566 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.567 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:46.567 DEBUG:boto:Data: 2023-10-17T18:47:46.567 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.568 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.569 DEBUG:boto:Params: {} 2023-10-17T18:47:46.569 DEBUG:boto:Token: None 2023-10-17T18:47:46.569 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g 2023-10-17T18:47:46.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI= 2023-10-17T18:47:46.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI=', 'Content-Length': '0'} 2023-10-17T18:47:46.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4681e75edb1f659-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.573 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.573 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.573 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.574 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:46.574 DEBUG:boto:Data: 2023-10-17T18:47:46.575 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.575 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.576 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.576 DEBUG:boto:Params: {} 2023-10-17T18:47:46.577 DEBUG:boto:Token: None 2023-10-17T18:47:46.577 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g 2023-10-17T18:47:46.577 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI= 2023-10-17T18:47:46.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI=', 'Content-Length': '0'} 2023-10-17T18:47:46.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080efcff4fcbecb9e-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:47:46.580 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.581 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.581 DEBUG:boto:Method: GET 2023-10-17T18:47:46.582 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:46.582 DEBUG:boto:Data: 2023-10-17T18:47:46.583 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.583 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.583 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.584 DEBUG:boto:Params: {} 2023-10-17T18:47:46.584 DEBUG:boto:Token: None 2023-10-17T18:47:46.585 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g 2023-10-17T18:47:46.585 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A= 2023-10-17T18:47:46.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A=', 'Content-Length': '0'} 2023-10-17T18:47:46.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ffb02d111155e4-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.588 DEBUG:boto:path=/hbmuog-55/g 2023-10-17T18:47:46.588 DEBUG:boto:auth_path=/hbmuog-55/g 2023-10-17T18:47:46.589 DEBUG:boto:Method: GET 2023-10-17T18:47:46.589 DEBUG:boto:Path: /hbmuog-55/g 2023-10-17T18:47:46.590 DEBUG:boto:Data: 2023-10-17T18:47:46.590 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.591 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.592 DEBUG:boto:Params: {} 2023-10-17T18:47:46.592 DEBUG:boto:Token: None 2023-10-17T18:47:46.592 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/g 2023-10-17T18:47:46.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A= 2023-10-17T18:47:46.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A=', 'Content-Length': '0'} 2023-10-17T18:47:46.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010181189f7ef020c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.596 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.596 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.597 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.597 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.598 DEBUG:boto:Method: GET 2023-10-17T18:47:46.598 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:46.599 DEBUG:boto:Data: 2023-10-17T18:47:46.599 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.600 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.600 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.601 DEBUG:boto:Params: {} 2023-10-17T18:47:46.601 DEBUG:boto:Token: None 2023-10-17T18:47:46.601 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.602 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c24d642dc4c93a9-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.605 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.606 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.606 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.607 DEBUG:boto:path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.607 DEBUG:boto:auth_path=/hbmuog-55/?prefix=g 2023-10-17T18:47:46.607 DEBUG:boto:Method: GET 2023-10-17T18:47:46.608 DEBUG:boto:Path: /hbmuog-55/?prefix=g 2023-10-17T18:47:46.608 DEBUG:boto:Data: 2023-10-17T18:47:46.608 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.609 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.610 DEBUG:boto:Params: {} 2023-10-17T18:47:46.610 DEBUG:boto:Token: None 2023-10-17T18:47:46.610 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.611 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb76d5d49fce50b-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.614 DEBUG:boto:b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:46.615 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.615 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.615 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.616 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.616 DEBUG:boto:Method: GET 2023-10-17T18:47:46.617 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.617 DEBUG:boto:Data: 2023-10-17T18:47:46.617 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.618 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.618 DEBUG:boto:Params: {} 2023-10-17T18:47:46.619 DEBUG:boto:Token: None 2023-10-17T18:47:46.619 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T18:47:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T18:47:46.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd990e2ac0a770c-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.623 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.623 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.624 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.624 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.625 DEBUG:boto:Method: GET 2023-10-17T18:47:46.625 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.625 DEBUG:boto:Data: 2023-10-17T18:47:46.626 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.627 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.627 DEBUG:boto:Params: {} 2023-10-17T18:47:46.628 DEBUG:boto:Token: None 2023-10-17T18:47:46.628 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T18:47:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T18:47:46.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b0ae97fc6bfbc5-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.632 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.632 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.633 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.633 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.633 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.634 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.634 DEBUG:boto:Data: 2023-10-17T18:47:46.635 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.635 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.636 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.636 DEBUG:boto:Params: {} 2023-10-17T18:47:46.636 DEBUG:boto:Token: None 2023-10-17T18:47:46.637 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw= 2023-10-17T18:47:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw=', 'Content-Length': '0'} 2023-10-17T18:47:46.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000074a932944dccdfa-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.640 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.640 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.641 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.641 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.642 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.642 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.642 DEBUG:boto:Data: 2023-10-17T18:47:46.643 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.644 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.644 DEBUG:boto:Params: {} 2023-10-17T18:47:46.644 DEBUG:boto:Token: None 2023-10-17T18:47:46.645 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw= 2023-10-17T18:47:46.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw=', 'Content-Length': '0'} 2023-10-17T18:47:46.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d297f060a2bf95-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:46.649 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.649 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.650 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.650 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.650 DEBUG:boto:Method: GET 2023-10-17T18:47:46.651 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.651 DEBUG:boto:Data: 2023-10-17T18:47:46.652 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.652 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.653 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.653 DEBUG:boto:Params: {} 2023-10-17T18:47:46.654 DEBUG:boto:Token: None 2023-10-17T18:47:46.654 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T18:47:46.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T18:47:46.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f250f3b37829dc4-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.698 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.699 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.699 DEBUG:boto:path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.700 DEBUG:boto:auth_path=/hbmuog-55/h?versionId=null 2023-10-17T18:47:46.700 DEBUG:boto:Method: GET 2023-10-17T18:47:46.701 DEBUG:boto:Path: /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.701 DEBUG:boto:Data: 2023-10-17T18:47:46.701 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.702 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.703 DEBUG:boto:Params: {} 2023-10-17T18:47:46.703 DEBUG:boto:Token: None 2023-10-17T18:47:46.704 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h?versionId=null 2023-10-17T18:47:46.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T18:47:46.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T18:47:46.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794f8af1451982fc-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.707 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.708 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.708 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.709 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.709 DEBUG:boto:Method: GET 2023-10-17T18:47:46.709 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:46.710 DEBUG:boto:Data: 2023-10-17T18:47:46.710 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.711 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.711 DEBUG:boto:Params: {} 2023-10-17T18:47:46.712 DEBUG:boto:Token: None 2023-10-17T18:47:46.712 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099929e60eb6fd41b-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.717 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.718 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.718 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.719 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.719 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.720 DEBUG:boto:Method: GET 2023-10-17T18:47:46.720 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:46.720 DEBUG:boto:Data: 2023-10-17T18:47:46.721 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.722 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.722 DEBUG:boto:Params: {} 2023-10-17T18:47:46.723 DEBUG:boto:Token: None 2023-10-17T18:47:46.723 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e4de7a0e00e2156-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.770 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.771 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.772 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.772 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.772 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:46.773 DEBUG:boto:Data: 2023-10-17T18:47:46.773 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.774 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.775 DEBUG:boto:Params: {} 2023-10-17T18:47:46.775 DEBUG:boto:Token: None 2023-10-17T18:47:46.776 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h 2023-10-17T18:47:46.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k= 2023-10-17T18:47:46.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k=', 'Content-Length': '0'} 2023-10-17T18:47:46.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001948caa28ebc160e-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.779 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.779 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.780 DEBUG:boto:Method: HEAD 2023-10-17T18:47:46.780 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:46.781 DEBUG:boto:Data: 2023-10-17T18:47:46.781 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.781 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.782 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.782 DEBUG:boto:Params: {} 2023-10-17T18:47:46.783 DEBUG:boto:Token: None 2023-10-17T18:47:46.783 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h 2023-10-17T18:47:46.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k= 2023-10-17T18:47:46.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k=', 'Content-Length': '0'} 2023-10-17T18:47:46.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad8255c4d993591c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:47:46.787 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.787 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.788 DEBUG:boto:Method: GET 2023-10-17T18:47:46.788 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:46.789 DEBUG:boto:Data: 2023-10-17T18:47:46.789 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.790 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.790 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.790 DEBUG:boto:Params: {} 2023-10-17T18:47:46.791 DEBUG:boto:Token: None 2023-10-17T18:47:46.791 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h 2023-10-17T18:47:46.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0= 2023-10-17T18:47:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0=', 'Content-Length': '0'} 2023-10-17T18:47:46.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec26ef752fcddb2f-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.795 DEBUG:boto:path=/hbmuog-55/h 2023-10-17T18:47:46.795 DEBUG:boto:auth_path=/hbmuog-55/h 2023-10-17T18:47:46.796 DEBUG:boto:Method: GET 2023-10-17T18:47:46.796 DEBUG:boto:Path: /hbmuog-55/h 2023-10-17T18:47:46.797 DEBUG:boto:Data: 2023-10-17T18:47:46.797 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.798 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.798 DEBUG:boto:Params: {} 2023-10-17T18:47:46.799 DEBUG:boto:Token: None 2023-10-17T18:47:46.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/h 2023-10-17T18:47:46.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0= 2023-10-17T18:47:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0=', 'Content-Length': '0'} 2023-10-17T18:47:46.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041797f1099257606-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.803 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.803 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.804 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.804 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.804 DEBUG:boto:Method: GET 2023-10-17T18:47:46.805 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:46.805 DEBUG:boto:Data: 2023-10-17T18:47:46.806 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:47:46.807 DEBUG:boto:Port: 8001 2023-10-17T18:47:46.807 DEBUG:boto:Params: {} 2023-10-17T18:47:46.807 DEBUG:boto:Token: None 2023-10-17T18:47:46.808 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.808 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a590c5cef114d1da-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.858 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.876 DEBUG:boto:path=/hbmuog-55/ 2023-10-17T18:47:46.876 DEBUG:boto:auth_path=/hbmuog-55/ 2023-10-17T18:47:46.877 DEBUG:boto:path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.877 DEBUG:boto:auth_path=/hbmuog-55/?prefix=h 2023-10-17T18:47:46.877 DEBUG:boto:Method: GET 2023-10-17T18:47:46.878 DEBUG:boto:Path: /hbmuog-55/?prefix=h 2023-10-17T18:47:46.878 DEBUG:boto:Data: 2023-10-17T18:47:46.879 DEBUG:boto:Headers: {} 2023-10-17T18:47:46.879 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:46.880 DEBUG:boto:Port: 8000 2023-10-17T18:47:46.880 DEBUG:boto:Params: {} 2023-10-17T18:47:46.881 DEBUG:boto:Token: None 2023-10-17T18:47:46.881 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:47:46 GMT /hbmuog-55/ 2023-10-17T18:47:46.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T18:47:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T18:47:46.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb055c670e01e3a-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:46.885 DEBUG:boto:b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:47:46.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T18:47:46.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:46.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:46.887 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:46.979 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:46.979+0000 7f5e948b0700 -1 req 4164692305357405680 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:46.980 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:46.979+0000 7f5ea78d6700 -1 req 8993893449878962159 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.031 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.031+0000 7f5e9a0bb700 -1 req 16793100318319100887 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.032 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.031+0000 7f5e7987a700 -1 req 2406639096585865596 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.076 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.076+0000 7f5e57035700 -1 req 278019542600793309 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.081 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.081+0000 7f5e4f826700 -1 req 3692153941252346995 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.140 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:47.187 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:47.187 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:47.232 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:47:47.232 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.232 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding": "None", 2023-10-17T18:47:47.233 INFO:teuthology.orchestra.run.smithi049.stdout: "current_index": { 2023-10-17T18:47:47.233 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.233 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "Normal", 2023-10-17T18:47:47.234 INFO:teuthology.orchestra.run.smithi049.stdout: "normal": { 2023-10-17T18:47:47.234 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.234 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.234 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.235 INFO:teuthology.orchestra.run.smithi049.stdout: "logs": [ 2023-10-17T18:47:47.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.235 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:47.236 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.236 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:47.236 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:47.236 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:47.237 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.237 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.237 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.238 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.238 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.238 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.239 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:47.239 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:47.239 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:47:47.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.241 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:47:47.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.243 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:47.291 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:47.332 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.331+0000 7f5e67055700 -1 req 15020052301451087982 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.332 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.331+0000 7f5e40808700 -1 req 7782735937252187366 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.332 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.331+0000 7f5e5c03f700 -1 req 5516016212627392655 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.333 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.331+0000 7f5e54830700 -1 req 14088950252049687166 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.333 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.333+0000 7f5e31feb700 -1 req 7998560759297130222 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.334 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.333+0000 7f5e46814700 -1 req 16492183496669393951 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.334 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.333+0000 7f5e227cc700 -1 req 3758129637006002652 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.334 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.333+0000 7f5e3e003700 -1 req 17395515403808682674 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.427 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.426+0000 7f5e2c7e0700 -1 req 7477457638819705725 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.428 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.427+0000 7f5e277d6700 -1 req 8245506584526227205 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.428 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:47:47.427+0000 7f5e1c7c0700 -1 req 7234563436123005083 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T18:47:47.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:47:47.488 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:47:47.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:47.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:47.584 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:47:47.584 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.585 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding": "None", 2023-10-17T18:47:47.585 INFO:teuthology.orchestra.run.smithi049.stdout: "current_index": { 2023-10-17T18:47:47.585 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.585 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "Normal", 2023-10-17T18:47:47.586 INFO:teuthology.orchestra.run.smithi049.stdout: "normal": { 2023-10-17T18:47:47.586 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.586 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.587 INFO:teuthology.orchestra.run.smithi049.stdout: "logs": [ 2023-10-17T18:47:47.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.588 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:47.588 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.588 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:47.588 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:47.589 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 1, 2023-10-17T18:47:47.589 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.589 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.589 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.590 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.591 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "InIndex", 2023-10-17T18:47:47.591 INFO:teuthology.orchestra.run.smithi049.stdout: "in_index": { 2023-10-17T18:47:47.591 INFO:teuthology.orchestra.run.smithi049.stdout: "gen": 2, 2023-10-17T18:47:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: "layout": { 2023-10-17T18:47:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 3, 2023-10-17T18:47:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: "hash_type": "Mod" 2023-10-17T18:47:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.593 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:47:47.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:47:47.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-10-17T18:47:47.599 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:47:47.600 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:47:47.600 DEBUG:boto:Method: PUT 2023-10-17T18:47:47.601 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:47:47.601 DEBUG:boto:Data: 2023-10-17T18:47:47.601 DEBUG:boto:Headers: {} 2023-10-17T18:47:47.602 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.602 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.603 DEBUG:boto:Params: {} 2023-10-17T18:47:47.603 DEBUG:boto:Token: None 2023-10-17T18:47:47.604 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/ 2023-10-17T18:47:47.604 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3my1/Bu19Jbkio98a6GeiFF/b/Q= 2023-10-17T18:47:47.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3my1/Bu19Jbkio98a6GeiFF/b/Q=', 'Content-Length': '0'} 2023-10-17T18:47:47.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1da836e7a6ef12f-00652ed6d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.620 DEBUG:rgw_multi.tests:created bucket=hbmuog-56 2023-10-17T18:47:47.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:47:47.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:47:47.622 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:47:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:47.675 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:47.706 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:47:47.719 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:47:47.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.719 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568467.618297_664.1", 2023-10-17T18:47:47.720 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:47:47.618297Z" 2023-10-17T18:47:47.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:47:47.721 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:47:47.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.721 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:47:47.721 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:47:47.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.722 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:47:47.722 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:47:47.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.723 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:47:47.724 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568467.618297_664.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:47:47.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:47:47.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:47:47.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:47:47.726 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:47:47.776 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:47.776 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:47.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:47.804+0000 7ff74dbf2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:47:47.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:47.805+0000 7ff74dbf2780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:47:47.818 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:47:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:47:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:47:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:47:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:47:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:47:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:47:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:47:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:47:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:47:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:47:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568467.618297_664.1", 2023-10-17T18:47:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:47:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:47:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:47:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:47:47.618297Z", 2023-10-17T18:47:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:47:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:47:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:47:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:47:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:47:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:47:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:47:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:47:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:47:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:47:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:47:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:47:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:47:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:47:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:47:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:47:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:47:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:47:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:47:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:47:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:47:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:47:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:47:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:47:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:47:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:47:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:47:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:47:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:47:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:47:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:47:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:47:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:47:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:47:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:47:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:47:47Z" 2023-10-17T18:47:47.832 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:47:47.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568467.618297_664.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:47:47.618297Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:47:47Z" } 2023-10-17T18:47:47.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568467.618297_664.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:47:47.618297Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:47:47.834 DEBUG:rgw_multi.tests:log_status={0: '1_1697568467.618297_664.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:47:47.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568467.618297_664.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:47:47.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:47:47.836 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:47:47.836 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:47:47.836 DEBUG:boto:Method: HEAD 2023-10-17T18:47:47.837 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:47:47.837 DEBUG:boto:Data: 2023-10-17T18:47:47.838 DEBUG:boto:Headers: {} 2023-10-17T18:47:47.838 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.839 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.839 DEBUG:boto:Params: {} 2023-10-17T18:47:47.839 DEBUG:boto:Token: None 2023-10-17T18:47:47.840 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/ 2023-10-17T18:47:47.840 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E= 2023-10-17T18:47:47.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E=', 'Content-Length': '0'} 2023-10-17T18:47:47.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ddd5250631b6a3d-00652ed6d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.844 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:47:47.844 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:47:47.845 DEBUG:boto:Method: PUT 2023-10-17T18:47:47.845 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:47:47.846 DEBUG:boto:Data: 2023-10-17T18:47:47.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:47:47.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.847 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.847 DEBUG:boto:Params: {} 2023-10-17T18:47:47.848 DEBUG:boto:Token: None 2023-10-17T18:47:47.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/a 2023-10-17T18:47:47.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zDVd9NRgZSSMTLIgPH5YxHIykVA= 2023-10-17T18:47:47.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zDVd9NRgZSSMTLIgPH5YxHIykVA='} 2023-10-17T18:47:47.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dff8f1592180f31-00652ed6d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.855 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:47:47.856 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:47:47.856 DEBUG:boto:Method: HEAD 2023-10-17T18:47:47.857 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:47:47.857 DEBUG:boto:Data: 2023-10-17T18:47:47.857 DEBUG:boto:Headers: {} 2023-10-17T18:47:47.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.858 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.859 DEBUG:boto:Params: {} 2023-10-17T18:47:47.859 DEBUG:boto:Token: None 2023-10-17T18:47:47.859 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/ 2023-10-17T18:47:47.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E= 2023-10-17T18:47:47.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E=', 'Content-Length': '0'} 2023-10-17T18:47:47.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020ecab6da0233b38-00652ed6d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.863 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:47:47.864 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:47:47.864 DEBUG:boto:Method: PUT 2023-10-17T18:47:47.865 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:47:47.865 DEBUG:boto:Data: 2023-10-17T18:47:47.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:47:47.866 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.866 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.867 DEBUG:boto:Params: {} 2023-10-17T18:47:47.867 DEBUG:boto:Token: None 2023-10-17T18:47:47.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/b 2023-10-17T18:47:47.868 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hsh/1N50tWhpGVUrFKZVJw3V7wQ= 2023-10-17T18:47:47.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hsh/1N50tWhpGVUrFKZVJw3V7wQ='} 2023-10-17T18:47:47.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005027c2eb843e3363-00652ed6d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.911 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:47:47.912 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:47:47.912 DEBUG:boto:Method: HEAD 2023-10-17T18:47:47.912 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:47:47.913 DEBUG:boto:Data: 2023-10-17T18:47:47.913 DEBUG:boto:Headers: {} 2023-10-17T18:47:47.914 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.914 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.914 DEBUG:boto:Params: {} 2023-10-17T18:47:47.915 DEBUG:boto:Token: None 2023-10-17T18:47:47.915 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/ 2023-10-17T18:47:47.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E= 2023-10-17T18:47:47.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E=', 'Content-Length': '0'} 2023-10-17T18:47:47.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c2568504731b74a-00652ed6d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.920 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:47:47.920 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:47:47.921 DEBUG:boto:Method: PUT 2023-10-17T18:47:47.921 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:47:47.922 DEBUG:boto:Data: 2023-10-17T18:47:47.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:47:47.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.923 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.923 DEBUG:boto:Params: {} 2023-10-17T18:47:47.924 DEBUG:boto:Token: None 2023-10-17T18:47:47.924 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/c 2023-10-17T18:47:47.925 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2kD7UGpvegYUaEWysk0ybvnfYFU= 2023-10-17T18:47:47.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2kD7UGpvegYUaEWysk0ybvnfYFU='} 2023-10-17T18:47:47.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3232ff7c9bb9138-00652ed6d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.931 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:47:47.932 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:47:47.932 DEBUG:boto:Method: HEAD 2023-10-17T18:47:47.932 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:47:47.933 DEBUG:boto:Data: 2023-10-17T18:47:47.933 DEBUG:boto:Headers: {} 2023-10-17T18:47:47.933 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.934 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.934 DEBUG:boto:Params: {} 2023-10-17T18:47:47.935 DEBUG:boto:Token: None 2023-10-17T18:47:47.935 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/ 2023-10-17T18:47:47.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E= 2023-10-17T18:47:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u0WY+32egznnnjOLsLkUsdjsj9E=', 'Content-Length': '0'} 2023-10-17T18:47:47.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d2a9d959cf55cde-00652ed6d3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.939 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:47:47.939 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:47:47.940 DEBUG:boto:Method: PUT 2023-10-17T18:47:47.940 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:47:47.941 DEBUG:boto:Data: 2023-10-17T18:47:47.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:47:47.941 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:47:47.942 DEBUG:boto:Port: 8000 2023-10-17T18:47:47.943 DEBUG:boto:Params: {} 2023-10-17T18:47:47.943 DEBUG:boto:Token: None 2023-10-17T18:47:47.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:47:47 GMT /hbmuog-56/d 2023-10-17T18:47:47.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OeV5ZZxTlYgDfb0Td1DX0fHThf4= 2023-10-17T18:47:47.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:47:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OeV5ZZxTlYgDfb0Td1DX0fHThf4='} 2023-10-17T18:47:47.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000952c72edddab675a-00652ed6d3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:47:47.991 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-56 2023-10-17T18:47:47.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:47.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:47:47.993 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:47:48.051 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:47:48.051 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:47:48.087 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:48.086+0000 7fc9d62d3780 1 waiting to reach incremental sync.. 2023-10-17T18:47:53.087 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:53.087+0000 7fc9d62d3780 1 waiting to reach incremental sync.. 2023-10-17T18:47:58.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:47:58.087+0000 7fc9d62d3780 1 waiting to reach incremental sync.. 2023-10-17T18:48:03.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:03.088+0000 7fc9d62d3780 1 bucket sync caught up with source: 2023-10-17T18:48:03.090 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.33.6, , , , 00000000001.48.6, , , 00000000001.46.6, , 00000000001.32.6] 2023-10-17T18:48:03.090 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.33.6, , , , 00000000001.48.6, , , 00000000001.46.6, , 00000000001.32.6] 2023-10-17T18:48:03.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:03.088+0000 7fc9d62d3780 0 bucket checkpoint complete 2023-10-17T18:48:03.108 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-56 2023-10-17T18:48:03.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:48:03.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:48:03.109 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:48:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:48:03.173 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:48:03.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:03.212+0000 7f295f7d0780 1 waiting to reach incremental sync.. 2023-10-17T18:48:08.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:08.213+0000 7f295f7d0780 1 waiting to reach incremental sync.. 2023-10-17T18:48:13.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:13.214+0000 7f295f7d0780 1 bucket sync caught up with source: 2023-10-17T18:48:13.216 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.168.6, , , , 00000000001.188.6, , , 00000000001.187.6, , 00000000001.318.6] 2023-10-17T18:48:13.217 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.168.6, , , , 00000000001.188.6, , , 00000000001.187.6, , 00000000001.318.6] 2023-10-17T18:48:13.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:13.214+0000 7f295f7d0780 0 bucket checkpoint complete 2023-10-17T18:48:13.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-56 zones={a2, a1} 2023-10-17T18:48:13.234 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.234 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.235 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.235 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:13.235 DEBUG:boto:Data: 2023-10-17T18:48:13.236 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.237 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.237 DEBUG:boto:Params: {} 2023-10-17T18:48:13.238 DEBUG:boto:Token: None 2023-10-17T18:48:13.238 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DFgNtLsyq+zQtNyBpLmHoIkyLEE= 2023-10-17T18:48:13.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DFgNtLsyq+zQtNyBpLmHoIkyLEE=', 'Content-Length': '0'} 2023-10-17T18:48:13.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a9be0e1f5251175-00652ed6ed-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.243 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.243 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.243 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.244 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:13.244 DEBUG:boto:Data: 2023-10-17T18:48:13.245 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.245 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.246 DEBUG:boto:Params: {} 2023-10-17T18:48:13.246 DEBUG:boto:Token: None 2023-10-17T18:48:13.247 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DFgNtLsyq+zQtNyBpLmHoIkyLEE= 2023-10-17T18:48:13.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DFgNtLsyq+zQtNyBpLmHoIkyLEE=', 'Content-Length': '0'} 2023-10-17T18:48:13.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f6f496668bd09cc-00652ed6ed-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.251 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:48:13.251 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.252 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.252 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.253 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.253 DEBUG:boto:Method: GET 2023-10-17T18:48:13.253 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.254 DEBUG:boto:Data: 2023-10-17T18:48:13.254 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.255 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.255 DEBUG:boto:Params: {} 2023-10-17T18:48:13.256 DEBUG:boto:Token: None 2023-10-17T18:48:13.256 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/?versions 2023-10-17T18:48:13.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs= 2023-10-17T18:48:13.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs=', 'Content-Length': '0'} 2023-10-17T18:48:13.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000426fef9c7bf9a37b-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.261 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.262 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:48:13.262 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:48:13.263 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:48:13.263 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:48:13.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:48:13.264 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.264 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.265 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.265 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.265 DEBUG:boto:Method: GET 2023-10-17T18:48:13.266 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.266 DEBUG:boto:Data: 2023-10-17T18:48:13.267 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.268 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.268 DEBUG:boto:Params: {} 2023-10-17T18:48:13.269 DEBUG:boto:Token: None 2023-10-17T18:48:13.269 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/?versions 2023-10-17T18:48:13.270 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs= 2023-10-17T18:48:13.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs=', 'Content-Length': '0'} 2023-10-17T18:48:13.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7578d70acdb349-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.273 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.274 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:48:13.275 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:48:13.275 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:48:13.275 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:48:13.276 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.276 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.277 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.277 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.277 DEBUG:boto:Method: GET 2023-10-17T18:48:13.278 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.279 DEBUG:boto:Data: 2023-10-17T18:48:13.279 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.280 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.280 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.281 DEBUG:boto:Params: {} 2023-10-17T18:48:13.281 DEBUG:boto:Token: None 2023-10-17T18:48:13.282 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/?versions 2023-10-17T18:48:13.282 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs= 2023-10-17T18:48:13.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs=', 'Content-Length': '0'} 2023-10-17T18:48:13.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daacad1792808015-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.290 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.291 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.292 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.292 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.293 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.293 DEBUG:boto:Method: GET 2023-10-17T18:48:13.294 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:48:13.295 DEBUG:boto:Data: 2023-10-17T18:48:13.295 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.296 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.296 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.297 DEBUG:boto:Params: {} 2023-10-17T18:48:13.297 DEBUG:boto:Token: None 2023-10-17T18:48:13.298 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/?versions 2023-10-17T18:48:13.298 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs= 2023-10-17T18:48:13.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AWyD9zCkYjQHO9ckT2TMlHTWUbs=', 'Content-Length': '0'} 2023-10-17T18:48:13.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94a428e2f894601-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.303 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:48:13.304 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.304 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.305 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.305 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.306 DEBUG:boto:Method: GET 2023-10-17T18:48:13.306 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.307 DEBUG:boto:Data: 2023-10-17T18:48:13.307 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.308 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.308 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.308 DEBUG:boto:Params: {} 2023-10-17T18:48:13.309 DEBUG:boto:Token: None 2023-10-17T18:48:13.310 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.310 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY= 2023-10-17T18:48:13.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY=', 'Content-Length': '0'} 2023-10-17T18:48:13.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97286ced61a024f-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.313 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.314 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.314 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.315 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.315 DEBUG:boto:Method: GET 2023-10-17T18:48:13.316 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.316 DEBUG:boto:Data: 2023-10-17T18:48:13.317 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.317 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.318 DEBUG:boto:Params: {} 2023-10-17T18:48:13.318 DEBUG:boto:Token: None 2023-10-17T18:48:13.319 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY= 2023-10-17T18:48:13.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY=', 'Content-Length': '0'} 2023-10-17T18:48:13.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224a6309fa2f655c-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.323 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.323 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.323 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.324 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.324 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.325 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.325 DEBUG:boto:Data: 2023-10-17T18:48:13.325 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.326 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.327 DEBUG:boto:Params: {} 2023-10-17T18:48:13.327 DEBUG:boto:Token: None 2023-10-17T18:48:13.328 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v5kzDZT7TnL1j87SJAzALi1fEoY= 2023-10-17T18:48:13.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v5kzDZT7TnL1j87SJAzALi1fEoY=', 'Content-Length': '0'} 2023-10-17T18:48:13.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78dd49a22ce5439-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.331 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.332 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.332 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.332 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.333 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.333 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.334 DEBUG:boto:Data: 2023-10-17T18:48:13.334 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.335 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.336 DEBUG:boto:Params: {} 2023-10-17T18:48:13.336 DEBUG:boto:Token: None 2023-10-17T18:48:13.336 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v5kzDZT7TnL1j87SJAzALi1fEoY= 2023-10-17T18:48:13.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v5kzDZT7TnL1j87SJAzALi1fEoY=', 'Content-Length': '0'} 2023-10-17T18:48:13.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064b8d540b28f55b3-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:48:13.340 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.341 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.341 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.341 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.342 DEBUG:boto:Method: GET 2023-10-17T18:48:13.342 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.343 DEBUG:boto:Data: 2023-10-17T18:48:13.343 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.344 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.344 DEBUG:boto:Params: {} 2023-10-17T18:48:13.345 DEBUG:boto:Token: None 2023-10-17T18:48:13.345 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY= 2023-10-17T18:48:13.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY=', 'Content-Length': '0'} 2023-10-17T18:48:13.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c3ba758b01d096-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.349 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.349 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.350 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.350 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:48:13.351 DEBUG:boto:Method: GET 2023-10-17T18:48:13.351 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.352 DEBUG:boto:Data: 2023-10-17T18:48:13.352 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.353 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.353 DEBUG:boto:Params: {} 2023-10-17T18:48:13.354 DEBUG:boto:Token: None 2023-10-17T18:48:13.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:48:13.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY= 2023-10-17T18:48:13.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFps7hM60qSjCT0mIvtQhsFNhzY=', 'Content-Length': '0'} 2023-10-17T18:48:13.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a19932fa9b619edb-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.398 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.399 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.399 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.400 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.400 DEBUG:boto:Method: GET 2023-10-17T18:48:13.400 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:48:13.401 DEBUG:boto:Data: 2023-10-17T18:48:13.401 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.402 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.403 DEBUG:boto:Params: {} 2023-10-17T18:48:13.403 DEBUG:boto:Token: None 2023-10-17T18:48:13.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73286b3af8c89ab-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.408 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.409 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.409 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.410 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.410 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.411 DEBUG:boto:Method: GET 2023-10-17T18:48:13.411 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:48:13.412 DEBUG:boto:Data: 2023-10-17T18:48:13.412 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.412 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.413 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.413 DEBUG:boto:Params: {} 2023-10-17T18:48:13.414 DEBUG:boto:Token: None 2023-10-17T18:48:13.414 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12b03339916007c-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.419 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.420 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.420 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.421 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.421 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:48:13.422 DEBUG:boto:Data: 2023-10-17T18:48:13.422 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.423 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.423 DEBUG:boto:Params: {} 2023-10-17T18:48:13.424 DEBUG:boto:Token: None 2023-10-17T18:48:13.424 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a 2023-10-17T18:48:13.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3PZJB4Trnf/0i1tRIG357nAHnvc= 2023-10-17T18:48:13.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3PZJB4Trnf/0i1tRIG357nAHnvc=', 'Content-Length': '0'} 2023-10-17T18:48:13.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e6762e4b1905b1-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.428 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.428 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.429 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.429 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:48:13.429 DEBUG:boto:Data: 2023-10-17T18:48:13.430 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.431 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.431 DEBUG:boto:Params: {} 2023-10-17T18:48:13.432 DEBUG:boto:Token: None 2023-10-17T18:48:13.432 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a 2023-10-17T18:48:13.433 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3PZJB4Trnf/0i1tRIG357nAHnvc= 2023-10-17T18:48:13.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3PZJB4Trnf/0i1tRIG357nAHnvc=', 'Content-Length': '0'} 2023-10-17T18:48:13.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7bbc35931400e06-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:48:13.436 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.437 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.437 DEBUG:boto:Method: GET 2023-10-17T18:48:13.438 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:48:13.438 DEBUG:boto:Data: 2023-10-17T18:48:13.439 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.440 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.440 DEBUG:boto:Params: {} 2023-10-17T18:48:13.441 DEBUG:boto:Token: None 2023-10-17T18:48:13.441 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a 2023-10-17T18:48:13.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QEuBnSxu1WnCqMf3NeWcYrL2JLo= 2023-10-17T18:48:13.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QEuBnSxu1WnCqMf3NeWcYrL2JLo=', 'Content-Length': '0'} 2023-10-17T18:48:13.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89d3455af3b9b5b-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.490 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:48:13.491 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:48:13.491 DEBUG:boto:Method: GET 2023-10-17T18:48:13.492 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:48:13.492 DEBUG:boto:Data: 2023-10-17T18:48:13.493 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.494 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.494 DEBUG:boto:Params: {} 2023-10-17T18:48:13.495 DEBUG:boto:Token: None 2023-10-17T18:48:13.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/a 2023-10-17T18:48:13.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QEuBnSxu1WnCqMf3NeWcYrL2JLo= 2023-10-17T18:48:13.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QEuBnSxu1WnCqMf3NeWcYrL2JLo=', 'Content-Length': '0'} 2023-10-17T18:48:13.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e31e98064eaf353-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.499 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.499 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.500 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.500 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.501 DEBUG:boto:Method: GET 2023-10-17T18:48:13.501 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:48:13.502 DEBUG:boto:Data: 2023-10-17T18:48:13.502 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.503 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.503 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.503 DEBUG:boto:Params: {} 2023-10-17T18:48:13.504 DEBUG:boto:Token: None 2023-10-17T18:48:13.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ba5fca6ee4fe22-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.509 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.509 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.510 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.510 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.511 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:48:13.511 DEBUG:boto:Method: GET 2023-10-17T18:48:13.513 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:48:13.513 DEBUG:boto:Data: 2023-10-17T18:48:13.514 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.515 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.515 DEBUG:boto:Params: {} 2023-10-17T18:48:13.516 DEBUG:boto:Token: None 2023-10-17T18:48:13.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000615f772fff97afff-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.562 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:48:13.564 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.564 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.565 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.566 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.566 DEBUG:boto:Method: GET 2023-10-17T18:48:13.567 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.567 DEBUG:boto:Data: 2023-10-17T18:48:13.567 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.568 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.569 DEBUG:boto:Params: {} 2023-10-17T18:48:13.569 DEBUG:boto:Token: None 2023-10-17T18:48:13.570 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48= 2023-10-17T18:48:13.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48=', 'Content-Length': '0'} 2023-10-17T18:48:13.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed2bcc217821eb98-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.574 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.574 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.575 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.575 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.576 DEBUG:boto:Method: GET 2023-10-17T18:48:13.576 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.576 DEBUG:boto:Data: 2023-10-17T18:48:13.577 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.578 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.578 DEBUG:boto:Params: {} 2023-10-17T18:48:13.579 DEBUG:boto:Token: None 2023-10-17T18:48:13.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48= 2023-10-17T18:48:13.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48=', 'Content-Length': '0'} 2023-10-17T18:48:13.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003977233713761123-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.583 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.583 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.584 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.584 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.585 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.585 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.586 DEBUG:boto:Data: 2023-10-17T18:48:13.586 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.587 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.587 DEBUG:boto:Params: {} 2023-10-17T18:48:13.588 DEBUG:boto:Token: None 2023-10-17T18:48:13.588 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e8phyM/GS3WZ3aVwGmeQqS0+Znc= 2023-10-17T18:48:13.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e8phyM/GS3WZ3aVwGmeQqS0+Znc=', 'Content-Length': '0'} 2023-10-17T18:48:13.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a67c5f3dc63195-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.592 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.592 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.593 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.593 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.594 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.594 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.594 DEBUG:boto:Data: 2023-10-17T18:48:13.595 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.596 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.596 DEBUG:boto:Params: {} 2023-10-17T18:48:13.597 DEBUG:boto:Token: None 2023-10-17T18:48:13.597 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e8phyM/GS3WZ3aVwGmeQqS0+Znc= 2023-10-17T18:48:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e8phyM/GS3WZ3aVwGmeQqS0+Znc=', 'Content-Length': '0'} 2023-10-17T18:48:13.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb095742585d0dc3-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:48:13.601 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.601 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.602 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.602 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.603 DEBUG:boto:Method: GET 2023-10-17T18:48:13.603 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.603 DEBUG:boto:Data: 2023-10-17T18:48:13.604 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.604 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.605 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.605 DEBUG:boto:Params: {} 2023-10-17T18:48:13.606 DEBUG:boto:Token: None 2023-10-17T18:48:13.606 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.607 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48= 2023-10-17T18:48:13.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48=', 'Content-Length': '0'} 2023-10-17T18:48:13.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8dcd22baf7ac16-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.610 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.610 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.611 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.611 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:48:13.612 DEBUG:boto:Method: GET 2023-10-17T18:48:13.612 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.612 DEBUG:boto:Data: 2023-10-17T18:48:13.613 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.613 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.614 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.614 DEBUG:boto:Params: {} 2023-10-17T18:48:13.615 DEBUG:boto:Token: None 2023-10-17T18:48:13.615 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:48:13.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48= 2023-10-17T18:48:13.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mEzkfS9+cb1drR/HU8dxPzFET48=', 'Content-Length': '0'} 2023-10-17T18:48:13.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4426a9e97c4b367-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.619 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.619 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.620 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.620 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.621 DEBUG:boto:Method: GET 2023-10-17T18:48:13.621 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:48:13.622 DEBUG:boto:Data: 2023-10-17T18:48:13.622 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.623 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.623 DEBUG:boto:Params: {} 2023-10-17T18:48:13.624 DEBUG:boto:Token: None 2023-10-17T18:48:13.624 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b085ec420e9ed355-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.629 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.629 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.630 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.630 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.631 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.631 DEBUG:boto:Method: GET 2023-10-17T18:48:13.632 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:48:13.632 DEBUG:boto:Data: 2023-10-17T18:48:13.633 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.633 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.634 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.634 DEBUG:boto:Params: {} 2023-10-17T18:48:13.634 DEBUG:boto:Token: None 2023-10-17T18:48:13.635 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.635 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac185a6435532b9-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.686 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.687 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.688 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.688 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.688 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:48:13.689 DEBUG:boto:Data: 2023-10-17T18:48:13.689 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.690 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.690 DEBUG:boto:Params: {} 2023-10-17T18:48:13.691 DEBUG:boto:Token: None 2023-10-17T18:48:13.691 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b 2023-10-17T18:48:13.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vqsvONFAZE/NAu6GifivyhkMzaE= 2023-10-17T18:48:13.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vqsvONFAZE/NAu6GifivyhkMzaE=', 'Content-Length': '0'} 2023-10-17T18:48:13.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07202b158585dfc-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.695 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.695 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.696 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.696 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:48:13.697 DEBUG:boto:Data: 2023-10-17T18:48:13.697 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.698 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.698 DEBUG:boto:Params: {} 2023-10-17T18:48:13.698 DEBUG:boto:Token: None 2023-10-17T18:48:13.699 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b 2023-10-17T18:48:13.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vqsvONFAZE/NAu6GifivyhkMzaE= 2023-10-17T18:48:13.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vqsvONFAZE/NAu6GifivyhkMzaE=', 'Content-Length': '0'} 2023-10-17T18:48:13.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d0374148788b75-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:48:13.703 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.703 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.704 DEBUG:boto:Method: GET 2023-10-17T18:48:13.704 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:48:13.705 DEBUG:boto:Data: 2023-10-17T18:48:13.705 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.706 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.707 DEBUG:boto:Params: {} 2023-10-17T18:48:13.707 DEBUG:boto:Token: None 2023-10-17T18:48:13.707 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b 2023-10-17T18:48:13.708 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oUPcfiKchsLnEMg+LUQ8znkda18= 2023-10-17T18:48:13.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oUPcfiKchsLnEMg+LUQ8znkda18=', 'Content-Length': '0'} 2023-10-17T18:48:13.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363e9c14b7f89802-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.758 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:48:13.759 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:48:13.759 DEBUG:boto:Method: GET 2023-10-17T18:48:13.760 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:48:13.760 DEBUG:boto:Data: 2023-10-17T18:48:13.761 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.761 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.762 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.762 DEBUG:boto:Params: {} 2023-10-17T18:48:13.763 DEBUG:boto:Token: None 2023-10-17T18:48:13.763 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/b 2023-10-17T18:48:13.764 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oUPcfiKchsLnEMg+LUQ8znkda18= 2023-10-17T18:48:13.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oUPcfiKchsLnEMg+LUQ8znkda18=', 'Content-Length': '0'} 2023-10-17T18:48:13.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df524dcf2b9f220a-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.767 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.767 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.768 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.768 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.769 DEBUG:boto:Method: GET 2023-10-17T18:48:13.769 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:48:13.770 DEBUG:boto:Data: 2023-10-17T18:48:13.770 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.771 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.771 DEBUG:boto:Params: {} 2023-10-17T18:48:13.772 DEBUG:boto:Token: None 2023-10-17T18:48:13.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd927c61421de43-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.777 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.778 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.778 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.779 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.779 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:48:13.780 DEBUG:boto:Method: GET 2023-10-17T18:48:13.780 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:48:13.781 DEBUG:boto:Data: 2023-10-17T18:48:13.781 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.782 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.782 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.783 DEBUG:boto:Params: {} 2023-10-17T18:48:13.783 DEBUG:boto:Token: None 2023-10-17T18:48:13.784 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d939ba6955174e4d-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.788 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:48:13.789 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.790 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.790 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.791 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.791 DEBUG:boto:Method: GET 2023-10-17T18:48:13.792 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.792 DEBUG:boto:Data: 2023-10-17T18:48:13.793 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.793 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.794 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.794 DEBUG:boto:Params: {} 2023-10-17T18:48:13.795 DEBUG:boto:Token: None 2023-10-17T18:48:13.795 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA= 2023-10-17T18:48:13.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA=', 'Content-Length': '0'} 2023-10-17T18:48:13.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2699ad09cc33c8e-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.799 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.799 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.800 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.800 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.801 DEBUG:boto:Method: GET 2023-10-17T18:48:13.801 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.802 DEBUG:boto:Data: 2023-10-17T18:48:13.802 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.803 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.803 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.803 DEBUG:boto:Params: {} 2023-10-17T18:48:13.804 DEBUG:boto:Token: None 2023-10-17T18:48:13.804 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA= 2023-10-17T18:48:13.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA=', 'Content-Length': '0'} 2023-10-17T18:48:13.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f763691961aeb50-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.854 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.855 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.855 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.856 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.856 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.857 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.857 DEBUG:boto:Data: 2023-10-17T18:48:13.857 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.858 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.859 DEBUG:boto:Params: {} 2023-10-17T18:48:13.859 DEBUG:boto:Token: None 2023-10-17T18:48:13.860 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMfx0ITwKnwja88B8C1uJX7VgwM= 2023-10-17T18:48:13.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMfx0ITwKnwja88B8C1uJX7VgwM=', 'Content-Length': '0'} 2023-10-17T18:48:13.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67c4e82dc1f89d5-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.863 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.864 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.864 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.864 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.865 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.865 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.866 DEBUG:boto:Data: 2023-10-17T18:48:13.866 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.867 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.867 DEBUG:boto:Params: {} 2023-10-17T18:48:13.868 DEBUG:boto:Token: None 2023-10-17T18:48:13.868 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMfx0ITwKnwja88B8C1uJX7VgwM= 2023-10-17T18:48:13.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMfx0ITwKnwja88B8C1uJX7VgwM=', 'Content-Length': '0'} 2023-10-17T18:48:13.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1eea555cdc3bfb-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:48:13.872 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.873 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.873 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.874 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.874 DEBUG:boto:Method: GET 2023-10-17T18:48:13.874 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.875 DEBUG:boto:Data: 2023-10-17T18:48:13.875 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.876 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.876 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.876 DEBUG:boto:Params: {} 2023-10-17T18:48:13.877 DEBUG:boto:Token: None 2023-10-17T18:48:13.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA= 2023-10-17T18:48:13.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA=', 'Content-Length': '0'} 2023-10-17T18:48:13.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6bbf417a079213-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.881 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.882 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.882 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.883 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:48:13.883 DEBUG:boto:Method: GET 2023-10-17T18:48:13.883 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.884 DEBUG:boto:Data: 2023-10-17T18:48:13.884 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.885 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.886 DEBUG:boto:Params: {} 2023-10-17T18:48:13.886 DEBUG:boto:Token: None 2023-10-17T18:48:13.887 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:48:13.887 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA= 2023-10-17T18:48:13.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZVEyaX+LcwfYzHepPrs4ehvxuA=', 'Content-Length': '0'} 2023-10-17T18:48:13.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a42bff82787a58cf-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.890 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.891 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.891 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.892 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.892 DEBUG:boto:Method: GET 2023-10-17T18:48:13.892 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:48:13.893 DEBUG:boto:Data: 2023-10-17T18:48:13.893 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.894 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.895 DEBUG:boto:Params: {} 2023-10-17T18:48:13.895 DEBUG:boto:Token: None 2023-10-17T18:48:13.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f01a1e80c52c9cb5-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.900 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.900 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.901 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.901 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.902 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.902 DEBUG:boto:Method: GET 2023-10-17T18:48:13.903 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:48:13.903 DEBUG:boto:Data: 2023-10-17T18:48:13.904 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.904 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.904 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.905 DEBUG:boto:Params: {} 2023-10-17T18:48:13.905 DEBUG:boto:Token: None 2023-10-17T18:48:13.906 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.906 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002891a65fc23da303-00652ed6ed-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.910 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.911 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.911 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.912 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.912 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:48:13.912 DEBUG:boto:Data: 2023-10-17T18:48:13.913 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.914 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.914 DEBUG:boto:Params: {} 2023-10-17T18:48:13.915 DEBUG:boto:Token: None 2023-10-17T18:48:13.915 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c 2023-10-17T18:48:13.916 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tc8EfFO9aa+qFSpaHWN5aIt/3w4= 2023-10-17T18:48:13.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tc8EfFO9aa+qFSpaHWN5aIt/3w4=', 'Content-Length': '0'} 2023-10-17T18:48:13.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006408ab705a3d8d39-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.919 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.919 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.921 DEBUG:boto:Method: HEAD 2023-10-17T18:48:13.922 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:48:13.922 DEBUG:boto:Data: 2023-10-17T18:48:13.922 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.923 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.924 DEBUG:boto:Params: {} 2023-10-17T18:48:13.924 DEBUG:boto:Token: None 2023-10-17T18:48:13.925 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c 2023-10-17T18:48:13.925 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tc8EfFO9aa+qFSpaHWN5aIt/3w4= 2023-10-17T18:48:13.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tc8EfFO9aa+qFSpaHWN5aIt/3w4=', 'Content-Length': '0'} 2023-10-17T18:48:13.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2ea387d72ba6fc-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:48:13.929 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.929 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.930 DEBUG:boto:Method: GET 2023-10-17T18:48:13.930 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:48:13.930 DEBUG:boto:Data: 2023-10-17T18:48:13.931 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.931 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.932 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.932 DEBUG:boto:Params: {} 2023-10-17T18:48:13.933 DEBUG:boto:Token: None 2023-10-17T18:48:13.933 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c 2023-10-17T18:48:13.934 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VAqc/9PTdWUwIPgDDYO1zw6+bDY= 2023-10-17T18:48:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VAqc/9PTdWUwIPgDDYO1zw6+bDY=', 'Content-Length': '0'} 2023-10-17T18:48:13.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca0b3478233d55e-00652ed6ed-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.978 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:48:13.979 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:48:13.979 DEBUG:boto:Method: GET 2023-10-17T18:48:13.980 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:48:13.980 DEBUG:boto:Data: 2023-10-17T18:48:13.981 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.981 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:13.981 DEBUG:boto:Port: 8000 2023-10-17T18:48:13.982 DEBUG:boto:Params: {} 2023-10-17T18:48:13.982 DEBUG:boto:Token: None 2023-10-17T18:48:13.983 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/c 2023-10-17T18:48:13.983 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VAqc/9PTdWUwIPgDDYO1zw6+bDY= 2023-10-17T18:48:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VAqc/9PTdWUwIPgDDYO1zw6+bDY=', 'Content-Length': '0'} 2023-10-17T18:48:13.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b18f6ee9b940b88-00652ed6ed-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.987 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.987 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.988 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.988 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.989 DEBUG:boto:Method: GET 2023-10-17T18:48:13.989 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:48:13.990 DEBUG:boto:Data: 2023-10-17T18:48:13.990 DEBUG:boto:Headers: {} 2023-10-17T18:48:13.990 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:13.991 DEBUG:boto:Port: 8001 2023-10-17T18:48:13.991 DEBUG:boto:Params: {} 2023-10-17T18:48:13.992 DEBUG:boto:Token: None 2023-10-17T18:48:13.992 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:13 GMT /hbmuog-56/ 2023-10-17T18:48:13.993 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA= 2023-10-17T18:48:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6woepJhWqEegC0habgo5+aln8GA=', 'Content-Length': '0'} 2023-10-17T18:48:13.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d5a71182409aae-00652ed6ed-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:13.997 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:13.998 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:13.998 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:13.998 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.999 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:48:13.999 DEBUG:boto:Method: GET 2023-10-17T18:48:14.000 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:48:14.000 DEBUG:boto:Data: 2023-10-17T18:48:14.000 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.001 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.001 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.002 DEBUG:boto:Params: {} 2023-10-17T18:48:14.002 DEBUG:boto:Token: None 2023-10-17T18:48:14.003 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.003 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg= 2023-10-17T18:48:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg=', 'Content-Length': '0'} 2023-10-17T18:48:14.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca425e866daa663-00652ed6ee-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.050 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:14.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:48:14.052 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.052 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.053 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.053 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.054 DEBUG:boto:Method: GET 2023-10-17T18:48:14.054 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.055 DEBUG:boto:Data: 2023-10-17T18:48:14.055 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.056 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.056 DEBUG:boto:Params: {} 2023-10-17T18:48:14.057 DEBUG:boto:Token: None 2023-10-17T18:48:14.057 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA= 2023-10-17T18:48:14.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA=', 'Content-Length': '0'} 2023-10-17T18:48:14.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d292c828fade3f55-00652ed6ee-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.061 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.062 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.062 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.063 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.063 DEBUG:boto:Method: GET 2023-10-17T18:48:14.064 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.064 DEBUG:boto:Data: 2023-10-17T18:48:14.064 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.065 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.066 DEBUG:boto:Params: {} 2023-10-17T18:48:14.066 DEBUG:boto:Token: None 2023-10-17T18:48:14.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA= 2023-10-17T18:48:14.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA=', 'Content-Length': '0'} 2023-10-17T18:48:14.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb5d424d41e654d0-00652ed6ee-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.070 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.071 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.071 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.072 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.072 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.072 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.073 DEBUG:boto:Data: 2023-10-17T18:48:14.073 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.074 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.074 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.075 DEBUG:boto:Params: {} 2023-10-17T18:48:14.075 DEBUG:boto:Token: None 2023-10-17T18:48:14.076 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WmdQrS1xUoUL5wb+bulGX5igNqc= 2023-10-17T18:48:14.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WmdQrS1xUoUL5wb+bulGX5igNqc=', 'Content-Length': '0'} 2023-10-17T18:48:14.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7aa77eb32ad909-00652ed6ee-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.079 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.080 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.080 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.081 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.081 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.081 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.082 DEBUG:boto:Data: 2023-10-17T18:48:14.082 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.083 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.083 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.083 DEBUG:boto:Params: {} 2023-10-17T18:48:14.084 DEBUG:boto:Token: None 2023-10-17T18:48:14.084 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WmdQrS1xUoUL5wb+bulGX5igNqc= 2023-10-17T18:48:14.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WmdQrS1xUoUL5wb+bulGX5igNqc=', 'Content-Length': '0'} 2023-10-17T18:48:14.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012597a9e979ce1c9-00652ed6ee-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:48:14.088 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.089 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.089 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.090 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.090 DEBUG:boto:Method: GET 2023-10-17T18:48:14.091 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.091 DEBUG:boto:Data: 2023-10-17T18:48:14.092 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.092 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.092 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.093 DEBUG:boto:Params: {} 2023-10-17T18:48:14.093 DEBUG:boto:Token: None 2023-10-17T18:48:14.094 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.095 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA= 2023-10-17T18:48:14.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA=', 'Content-Length': '0'} 2023-10-17T18:48:14.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a4cf755fcfd3aa-00652ed6ee-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.098 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.098 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.099 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.099 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:48:14.100 DEBUG:boto:Method: GET 2023-10-17T18:48:14.100 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.101 DEBUG:boto:Data: 2023-10-17T18:48:14.101 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.102 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.102 DEBUG:boto:Params: {} 2023-10-17T18:48:14.103 DEBUG:boto:Token: None 2023-10-17T18:48:14.103 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:48:14.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA= 2023-10-17T18:48:14.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cP9SEO8I1/yma8DuuhFVFsuZfYA=', 'Content-Length': '0'} 2023-10-17T18:48:14.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e40febea125d14e-00652ed6ee-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.107 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.108 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.108 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.109 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.109 DEBUG:boto:Method: GET 2023-10-17T18:48:14.110 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:48:14.110 DEBUG:boto:Data: 2023-10-17T18:48:14.111 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.112 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.112 DEBUG:boto:Params: {} 2023-10-17T18:48:14.113 DEBUG:boto:Token: None 2023-10-17T18:48:14.113 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.114 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg= 2023-10-17T18:48:14.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg=', 'Content-Length': '0'} 2023-10-17T18:48:14.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000711f8223d2b5c6af-00652ed6ee-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.118 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:14.119 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.119 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.120 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.120 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.121 DEBUG:boto:Method: GET 2023-10-17T18:48:14.121 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:48:14.122 DEBUG:boto:Data: 2023-10-17T18:48:14.122 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.122 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.123 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.123 DEBUG:boto:Params: {} 2023-10-17T18:48:14.124 DEBUG:boto:Token: None 2023-10-17T18:48:14.124 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg= 2023-10-17T18:48:14.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg=', 'Content-Length': '0'} 2023-10-17T18:48:14.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb219b0a4a264d1-00652ed6ee-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.130 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:14.130 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.131 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.131 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.132 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:48:14.132 DEBUG:boto:Data: 2023-10-17T18:48:14.133 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.133 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.133 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.134 DEBUG:boto:Params: {} 2023-10-17T18:48:14.134 DEBUG:boto:Token: None 2023-10-17T18:48:14.135 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d 2023-10-17T18:48:14.135 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/F8aR0ASJCS+JaJJOXDv1u3eZOc= 2023-10-17T18:48:14.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/F8aR0ASJCS+JaJJOXDv1u3eZOc=', 'Content-Length': '0'} 2023-10-17T18:48:14.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f1d7087af23982-00652ed6ee-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.139 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.139 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.139 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.140 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:48:14.140 DEBUG:boto:Data: 2023-10-17T18:48:14.141 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.141 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.142 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.142 DEBUG:boto:Params: {} 2023-10-17T18:48:14.143 DEBUG:boto:Token: None 2023-10-17T18:48:14.143 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d 2023-10-17T18:48:14.144 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/F8aR0ASJCS+JaJJOXDv1u3eZOc= 2023-10-17T18:48:14.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/F8aR0ASJCS+JaJJOXDv1u3eZOc=', 'Content-Length': '0'} 2023-10-17T18:48:14.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63c9ec472c7de8a-00652ed6ee-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:48:14.147 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.147 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.148 DEBUG:boto:Method: GET 2023-10-17T18:48:14.148 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:48:14.149 DEBUG:boto:Data: 2023-10-17T18:48:14.149 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.150 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.150 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.151 DEBUG:boto:Params: {} 2023-10-17T18:48:14.151 DEBUG:boto:Token: None 2023-10-17T18:48:14.152 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d 2023-10-17T18:48:14.152 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CC3oysxnRh9M/xwt9xZ4ympUxb4= 2023-10-17T18:48:14.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CC3oysxnRh9M/xwt9xZ4ympUxb4=', 'Content-Length': '0'} 2023-10-17T18:48:14.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a22b3920fed63b9-00652ed6ee-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.156 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:48:14.156 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:48:14.157 DEBUG:boto:Method: GET 2023-10-17T18:48:14.157 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:48:14.158 DEBUG:boto:Data: 2023-10-17T18:48:14.158 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.159 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.159 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.160 DEBUG:boto:Params: {} 2023-10-17T18:48:14.160 DEBUG:boto:Token: None 2023-10-17T18:48:14.161 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/d 2023-10-17T18:48:14.161 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CC3oysxnRh9M/xwt9xZ4ympUxb4= 2023-10-17T18:48:14.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CC3oysxnRh9M/xwt9xZ4ympUxb4=', 'Content-Length': '0'} 2023-10-17T18:48:14.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e512537e18095986-00652ed6ee-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.165 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.165 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.166 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.166 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.167 DEBUG:boto:Method: GET 2023-10-17T18:48:14.167 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:48:14.168 DEBUG:boto:Data: 2023-10-17T18:48:14.168 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.168 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:48:14.169 DEBUG:boto:Port: 8001 2023-10-17T18:48:14.169 DEBUG:boto:Params: {} 2023-10-17T18:48:14.170 DEBUG:boto:Token: None 2023-10-17T18:48:14.170 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg= 2023-10-17T18:48:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg=', 'Content-Length': '0'} 2023-10-17T18:48:14.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a9f416cf0846fef-00652ed6ee-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.218 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:14.219 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.220 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.220 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.221 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:48:14.221 DEBUG:boto:Method: GET 2023-10-17T18:48:14.221 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:48:14.222 DEBUG:boto:Data: 2023-10-17T18:48:14.222 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.223 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.223 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.224 DEBUG:boto:Params: {} 2023-10-17T18:48:14.224 DEBUG:boto:Token: None 2023-10-17T18:48:14.225 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.225 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg= 2023-10-17T18:48:14.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0mvDoUJsEiSyCWTvuFAWBT6brfg=', 'Content-Length': '0'} 2023-10-17T18:48:14.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a84784673fd28b-00652ed6ee-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.230 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:48:14.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-56 zones={a2, a1} 2023-10-17T18:48:14.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:48:14.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:48:14.232 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:48:14.291 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:48:14.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:48:14.339 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:48:14.339 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-56 2023-10-17T18:48:14.342 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 4 2023-10-17T18:48:14.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:48:14.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:48:14.380 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:48:14.431 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:48:14.431 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:48:14.470 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:48:14.470 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-56 2023-10-17T18:48:14.470 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 4 2023-10-17T18:48:14.496 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.497 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.497 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.498 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:14.498 DEBUG:boto:Data: 2023-10-17T18:48:14.499 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.499 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.500 DEBUG:boto:Params: {} 2023-10-17T18:48:14.500 DEBUG:boto:Token: None 2023-10-17T18:48:14.501 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64= 2023-10-17T18:48:14.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64=', 'Content-Length': '0'} 2023-10-17T18:48:14.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9d1b187a479072b-00652ed6ee-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.505 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:48:14.505 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:48:14.505 DEBUG:boto:Method: PUT 2023-10-17T18:48:14.506 DEBUG:boto:Path: /hbmuog-56/e 2023-10-17T18:48:14.506 DEBUG:boto:Data: 2023-10-17T18:48:14.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:48:14.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.508 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.508 DEBUG:boto:Params: {} 2023-10-17T18:48:14.508 DEBUG:boto:Token: None 2023-10-17T18:48:14.509 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/e 2023-10-17T18:48:14.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VmrvTVgDVPQ7/WkUaeGZFQ5bpFU= 2023-10-17T18:48:14.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VmrvTVgDVPQ7/WkUaeGZFQ5bpFU='} 2023-10-17T18:48:14.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c8748f0996ed7fd-00652ed6ee-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.516 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.516 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.517 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.517 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:14.518 DEBUG:boto:Data: 2023-10-17T18:48:14.518 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.519 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.519 DEBUG:boto:Params: {} 2023-10-17T18:48:14.520 DEBUG:boto:Token: None 2023-10-17T18:48:14.520 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.521 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64= 2023-10-17T18:48:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64=', 'Content-Length': '0'} 2023-10-17T18:48:14.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004048901c36b944a0-00652ed6ee-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.523 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:48:14.524 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:48:14.525 DEBUG:boto:Method: PUT 2023-10-17T18:48:14.525 DEBUG:boto:Path: /hbmuog-56/f 2023-10-17T18:48:14.525 DEBUG:boto:Data: 2023-10-17T18:48:14.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:48:14.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.527 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.527 DEBUG:boto:Params: {} 2023-10-17T18:48:14.528 DEBUG:boto:Token: None 2023-10-17T18:48:14.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/f 2023-10-17T18:48:14.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:408Q3SEPF1RRYgzBhoWX7BW4HSk= 2023-10-17T18:48:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:408Q3SEPF1RRYgzBhoWX7BW4HSk='} 2023-10-17T18:48:14.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006419cb8ea2a0719c-00652ed6ee-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.575 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.575 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.576 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.576 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:14.577 DEBUG:boto:Data: 2023-10-17T18:48:14.577 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.578 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.578 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.578 DEBUG:boto:Params: {} 2023-10-17T18:48:14.579 DEBUG:boto:Token: None 2023-10-17T18:48:14.579 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64= 2023-10-17T18:48:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64=', 'Content-Length': '0'} 2023-10-17T18:48:14.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a02290c87a16e341-00652ed6ee-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.583 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:48:14.583 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:48:14.583 DEBUG:boto:Method: PUT 2023-10-17T18:48:14.584 DEBUG:boto:Path: /hbmuog-56/g 2023-10-17T18:48:14.584 DEBUG:boto:Data: 2023-10-17T18:48:14.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:48:14.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.585 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.586 DEBUG:boto:Params: {} 2023-10-17T18:48:14.586 DEBUG:boto:Token: None 2023-10-17T18:48:14.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/g 2023-10-17T18:48:14.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SoBuu5VbBMNKhzqgS+wwwTCa5J8= 2023-10-17T18:48:14.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SoBuu5VbBMNKhzqgS+wwwTCa5J8='} 2023-10-17T18:48:14.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b40892106d3b7020-00652ed6ee-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.596 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:48:14.596 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:48:14.597 DEBUG:boto:Method: HEAD 2023-10-17T18:48:14.597 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:48:14.597 DEBUG:boto:Data: 2023-10-17T18:48:14.598 DEBUG:boto:Headers: {} 2023-10-17T18:48:14.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.599 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.599 DEBUG:boto:Params: {} 2023-10-17T18:48:14.600 DEBUG:boto:Token: None 2023-10-17T18:48:14.600 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/ 2023-10-17T18:48:14.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64= 2023-10-17T18:48:14.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YivANBW5og7dGFvSWhDwd3YFf64=', 'Content-Length': '0'} 2023-10-17T18:48:14.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def356cc77c531ca-00652ed6ee-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.603 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:48:14.604 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:48:14.604 DEBUG:boto:Method: PUT 2023-10-17T18:48:14.604 DEBUG:boto:Path: /hbmuog-56/h 2023-10-17T18:48:14.605 DEBUG:boto:Data: 2023-10-17T18:48:14.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:48:14.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:48:14.606 DEBUG:boto:Port: 8000 2023-10-17T18:48:14.607 DEBUG:boto:Params: {} 2023-10-17T18:48:14.607 DEBUG:boto:Token: None 2023-10-17T18:48:14.607 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:48:14 GMT /hbmuog-56/h 2023-10-17T18:48:14.608 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9UZHAfd3OwyWQ09KlEeiGCOYiCk= 2023-10-17T18:48:14.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:48:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9UZHAfd3OwyWQ09KlEeiGCOYiCk='} 2023-10-17T18:48:14.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e4f8c13f7af27b5-00652ed6ee-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:48:14.651 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-56 2023-10-17T18:48:14.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:48:14.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:48:14.653 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:48:14.710 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:48:14.711 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:48:14.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:14.743+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 0.. 2023-10-17T18:48:19.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:19.744+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 0.. 2023-10-17T18:48:24.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:24.746+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:29.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:29.746+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:34.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:34.747+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:39.746+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:44.747+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:49.748+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:54.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:54.749+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:48:59.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:48:59.750+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:49:04.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:04.751+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:49:09.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:09.751+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:49:14.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:14.751+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:49:19.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:19.751+0000 7f7422d5e780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:49:24.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:24.753+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:24.756 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, , ] 2023-10-17T18:49:24.756 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:29.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:29.753+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:29.755 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, , ] 2023-10-17T18:49:29.756 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:34.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:34.754+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:34.756 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, , ] 2023-10-17T18:49:34.757 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:39.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:39.756+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:39.757 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, , ] 2023-10-17T18:49:39.757 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:44.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:44.757+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:44.758 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:49:44.758 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:49.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:49.757+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:49.759 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:49:49.759 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:54.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:54.758+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:54.760 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:49:54.760 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:49:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:49:59.759+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:49:59.761 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:49:59.761 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:04.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:04.760+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:04.761 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:50:04.761 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:09.760+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:09.762 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:50:09.762 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:14.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:14.762+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:14.763 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:50:14.763 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:19.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:19.763+0000 7f7422d5e780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:19.764 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, ] 2023-10-17T18:50:19.764 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:24.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:24.763+0000 7f7422d5e780 1 bucket sync caught up with source: 2023-10-17T18:50:24.765 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:24.765 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.44.6, 00000000002.22.6, 00000000002.38.6] 2023-10-17T18:50:24.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:24.763+0000 7f7422d5e780 0 bucket checkpoint complete 2023-10-17T18:50:24.783 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-56 2023-10-17T18:50:24.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:50:24.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:50:24.785 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:50:24.850 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:50:24.850 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:50:24.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:24.888+0000 7fad76940780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:24.890 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000001.187.6, , 00000000002.338.6] 2023-10-17T18:50:24.890 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000002.228.6, , 00000000002.338.6] 2023-10-17T18:50:29.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:29.890+0000 7fad76940780 1 waiting for incremental sync to catch up: 2023-10-17T18:50:29.892 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000001.187.6, , 00000000002.338.6] 2023-10-17T18:50:29.892 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000002.228.6, , 00000000002.338.6] 2023-10-17T18:50:34.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:34.891+0000 7fad76940780 1 bucket sync caught up with source: 2023-10-17T18:50:34.893 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000002.228.6, , 00000000002.338.6] 2023-10-17T18:50:34.894 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.179.6, , , 00000000001.226.6, 00000000001.188.6, , , 00000000002.228.6, , 00000000002.338.6] 2023-10-17T18:50:34.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:34.891+0000 7fad76940780 0 bucket checkpoint complete 2023-10-17T18:50:34.910 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-56 zones={a2, a1} 2023-10-17T18:50:34.910 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.911 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.911 DEBUG:boto:Method: HEAD 2023-10-17T18:50:34.912 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:50:34.912 DEBUG:boto:Data: 2023-10-17T18:50:34.913 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.913 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:34.914 DEBUG:boto:Port: 8001 2023-10-17T18:50:34.914 DEBUG:boto:Params: {} 2023-10-17T18:50:34.918 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:50:34.919 DEBUG:boto:Token: None 2023-10-17T18:50:34.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/ 2023-10-17T18:50:34.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PeHzV/zT5M9Z74RVwqZu+Dfhj6s= 2023-10-17T18:50:34.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PeHzV/zT5M9Z74RVwqZu+Dfhj6s=', 'Content-Length': '0'} 2023-10-17T18:50:34.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1f9c1027ff97a17-00652ed77a-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.926 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.926 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.926 DEBUG:boto:Method: HEAD 2023-10-17T18:50:34.927 DEBUG:boto:Path: /hbmuog-56/ 2023-10-17T18:50:34.927 DEBUG:boto:Data: 2023-10-17T18:50:34.928 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.928 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:34.929 DEBUG:boto:Port: 8000 2023-10-17T18:50:34.929 DEBUG:boto:Params: {} 2023-10-17T18:50:34.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:50:34.930 DEBUG:boto:Token: None 2023-10-17T18:50:34.931 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/ 2023-10-17T18:50:34.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PeHzV/zT5M9Z74RVwqZu+Dfhj6s= 2023-10-17T18:50:34.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PeHzV/zT5M9Z74RVwqZu+Dfhj6s=', 'Content-Length': '0'} 2023-10-17T18:50:34.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7ff536f17835921-00652ed77a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.936 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:50:34.936 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.937 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.937 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.937 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.938 DEBUG:boto:Method: GET 2023-10-17T18:50:34.938 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.939 DEBUG:boto:Data: 2023-10-17T18:50:34.939 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.940 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:34.940 DEBUG:boto:Port: 8001 2023-10-17T18:50:34.940 DEBUG:boto:Params: {} 2023-10-17T18:50:34.941 DEBUG:boto:Token: None 2023-10-17T18:50:34.941 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/?versions 2023-10-17T18:50:34.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A= 2023-10-17T18:50:34.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A=', 'Content-Length': '0'} 2023-10-17T18:50:34.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000768ed992dea2df58-00652ed77a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.946 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:34.947 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:50:34.948 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:50:34.948 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:50:34.948 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:50:34.949 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:50:34.949 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:50:34.950 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:50:34.950 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:50:34.950 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:50:34.951 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.951 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.952 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.952 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.953 DEBUG:boto:Method: GET 2023-10-17T18:50:34.953 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.953 DEBUG:boto:Data: 2023-10-17T18:50:34.954 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:34.955 DEBUG:boto:Port: 8000 2023-10-17T18:50:34.955 DEBUG:boto:Params: {} 2023-10-17T18:50:34.956 DEBUG:boto:Token: None 2023-10-17T18:50:34.956 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/?versions 2023-10-17T18:50:34.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A= 2023-10-17T18:50:34.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A=', 'Content-Length': '0'} 2023-10-17T18:50:34.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803ca68f9ac5e30f-00652ed77a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.960 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:34.960 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:50:34.961 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:50:34.961 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:50:34.962 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:50:34.962 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:50:34.962 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:50:34.963 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:50:34.963 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:50:34.963 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.964 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.964 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.965 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.965 DEBUG:boto:Method: GET 2023-10-17T18:50:34.965 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.966 DEBUG:boto:Data: 2023-10-17T18:50:34.966 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.967 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:34.967 DEBUG:boto:Port: 8001 2023-10-17T18:50:34.968 DEBUG:boto:Params: {} 2023-10-17T18:50:34.968 DEBUG:boto:Token: None 2023-10-17T18:50:34.969 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/?versions 2023-10-17T18:50:34.969 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A= 2023-10-17T18:50:34.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A=', 'Content-Length': '0'} 2023-10-17T18:50:34.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa1867e5a83f908-00652ed77a-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.973 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:34.974 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:34.974 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:34.975 DEBUG:boto:path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.975 DEBUG:boto:auth_path=/hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.976 DEBUG:boto:Method: GET 2023-10-17T18:50:34.976 DEBUG:boto:Path: /hbmuog-56/?versions&max-keys=999 2023-10-17T18:50:34.977 DEBUG:boto:Data: 2023-10-17T18:50:34.977 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:34.978 DEBUG:boto:Port: 8000 2023-10-17T18:50:34.978 DEBUG:boto:Params: {} 2023-10-17T18:50:34.979 DEBUG:boto:Token: None 2023-10-17T18:50:34.979 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/?versions 2023-10-17T18:50:34.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A= 2023-10-17T18:50:34.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3dFHVHjBnF6mRvnJfT1E/NoLD/A=', 'Content-Length': '0'} 2023-10-17T18:50:34.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a6c103acf8dbee-00652ed77a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.983 DEBUG:boto:b'hbmuog-56999falseanulltrue2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:34.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:50:34.984 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:34.985 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:34.985 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:34.986 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:34.986 DEBUG:boto:Method: GET 2023-10-17T18:50:34.987 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:34.987 DEBUG:boto:Data: 2023-10-17T18:50:34.987 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.988 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:34.988 DEBUG:boto:Port: 8001 2023-10-17T18:50:34.989 DEBUG:boto:Params: {} 2023-10-17T18:50:34.989 DEBUG:boto:Token: None 2023-10-17T18:50:34.990 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:34.990 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s/rOFa/+XB8LdDbyabBgyg8ideQ= 2023-10-17T18:50:34.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s/rOFa/+XB8LdDbyabBgyg8ideQ=', 'Content-Length': '0'} 2023-10-17T18:50:34.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefdc42a70c03b10-00652ed77a-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:34.994 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:34.994 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:34.995 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:34.995 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:34.996 DEBUG:boto:Method: GET 2023-10-17T18:50:34.996 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:34.996 DEBUG:boto:Data: 2023-10-17T18:50:34.997 DEBUG:boto:Headers: {} 2023-10-17T18:50:34.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:34.998 DEBUG:boto:Port: 8000 2023-10-17T18:50:34.998 DEBUG:boto:Params: {} 2023-10-17T18:50:34.999 DEBUG:boto:Token: None 2023-10-17T18:50:34.999 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:34 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.000 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s/rOFa/+XB8LdDbyabBgyg8ideQ= 2023-10-17T18:50:35.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s/rOFa/+XB8LdDbyabBgyg8ideQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069e3859c092527f3-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.050 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.051 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.051 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.052 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.052 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.053 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.053 DEBUG:boto:Data: 2023-10-17T18:50:35.054 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.055 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.055 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.055 DEBUG:boto:Params: {} 2023-10-17T18:50:35.056 DEBUG:boto:Token: None 2023-10-17T18:50:35.056 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.057 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:exL3KEV7E6T8r2RuSpKWamPfT3I= 2023-10-17T18:50:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:exL3KEV7E6T8r2RuSpKWamPfT3I=', 'Content-Length': '0'} 2023-10-17T18:50:35.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221d8c927b7a4023-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.060 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.060 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.061 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.061 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.062 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.062 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.062 DEBUG:boto:Data: 2023-10-17T18:50:35.063 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.063 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.064 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.064 DEBUG:boto:Params: {} 2023-10-17T18:50:35.065 DEBUG:boto:Token: None 2023-10-17T18:50:35.065 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:exL3KEV7E6T8r2RuSpKWamPfT3I= 2023-10-17T18:50:35.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:exL3KEV7E6T8r2RuSpKWamPfT3I=', 'Content-Length': '0'} 2023-10-17T18:50:35.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078988ad36b241f94-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:50:35.069 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.070 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.070 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.071 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.071 DEBUG:boto:Method: GET 2023-10-17T18:50:35.072 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.072 DEBUG:boto:Data: 2023-10-17T18:50:35.072 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.073 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.074 DEBUG:boto:Params: {} 2023-10-17T18:50:35.074 DEBUG:boto:Token: None 2023-10-17T18:50:35.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.075 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/8NosSsP5A6aY/x6ZIus5Cukv7s= 2023-10-17T18:50:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/8NosSsP5A6aY/x6ZIus5Cukv7s=', 'Content-Length': '0'} 2023-10-17T18:50:35.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b61ca13d4acf31b-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.079 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.079 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.080 DEBUG:boto:path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.080 DEBUG:boto:auth_path=/hbmuog-56/a?versionId=null 2023-10-17T18:50:35.080 DEBUG:boto:Method: GET 2023-10-17T18:50:35.081 DEBUG:boto:Path: /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.081 DEBUG:boto:Data: 2023-10-17T18:50:35.082 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.083 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.083 DEBUG:boto:Params: {} 2023-10-17T18:50:35.084 DEBUG:boto:Token: None 2023-10-17T18:50:35.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a?versionId=null 2023-10-17T18:50:35.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/8NosSsP5A6aY/x6ZIus5Cukv7s= 2023-10-17T18:50:35.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/8NosSsP5A6aY/x6ZIus5Cukv7s=', 'Content-Length': '0'} 2023-10-17T18:50:35.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5aaef28fbf76f6-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.088 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.088 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.089 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.089 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.090 DEBUG:boto:Method: GET 2023-10-17T18:50:35.090 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:50:35.091 DEBUG:boto:Data: 2023-10-17T18:50:35.091 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.092 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.092 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.092 DEBUG:boto:Params: {} 2023-10-17T18:50:35.093 DEBUG:boto:Token: None 2023-10-17T18:50:35.093 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a847b92dd8808973-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.098 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.099 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.099 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.099 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.100 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.100 DEBUG:boto:Method: GET 2023-10-17T18:50:35.101 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:50:35.101 DEBUG:boto:Data: 2023-10-17T18:50:35.102 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.103 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.103 DEBUG:boto:Params: {} 2023-10-17T18:50:35.103 DEBUG:boto:Token: None 2023-10-17T18:50:35.104 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0200f1df12e9fd-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.108 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.108 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.109 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.109 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.109 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:50:35.110 DEBUG:boto:Data: 2023-10-17T18:50:35.110 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.111 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.112 DEBUG:boto:Params: {} 2023-10-17T18:50:35.112 DEBUG:boto:Token: None 2023-10-17T18:50:35.113 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a 2023-10-17T18:50:35.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dFqvfDJJH2rNUaYMadriTPVPc3g= 2023-10-17T18:50:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dFqvfDJJH2rNUaYMadriTPVPc3g=', 'Content-Length': '0'} 2023-10-17T18:50:35.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005958ff67de6d6e4e-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.116 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.116 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.117 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.117 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:50:35.118 DEBUG:boto:Data: 2023-10-17T18:50:35.118 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.119 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.119 DEBUG:boto:Params: {} 2023-10-17T18:50:35.122 DEBUG:boto:Token: None 2023-10-17T18:50:35.122 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a 2023-10-17T18:50:35.123 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dFqvfDJJH2rNUaYMadriTPVPc3g= 2023-10-17T18:50:35.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dFqvfDJJH2rNUaYMadriTPVPc3g=', 'Content-Length': '0'} 2023-10-17T18:50:35.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4432adcda5cf582-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:50:35.126 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.127 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.127 DEBUG:boto:Method: GET 2023-10-17T18:50:35.127 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:50:35.128 DEBUG:boto:Data: 2023-10-17T18:50:35.128 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.129 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.129 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.129 DEBUG:boto:Params: {} 2023-10-17T18:50:35.130 DEBUG:boto:Token: None 2023-10-17T18:50:35.130 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a 2023-10-17T18:50:35.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oNNZZuD14FO4j9jmfuOJs8Iis0U= 2023-10-17T18:50:35.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oNNZZuD14FO4j9jmfuOJs8Iis0U=', 'Content-Length': '0'} 2023-10-17T18:50:35.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee5f4bb3ab52274-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.134 DEBUG:boto:path=/hbmuog-56/a 2023-10-17T18:50:35.135 DEBUG:boto:auth_path=/hbmuog-56/a 2023-10-17T18:50:35.135 DEBUG:boto:Method: GET 2023-10-17T18:50:35.135 DEBUG:boto:Path: /hbmuog-56/a 2023-10-17T18:50:35.136 DEBUG:boto:Data: 2023-10-17T18:50:35.136 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.137 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.137 DEBUG:boto:Params: {} 2023-10-17T18:50:35.138 DEBUG:boto:Token: None 2023-10-17T18:50:35.138 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/a 2023-10-17T18:50:35.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oNNZZuD14FO4j9jmfuOJs8Iis0U= 2023-10-17T18:50:35.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oNNZZuD14FO4j9jmfuOJs8Iis0U=', 'Content-Length': '0'} 2023-10-17T18:50:35.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1ced4fa1d00ddb-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.182 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.183 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.183 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.184 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.184 DEBUG:boto:Method: GET 2023-10-17T18:50:35.184 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:50:35.185 DEBUG:boto:Data: 2023-10-17T18:50:35.185 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.186 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.186 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.187 DEBUG:boto:Params: {} 2023-10-17T18:50:35.187 DEBUG:boto:Token: None 2023-10-17T18:50:35.188 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.188 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e53735a58290531-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.192 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.193 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.193 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.194 DEBUG:boto:path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.194 DEBUG:boto:auth_path=/hbmuog-56/?prefix=a 2023-10-17T18:50:35.195 DEBUG:boto:Method: GET 2023-10-17T18:50:35.195 DEBUG:boto:Path: /hbmuog-56/?prefix=a 2023-10-17T18:50:35.195 DEBUG:boto:Data: 2023-10-17T18:50:35.196 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.196 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.197 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.197 DEBUG:boto:Params: {} 2023-10-17T18:50:35.198 DEBUG:boto:Token: None 2023-10-17T18:50:35.198 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.199 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae45e1ee230e572-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.202 DEBUG:boto:b'hbmuog-56a1000falsea2023-10-17T18:47:47.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:50:35.203 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.203 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.204 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.204 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.204 DEBUG:boto:Method: GET 2023-10-17T18:50:35.205 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.205 DEBUG:boto:Data: 2023-10-17T18:50:35.206 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.207 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.207 DEBUG:boto:Params: {} 2023-10-17T18:50:35.208 DEBUG:boto:Token: None 2023-10-17T18:50:35.208 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo= 2023-10-17T18:50:35.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo=', 'Content-Length': '0'} 2023-10-17T18:50:35.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375f02509fa698dc-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.212 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.212 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.213 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.213 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.214 DEBUG:boto:Method: GET 2023-10-17T18:50:35.214 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.215 DEBUG:boto:Data: 2023-10-17T18:50:35.215 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.216 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.216 DEBUG:boto:Params: {} 2023-10-17T18:50:35.217 DEBUG:boto:Token: None 2023-10-17T18:50:35.217 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo= 2023-10-17T18:50:35.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo=', 'Content-Length': '0'} 2023-10-17T18:50:35.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b8ccf2e64cf0a3-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.221 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.221 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.222 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.222 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.223 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.223 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.224 DEBUG:boto:Data: 2023-10-17T18:50:35.224 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.224 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.225 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.225 DEBUG:boto:Params: {} 2023-10-17T18:50:35.226 DEBUG:boto:Token: None 2023-10-17T18:50:35.226 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:atVt5baX/KOfbcQvAdjvsopi8og= 2023-10-17T18:50:35.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:atVt5baX/KOfbcQvAdjvsopi8og=', 'Content-Length': '0'} 2023-10-17T18:50:35.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37ec76462b127eb-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.229 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.230 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.230 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.231 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.231 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.232 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.232 DEBUG:boto:Data: 2023-10-17T18:50:35.232 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.233 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.233 DEBUG:boto:Params: {} 2023-10-17T18:50:35.234 DEBUG:boto:Token: None 2023-10-17T18:50:35.234 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:atVt5baX/KOfbcQvAdjvsopi8og= 2023-10-17T18:50:35.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:atVt5baX/KOfbcQvAdjvsopi8og=', 'Content-Length': '0'} 2023-10-17T18:50:35.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009562424576531d5c-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:50:35.238 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.239 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.239 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.240 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.240 DEBUG:boto:Method: GET 2023-10-17T18:50:35.241 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.241 DEBUG:boto:Data: 2023-10-17T18:50:35.241 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.242 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.243 DEBUG:boto:Params: {} 2023-10-17T18:50:35.243 DEBUG:boto:Token: None 2023-10-17T18:50:35.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo= 2023-10-17T18:50:35.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo=', 'Content-Length': '0'} 2023-10-17T18:50:35.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b192a255dd77b231-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.248 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.248 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.248 DEBUG:boto:path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.249 DEBUG:boto:auth_path=/hbmuog-56/b?versionId=null 2023-10-17T18:50:35.249 DEBUG:boto:Method: GET 2023-10-17T18:50:35.250 DEBUG:boto:Path: /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.250 DEBUG:boto:Data: 2023-10-17T18:50:35.251 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.252 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.252 DEBUG:boto:Params: {} 2023-10-17T18:50:35.252 DEBUG:boto:Token: None 2023-10-17T18:50:35.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b?versionId=null 2023-10-17T18:50:35.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo= 2023-10-17T18:50:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCfj2K5yxosZEL2XeWyNVF0Yvgo=', 'Content-Length': '0'} 2023-10-17T18:50:35.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0e7558003aa700-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.257 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.257 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.258 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.258 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.259 DEBUG:boto:Method: GET 2023-10-17T18:50:35.259 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:50:35.260 DEBUG:boto:Data: 2023-10-17T18:50:35.260 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.261 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.261 DEBUG:boto:Params: {} 2023-10-17T18:50:35.262 DEBUG:boto:Token: None 2023-10-17T18:50:35.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069443ded596be1b9-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.314 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.315 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.316 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.316 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.317 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.317 DEBUG:boto:Method: GET 2023-10-17T18:50:35.317 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:50:35.318 DEBUG:boto:Data: 2023-10-17T18:50:35.318 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.319 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.320 DEBUG:boto:Params: {} 2023-10-17T18:50:35.320 DEBUG:boto:Token: None 2023-10-17T18:50:35.321 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d05b52eef6e879-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.324 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.325 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.326 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.326 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.326 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:50:35.327 DEBUG:boto:Data: 2023-10-17T18:50:35.327 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.328 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.328 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.329 DEBUG:boto:Params: {} 2023-10-17T18:50:35.329 DEBUG:boto:Token: None 2023-10-17T18:50:35.329 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b 2023-10-17T18:50:35.330 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hi7v0rwwFHxJvjeeqxnXU1qBmYk= 2023-10-17T18:50:35.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hi7v0rwwFHxJvjeeqxnXU1qBmYk=', 'Content-Length': '0'} 2023-10-17T18:50:35.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bdc40d243120f5-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.333 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.334 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.334 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.335 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:50:35.335 DEBUG:boto:Data: 2023-10-17T18:50:35.335 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.336 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.337 DEBUG:boto:Params: {} 2023-10-17T18:50:35.337 DEBUG:boto:Token: None 2023-10-17T18:50:35.337 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b 2023-10-17T18:50:35.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hi7v0rwwFHxJvjeeqxnXU1qBmYk= 2023-10-17T18:50:35.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hi7v0rwwFHxJvjeeqxnXU1qBmYk=', 'Content-Length': '0'} 2023-10-17T18:50:35.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ac1b5f69c535d1-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:50:35.342 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.342 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.343 DEBUG:boto:Method: GET 2023-10-17T18:50:35.343 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:50:35.343 DEBUG:boto:Data: 2023-10-17T18:50:35.344 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.345 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.345 DEBUG:boto:Params: {} 2023-10-17T18:50:35.346 DEBUG:boto:Token: None 2023-10-17T18:50:35.346 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b 2023-10-17T18:50:35.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9Y2xomLi790EcV5Z6MpsSHdLiEw= 2023-10-17T18:50:35.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9Y2xomLi790EcV5Z6MpsSHdLiEw=', 'Content-Length': '0'} 2023-10-17T18:50:35.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e4a81fce226442-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.350 DEBUG:boto:path=/hbmuog-56/b 2023-10-17T18:50:35.350 DEBUG:boto:auth_path=/hbmuog-56/b 2023-10-17T18:50:35.351 DEBUG:boto:Method: GET 2023-10-17T18:50:35.351 DEBUG:boto:Path: /hbmuog-56/b 2023-10-17T18:50:35.352 DEBUG:boto:Data: 2023-10-17T18:50:35.352 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.353 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.353 DEBUG:boto:Params: {} 2023-10-17T18:50:35.354 DEBUG:boto:Token: None 2023-10-17T18:50:35.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/b 2023-10-17T18:50:35.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9Y2xomLi790EcV5Z6MpsSHdLiEw= 2023-10-17T18:50:35.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9Y2xomLi790EcV5Z6MpsSHdLiEw=', 'Content-Length': '0'} 2023-10-17T18:50:35.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504ca7d6700ca077-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.358 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.359 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.359 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.359 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.360 DEBUG:boto:Method: GET 2023-10-17T18:50:35.360 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:50:35.361 DEBUG:boto:Data: 2023-10-17T18:50:35.361 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.362 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.362 DEBUG:boto:Params: {} 2023-10-17T18:50:35.363 DEBUG:boto:Token: None 2023-10-17T18:50:35.363 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb25769ee49d7e8-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.368 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.369 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.369 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.370 DEBUG:boto:path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.370 DEBUG:boto:auth_path=/hbmuog-56/?prefix=b 2023-10-17T18:50:35.370 DEBUG:boto:Method: GET 2023-10-17T18:50:35.371 DEBUG:boto:Path: /hbmuog-56/?prefix=b 2023-10-17T18:50:35.371 DEBUG:boto:Data: 2023-10-17T18:50:35.372 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.373 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.373 DEBUG:boto:Params: {} 2023-10-17T18:50:35.374 DEBUG:boto:Token: None 2023-10-17T18:50:35.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008bdca1b2548876b-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.378 DEBUG:boto:b'hbmuog-56b1000falseb2023-10-17T18:47:47.870Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:50:35.379 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.380 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.380 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.380 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.381 DEBUG:boto:Method: GET 2023-10-17T18:50:35.381 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.382 DEBUG:boto:Data: 2023-10-17T18:50:35.382 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.383 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.384 DEBUG:boto:Params: {} 2023-10-17T18:50:35.384 DEBUG:boto:Token: None 2023-10-17T18:50:35.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ= 2023-10-17T18:50:35.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b382a537c0865a-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.388 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.389 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.389 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.390 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.390 DEBUG:boto:Method: GET 2023-10-17T18:50:35.390 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.391 DEBUG:boto:Data: 2023-10-17T18:50:35.391 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.392 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.392 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.393 DEBUG:boto:Params: {} 2023-10-17T18:50:35.393 DEBUG:boto:Token: None 2023-10-17T18:50:35.394 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.394 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ= 2023-10-17T18:50:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b0fc649450ad1d-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.397 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.398 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.398 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.399 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.399 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.400 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.400 DEBUG:boto:Data: 2023-10-17T18:50:35.400 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.401 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.401 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.402 DEBUG:boto:Params: {} 2023-10-17T18:50:35.402 DEBUG:boto:Token: None 2023-10-17T18:50:35.403 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.403 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:19KoyDdgNS6gWTPTW3VXa79Rhgk= 2023-10-17T18:50:35.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:19KoyDdgNS6gWTPTW3VXa79Rhgk=', 'Content-Length': '0'} 2023-10-17T18:50:35.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae12383d240e385f-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.406 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.406 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.407 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.407 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.408 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.408 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.408 DEBUG:boto:Data: 2023-10-17T18:50:35.409 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.410 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.410 DEBUG:boto:Params: {} 2023-10-17T18:50:35.411 DEBUG:boto:Token: None 2023-10-17T18:50:35.411 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:19KoyDdgNS6gWTPTW3VXa79Rhgk= 2023-10-17T18:50:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:19KoyDdgNS6gWTPTW3VXa79Rhgk=', 'Content-Length': '0'} 2023-10-17T18:50:35.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621380945470939f-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:50:35.415 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.416 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.416 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.416 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.417 DEBUG:boto:Method: GET 2023-10-17T18:50:35.417 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.418 DEBUG:boto:Data: 2023-10-17T18:50:35.418 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.419 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.419 DEBUG:boto:Params: {} 2023-10-17T18:50:35.420 DEBUG:boto:Token: None 2023-10-17T18:50:35.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ= 2023-10-17T18:50:35.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c739c57e26ea575-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.424 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.425 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.425 DEBUG:boto:path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.426 DEBUG:boto:auth_path=/hbmuog-56/c?versionId=null 2023-10-17T18:50:35.426 DEBUG:boto:Method: GET 2023-10-17T18:50:35.427 DEBUG:boto:Path: /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.427 DEBUG:boto:Data: 2023-10-17T18:50:35.427 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.428 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.429 DEBUG:boto:Params: {} 2023-10-17T18:50:35.429 DEBUG:boto:Token: None 2023-10-17T18:50:35.430 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c?versionId=null 2023-10-17T18:50:35.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ= 2023-10-17T18:50:35.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D28nV0XtdhpgOvNw43mzoSS7/PQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ed618d178b0ea7-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.434 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.434 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.435 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.435 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.435 DEBUG:boto:Method: GET 2023-10-17T18:50:35.436 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:50:35.436 DEBUG:boto:Data: 2023-10-17T18:50:35.437 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.437 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.438 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.438 DEBUG:boto:Params: {} 2023-10-17T18:50:35.438 DEBUG:boto:Token: None 2023-10-17T18:50:35.439 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000174c3b0dbe419598-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.443 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.444 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.445 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.445 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.445 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.446 DEBUG:boto:Method: GET 2023-10-17T18:50:35.446 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:50:35.447 DEBUG:boto:Data: 2023-10-17T18:50:35.447 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.448 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.448 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.448 DEBUG:boto:Params: {} 2023-10-17T18:50:35.449 DEBUG:boto:Token: None 2023-10-17T18:50:35.449 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.450 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e74c1d76cf3ac75-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.453 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.454 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.454 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.455 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.455 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:50:35.455 DEBUG:boto:Data: 2023-10-17T18:50:35.456 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.457 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.457 DEBUG:boto:Params: {} 2023-10-17T18:50:35.457 DEBUG:boto:Token: None 2023-10-17T18:50:35.458 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c 2023-10-17T18:50:35.458 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K2C4WZGzMBWIaqVOvMX/oyFKh+4= 2023-10-17T18:50:35.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K2C4WZGzMBWIaqVOvMX/oyFKh+4=', 'Content-Length': '0'} 2023-10-17T18:50:35.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd16df39dcda03e9-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.461 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.462 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.462 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.462 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:50:35.463 DEBUG:boto:Data: 2023-10-17T18:50:35.463 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.464 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.464 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.464 DEBUG:boto:Params: {} 2023-10-17T18:50:35.465 DEBUG:boto:Token: None 2023-10-17T18:50:35.465 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c 2023-10-17T18:50:35.466 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K2C4WZGzMBWIaqVOvMX/oyFKh+4= 2023-10-17T18:50:35.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K2C4WZGzMBWIaqVOvMX/oyFKh+4=', 'Content-Length': '0'} 2023-10-17T18:50:35.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba4d8f678af9801-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:50:35.469 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.470 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.470 DEBUG:boto:Method: GET 2023-10-17T18:50:35.471 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:50:35.471 DEBUG:boto:Data: 2023-10-17T18:50:35.472 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.472 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.472 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.473 DEBUG:boto:Params: {} 2023-10-17T18:50:35.473 DEBUG:boto:Token: None 2023-10-17T18:50:35.474 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c 2023-10-17T18:50:35.474 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RgE8pnN0Y7CaNpz53gx1T5xcxw0= 2023-10-17T18:50:35.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RgE8pnN0Y7CaNpz53gx1T5xcxw0=', 'Content-Length': '0'} 2023-10-17T18:50:35.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e2226d42fe0b46-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.518 DEBUG:boto:path=/hbmuog-56/c 2023-10-17T18:50:35.519 DEBUG:boto:auth_path=/hbmuog-56/c 2023-10-17T18:50:35.520 DEBUG:boto:Method: GET 2023-10-17T18:50:35.520 DEBUG:boto:Path: /hbmuog-56/c 2023-10-17T18:50:35.520 DEBUG:boto:Data: 2023-10-17T18:50:35.521 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.522 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.522 DEBUG:boto:Params: {} 2023-10-17T18:50:35.523 DEBUG:boto:Token: None 2023-10-17T18:50:35.523 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/c 2023-10-17T18:50:35.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RgE8pnN0Y7CaNpz53gx1T5xcxw0= 2023-10-17T18:50:35.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RgE8pnN0Y7CaNpz53gx1T5xcxw0=', 'Content-Length': '0'} 2023-10-17T18:50:35.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ca586b63bf051f-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.527 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.528 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.528 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.528 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.529 DEBUG:boto:Method: GET 2023-10-17T18:50:35.529 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:50:35.530 DEBUG:boto:Data: 2023-10-17T18:50:35.534 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.535 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.535 DEBUG:boto:Params: {} 2023-10-17T18:50:35.536 DEBUG:boto:Token: None 2023-10-17T18:50:35.536 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.537 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03c4bf5ec4feb22-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.541 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.542 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.542 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.543 DEBUG:boto:path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.543 DEBUG:boto:auth_path=/hbmuog-56/?prefix=c 2023-10-17T18:50:35.544 DEBUG:boto:Method: GET 2023-10-17T18:50:35.544 DEBUG:boto:Path: /hbmuog-56/?prefix=c 2023-10-17T18:50:35.544 DEBUG:boto:Data: 2023-10-17T18:50:35.545 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.545 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.546 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.546 DEBUG:boto:Params: {} 2023-10-17T18:50:35.547 DEBUG:boto:Token: None 2023-10-17T18:50:35.547 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123749cb8ebecba8-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.551 DEBUG:boto:b'hbmuog-56c1000falsec2023-10-17T18:47:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:50:35.552 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.552 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.553 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.553 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.554 DEBUG:boto:Method: GET 2023-10-17T18:50:35.554 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.554 DEBUG:boto:Data: 2023-10-17T18:50:35.555 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.555 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.556 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.556 DEBUG:boto:Params: {} 2023-10-17T18:50:35.556 DEBUG:boto:Token: None 2023-10-17T18:50:35.557 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.557 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ= 2023-10-17T18:50:35.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26164c9e6be4b25-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.561 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.561 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.562 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.562 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.562 DEBUG:boto:Method: GET 2023-10-17T18:50:35.563 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.563 DEBUG:boto:Data: 2023-10-17T18:50:35.564 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.564 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.565 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.565 DEBUG:boto:Params: {} 2023-10-17T18:50:35.566 DEBUG:boto:Token: None 2023-10-17T18:50:35.566 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.566 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ= 2023-10-17T18:50:35.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f55a66882eb4e26-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.610 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.611 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.611 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.612 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.612 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.613 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.613 DEBUG:boto:Data: 2023-10-17T18:50:35.614 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.615 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.615 DEBUG:boto:Params: {} 2023-10-17T18:50:35.615 DEBUG:boto:Token: None 2023-10-17T18:50:35.616 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U18a7KLo42nyDvc5Y9kWgiTFct0= 2023-10-17T18:50:35.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U18a7KLo42nyDvc5Y9kWgiTFct0=', 'Content-Length': '0'} 2023-10-17T18:50:35.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e17a29c6b0ccfe8-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.619 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.620 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.620 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.621 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.621 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.621 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.622 DEBUG:boto:Data: 2023-10-17T18:50:35.622 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.623 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.623 DEBUG:boto:Params: {} 2023-10-17T18:50:35.624 DEBUG:boto:Token: None 2023-10-17T18:50:35.624 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U18a7KLo42nyDvc5Y9kWgiTFct0= 2023-10-17T18:50:35.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U18a7KLo42nyDvc5Y9kWgiTFct0=', 'Content-Length': '0'} 2023-10-17T18:50:35.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000059999b3b80d8fc7-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:50:35.628 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.629 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.629 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.629 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.630 DEBUG:boto:Method: GET 2023-10-17T18:50:35.630 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.631 DEBUG:boto:Data: 2023-10-17T18:50:35.631 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.632 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.633 DEBUG:boto:Params: {} 2023-10-17T18:50:35.633 DEBUG:boto:Token: None 2023-10-17T18:50:35.633 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ= 2023-10-17T18:50:35.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4525eac4645701c-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.678 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.679 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.680 DEBUG:boto:path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.680 DEBUG:boto:auth_path=/hbmuog-56/d?versionId=null 2023-10-17T18:50:35.681 DEBUG:boto:Method: GET 2023-10-17T18:50:35.681 DEBUG:boto:Path: /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.682 DEBUG:boto:Data: 2023-10-17T18:50:35.682 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.683 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.683 DEBUG:boto:Params: {} 2023-10-17T18:50:35.684 DEBUG:boto:Token: None 2023-10-17T18:50:35.684 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d?versionId=null 2023-10-17T18:50:35.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ= 2023-10-17T18:50:35.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jZebdAFSmWBgnrgR4aoilL4pjPQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140937ee82498ca4-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.689 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.689 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.689 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.690 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.690 DEBUG:boto:Method: GET 2023-10-17T18:50:35.691 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:50:35.691 DEBUG:boto:Data: 2023-10-17T18:50:35.692 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.693 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.693 DEBUG:boto:Params: {} 2023-10-17T18:50:35.694 DEBUG:boto:Token: None 2023-10-17T18:50:35.694 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.695 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b4ed3bbdb50bd7-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.699 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.699 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.700 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.700 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.701 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.701 DEBUG:boto:Method: GET 2023-10-17T18:50:35.702 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:50:35.702 DEBUG:boto:Data: 2023-10-17T18:50:35.703 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.703 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.704 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.704 DEBUG:boto:Params: {} 2023-10-17T18:50:35.705 DEBUG:boto:Token: None 2023-10-17T18:50:35.705 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.706 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071db00777a8d0612-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.711 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.712 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.713 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.713 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.713 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:50:35.714 DEBUG:boto:Data: 2023-10-17T18:50:35.714 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.715 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.716 DEBUG:boto:Params: {} 2023-10-17T18:50:35.716 DEBUG:boto:Token: None 2023-10-17T18:50:35.717 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d 2023-10-17T18:50:35.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZyQ3UO15RlJJtdoAhaCM/INqJTo= 2023-10-17T18:50:35.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZyQ3UO15RlJJtdoAhaCM/INqJTo=', 'Content-Length': '0'} 2023-10-17T18:50:35.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d5f228a0c67611-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.721 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.721 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.722 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.722 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:50:35.722 DEBUG:boto:Data: 2023-10-17T18:50:35.723 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.723 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.724 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.724 DEBUG:boto:Params: {} 2023-10-17T18:50:35.725 DEBUG:boto:Token: None 2023-10-17T18:50:35.726 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d 2023-10-17T18:50:35.727 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZyQ3UO15RlJJtdoAhaCM/INqJTo= 2023-10-17T18:50:35.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZyQ3UO15RlJJtdoAhaCM/INqJTo=', 'Content-Length': '0'} 2023-10-17T18:50:35.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58675e58735f059-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:50:35.731 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.731 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.732 DEBUG:boto:Method: GET 2023-10-17T18:50:35.732 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:50:35.733 DEBUG:boto:Data: 2023-10-17T18:50:35.733 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.733 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.734 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.734 DEBUG:boto:Params: {} 2023-10-17T18:50:35.735 DEBUG:boto:Token: None 2023-10-17T18:50:35.736 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d 2023-10-17T18:50:35.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3qUGZ6thtlRvaRvT8WbnJZi4ymA= 2023-10-17T18:50:35.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3qUGZ6thtlRvaRvT8WbnJZi4ymA=', 'Content-Length': '0'} 2023-10-17T18:50:35.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab07e5a27b056e1-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.739 DEBUG:boto:path=/hbmuog-56/d 2023-10-17T18:50:35.740 DEBUG:boto:auth_path=/hbmuog-56/d 2023-10-17T18:50:35.740 DEBUG:boto:Method: GET 2023-10-17T18:50:35.741 DEBUG:boto:Path: /hbmuog-56/d 2023-10-17T18:50:35.741 DEBUG:boto:Data: 2023-10-17T18:50:35.742 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.743 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.743 DEBUG:boto:Params: {} 2023-10-17T18:50:35.744 DEBUG:boto:Token: None 2023-10-17T18:50:35.744 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/d 2023-10-17T18:50:35.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3qUGZ6thtlRvaRvT8WbnJZi4ymA= 2023-10-17T18:50:35.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3qUGZ6thtlRvaRvT8WbnJZi4ymA=', 'Content-Length': '0'} 2023-10-17T18:50:35.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:47:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dccbad6f8ca3a3b2-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.748 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.749 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.749 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.750 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.750 DEBUG:boto:Method: GET 2023-10-17T18:50:35.751 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:50:35.751 DEBUG:boto:Data: 2023-10-17T18:50:35.752 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.752 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.753 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.753 DEBUG:boto:Params: {} 2023-10-17T18:50:35.754 DEBUG:boto:Token: None 2023-10-17T18:50:35.754 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.755 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba83e20766cbf92c-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.802 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.803 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.804 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.804 DEBUG:boto:path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.805 DEBUG:boto:auth_path=/hbmuog-56/?prefix=d 2023-10-17T18:50:35.805 DEBUG:boto:Method: GET 2023-10-17T18:50:35.806 DEBUG:boto:Path: /hbmuog-56/?prefix=d 2023-10-17T18:50:35.806 DEBUG:boto:Data: 2023-10-17T18:50:35.807 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.808 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.808 DEBUG:boto:Params: {} 2023-10-17T18:50:35.809 DEBUG:boto:Token: None 2023-10-17T18:50:35.809 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.810 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c2374e0274a354-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.813 DEBUG:boto:b'hbmuog-56d1000falsed2023-10-17T18:47:47.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:50:35.814 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.815 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.815 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.816 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.816 DEBUG:boto:Method: GET 2023-10-17T18:50:35.817 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.817 DEBUG:boto:Data: 2023-10-17T18:50:35.818 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.819 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.819 DEBUG:boto:Params: {} 2023-10-17T18:50:35.820 DEBUG:boto:Token: None 2023-10-17T18:50:35.820 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0= 2023-10-17T18:50:35.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0=', 'Content-Length': '0'} 2023-10-17T18:50:35.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f7675f956d28fd-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.824 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.825 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.825 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.825 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.826 DEBUG:boto:Method: GET 2023-10-17T18:50:35.826 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.827 DEBUG:boto:Data: 2023-10-17T18:50:35.827 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.828 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.828 DEBUG:boto:Params: {} 2023-10-17T18:50:35.829 DEBUG:boto:Token: None 2023-10-17T18:50:35.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0= 2023-10-17T18:50:35.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0=', 'Content-Length': '0'} 2023-10-17T18:50:35.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004abffff5db7c326-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.833 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.833 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.834 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.834 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.835 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.835 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.835 DEBUG:boto:Data: 2023-10-17T18:50:35.836 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.837 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.837 DEBUG:boto:Params: {} 2023-10-17T18:50:35.837 DEBUG:boto:Token: None 2023-10-17T18:50:35.838 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rtkmGtil+c9AuIRsKddukEyWHYk= 2023-10-17T18:50:35.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rtkmGtil+c9AuIRsKddukEyWHYk=', 'Content-Length': '0'} 2023-10-17T18:50:35.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a78118f77b1a970c-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.841 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.842 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.842 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.843 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.843 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.844 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.844 DEBUG:boto:Data: 2023-10-17T18:50:35.844 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.845 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.846 DEBUG:boto:Params: {} 2023-10-17T18:50:35.846 DEBUG:boto:Token: None 2023-10-17T18:50:35.846 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rtkmGtil+c9AuIRsKddukEyWHYk= 2023-10-17T18:50:35.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rtkmGtil+c9AuIRsKddukEyWHYk=', 'Content-Length': '0'} 2023-10-17T18:50:35.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b24c2a6283c9a1-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:50:35.850 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.851 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.851 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.852 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.852 DEBUG:boto:Method: GET 2023-10-17T18:50:35.852 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.853 DEBUG:boto:Data: 2023-10-17T18:50:35.853 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.854 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.854 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.855 DEBUG:boto:Params: {} 2023-10-17T18:50:35.855 DEBUG:boto:Token: None 2023-10-17T18:50:35.856 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0= 2023-10-17T18:50:35.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0=', 'Content-Length': '0'} 2023-10-17T18:50:35.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065bc9a546b7f8a86-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.859 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.860 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.860 DEBUG:boto:path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.860 DEBUG:boto:auth_path=/hbmuog-56/e?versionId=null 2023-10-17T18:50:35.861 DEBUG:boto:Method: GET 2023-10-17T18:50:35.861 DEBUG:boto:Path: /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.862 DEBUG:boto:Data: 2023-10-17T18:50:35.862 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.863 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.863 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.863 DEBUG:boto:Params: {} 2023-10-17T18:50:35.864 DEBUG:boto:Token: None 2023-10-17T18:50:35.864 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e?versionId=null 2023-10-17T18:50:35.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0= 2023-10-17T18:50:35.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vWpFQeDX6LKVwSsXWreyMjKGJl0=', 'Content-Length': '0'} 2023-10-17T18:50:35.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb463feb1507d074-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.868 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.869 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.869 DEBUG:boto:path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.869 DEBUG:boto:auth_path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.870 DEBUG:boto:Method: GET 2023-10-17T18:50:35.870 DEBUG:boto:Path: /hbmuog-56/?prefix=e 2023-10-17T18:50:35.871 DEBUG:boto:Data: 2023-10-17T18:50:35.871 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.872 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.872 DEBUG:boto:Params: {} 2023-10-17T18:50:35.873 DEBUG:boto:Token: None 2023-10-17T18:50:35.873 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058380b952e223acd-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.918 DEBUG:boto:b'hbmuog-56e1000falsee2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.919 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.920 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.920 DEBUG:boto:path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.921 DEBUG:boto:auth_path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.921 DEBUG:boto:Method: GET 2023-10-17T18:50:35.921 DEBUG:boto:Path: /hbmuog-56/?prefix=e 2023-10-17T18:50:35.922 DEBUG:boto:Data: 2023-10-17T18:50:35.922 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.923 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.923 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.924 DEBUG:boto:Params: {} 2023-10-17T18:50:35.924 DEBUG:boto:Token: None 2023-10-17T18:50:35.925 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.925 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c4e1e25283b33f-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.928 DEBUG:boto:b'hbmuog-56e1000falsee2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.929 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.930 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.930 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.930 DEBUG:boto:Path: /hbmuog-56/e 2023-10-17T18:50:35.931 DEBUG:boto:Data: 2023-10-17T18:50:35.931 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.932 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.932 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.932 DEBUG:boto:Params: {} 2023-10-17T18:50:35.933 DEBUG:boto:Token: None 2023-10-17T18:50:35.933 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e 2023-10-17T18:50:35.934 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x3PAEILUbHExvsvY9+OFRtI4Zgs= 2023-10-17T18:50:35.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x3PAEILUbHExvsvY9+OFRtI4Zgs=', 'Content-Length': '0'} 2023-10-17T18:50:35.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8dab3d0b96104b-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.937 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.937 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.938 DEBUG:boto:Method: HEAD 2023-10-17T18:50:35.938 DEBUG:boto:Path: /hbmuog-56/e 2023-10-17T18:50:35.939 DEBUG:boto:Data: 2023-10-17T18:50:35.939 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.940 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.940 DEBUG:boto:Params: {} 2023-10-17T18:50:35.941 DEBUG:boto:Token: None 2023-10-17T18:50:35.941 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e 2023-10-17T18:50:35.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x3PAEILUbHExvsvY9+OFRtI4Zgs= 2023-10-17T18:50:35.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x3PAEILUbHExvsvY9+OFRtI4Zgs=', 'Content-Length': '0'} 2023-10-17T18:50:35.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a95a4db3a9e00b49-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:50:35.945 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.946 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.946 DEBUG:boto:Method: GET 2023-10-17T18:50:35.947 DEBUG:boto:Path: /hbmuog-56/e 2023-10-17T18:50:35.947 DEBUG:boto:Data: 2023-10-17T18:50:35.947 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.948 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.948 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.949 DEBUG:boto:Params: {} 2023-10-17T18:50:35.949 DEBUG:boto:Token: None 2023-10-17T18:50:35.950 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e 2023-10-17T18:50:35.950 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CyjpSi6jQVRz393UCfWnLd0WIoM= 2023-10-17T18:50:35.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CyjpSi6jQVRz393UCfWnLd0WIoM=', 'Content-Length': '0'} 2023-10-17T18:50:35.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be94cfcfbd5e52cd-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.953 DEBUG:boto:path=/hbmuog-56/e 2023-10-17T18:50:35.954 DEBUG:boto:auth_path=/hbmuog-56/e 2023-10-17T18:50:35.954 DEBUG:boto:Method: GET 2023-10-17T18:50:35.955 DEBUG:boto:Path: /hbmuog-56/e 2023-10-17T18:50:35.955 DEBUG:boto:Data: 2023-10-17T18:50:35.955 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.956 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.956 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.957 DEBUG:boto:Params: {} 2023-10-17T18:50:35.957 DEBUG:boto:Token: None 2023-10-17T18:50:35.958 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/e 2023-10-17T18:50:35.958 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CyjpSi6jQVRz393UCfWnLd0WIoM= 2023-10-17T18:50:35.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CyjpSi6jQVRz393UCfWnLd0WIoM=', 'Content-Length': '0'} 2023-10-17T18:50:35.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a16a7923a4c6e185-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.961 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.962 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.962 DEBUG:boto:path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.963 DEBUG:boto:auth_path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.963 DEBUG:boto:Method: GET 2023-10-17T18:50:35.964 DEBUG:boto:Path: /hbmuog-56/?prefix=e 2023-10-17T18:50:35.964 DEBUG:boto:Data: 2023-10-17T18:50:35.964 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.965 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.966 DEBUG:boto:Params: {} 2023-10-17T18:50:35.966 DEBUG:boto:Token: None 2023-10-17T18:50:35.967 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbd9a50bac61a758-00652ed77b-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.971 DEBUG:boto:b'hbmuog-56e1000falsee2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.972 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:35.972 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:35.973 DEBUG:boto:path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.973 DEBUG:boto:auth_path=/hbmuog-56/?prefix=e 2023-10-17T18:50:35.974 DEBUG:boto:Method: GET 2023-10-17T18:50:35.974 DEBUG:boto:Path: /hbmuog-56/?prefix=e 2023-10-17T18:50:35.974 DEBUG:boto:Data: 2023-10-17T18:50:35.975 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.975 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.976 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.976 DEBUG:boto:Params: {} 2023-10-17T18:50:35.977 DEBUG:boto:Token: None 2023-10-17T18:50:35.977 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/ 2023-10-17T18:50:35.978 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ= 2023-10-17T18:50:35.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x1PTc5+uoNQ1nI00ear1MZGpTzQ=', 'Content-Length': '0'} 2023-10-17T18:50:35.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef1ca36ac6000e2-00652ed77b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.981 DEBUG:boto:b'hbmuog-56e1000falsee2023-10-17T18:48:14.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:35.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:50:35.982 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:35.982 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:35.983 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:35.983 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:35.984 DEBUG:boto:Method: GET 2023-10-17T18:50:35.984 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:35.985 DEBUG:boto:Data: 2023-10-17T18:50:35.985 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.985 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:35.986 DEBUG:boto:Port: 8001 2023-10-17T18:50:35.986 DEBUG:boto:Params: {} 2023-10-17T18:50:35.987 DEBUG:boto:Token: None 2023-10-17T18:50:35.987 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:35.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U9R+woWC4XKxmC06wSO+n5UxNJc= 2023-10-17T18:50:35.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U9R+woWC4XKxmC06wSO+n5UxNJc=', 'Content-Length': '0'} 2023-10-17T18:50:35.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f327c9dca5cffac9-00652ed77b-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:35.991 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:35.991 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:35.992 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:35.992 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:35.993 DEBUG:boto:Method: GET 2023-10-17T18:50:35.993 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:35.993 DEBUG:boto:Data: 2023-10-17T18:50:35.994 DEBUG:boto:Headers: {} 2023-10-17T18:50:35.994 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:35.995 DEBUG:boto:Port: 8000 2023-10-17T18:50:35.995 DEBUG:boto:Params: {} 2023-10-17T18:50:35.996 DEBUG:boto:Token: None 2023-10-17T18:50:35.996 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:35 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:35.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U9R+woWC4XKxmC06wSO+n5UxNJc= 2023-10-17T18:50:35.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U9R+woWC4XKxmC06wSO+n5UxNJc=', 'Content-Length': '0'} 2023-10-17T18:50:35.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f748c87f39a70324-00652ed77b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.000 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.000 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.000 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.001 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.001 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.002 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.002 DEBUG:boto:Data: 2023-10-17T18:50:36.002 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.003 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.004 DEBUG:boto:Params: {} 2023-10-17T18:50:36.004 DEBUG:boto:Token: None 2023-10-17T18:50:36.005 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G4j74y6ald/8jSf1/wbP9qO6/yQ= 2023-10-17T18:50:36.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G4j74y6ald/8jSf1/wbP9qO6/yQ=', 'Content-Length': '0'} 2023-10-17T18:50:36.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315cd10843cc7be3-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.008 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.008 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.009 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.009 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.010 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.010 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.010 DEBUG:boto:Data: 2023-10-17T18:50:36.011 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.012 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.012 DEBUG:boto:Params: {} 2023-10-17T18:50:36.013 DEBUG:boto:Token: None 2023-10-17T18:50:36.013 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G4j74y6ald/8jSf1/wbP9qO6/yQ= 2023-10-17T18:50:36.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G4j74y6ald/8jSf1/wbP9qO6/yQ=', 'Content-Length': '0'} 2023-10-17T18:50:36.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000461baf146e9a2ce-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:50:36.017 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.017 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.018 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.018 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.019 DEBUG:boto:Method: GET 2023-10-17T18:50:36.019 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.020 DEBUG:boto:Data: 2023-10-17T18:50:36.020 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.020 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.021 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.021 DEBUG:boto:Params: {} 2023-10-17T18:50:36.022 DEBUG:boto:Token: None 2023-10-17T18:50:36.022 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aVks5PSpJO1+RBmzz2Ugkrotets= 2023-10-17T18:50:36.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aVks5PSpJO1+RBmzz2Ugkrotets=', 'Content-Length': '0'} 2023-10-17T18:50:36.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76eb57f2ce4854b-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.066 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.067 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.067 DEBUG:boto:path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.068 DEBUG:boto:auth_path=/hbmuog-56/f?versionId=null 2023-10-17T18:50:36.068 DEBUG:boto:Method: GET 2023-10-17T18:50:36.069 DEBUG:boto:Path: /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.069 DEBUG:boto:Data: 2023-10-17T18:50:36.070 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.070 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.070 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.071 DEBUG:boto:Params: {} 2023-10-17T18:50:36.071 DEBUG:boto:Token: None 2023-10-17T18:50:36.072 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f?versionId=null 2023-10-17T18:50:36.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aVks5PSpJO1+RBmzz2Ugkrotets= 2023-10-17T18:50:36.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aVks5PSpJO1+RBmzz2Ugkrotets=', 'Content-Length': '0'} 2023-10-17T18:50:36.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4cba558f8bb6de-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.076 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.076 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.077 DEBUG:boto:path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.077 DEBUG:boto:auth_path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.077 DEBUG:boto:Method: GET 2023-10-17T18:50:36.078 DEBUG:boto:Path: /hbmuog-56/?prefix=f 2023-10-17T18:50:36.078 DEBUG:boto:Data: 2023-10-17T18:50:36.079 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.080 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.080 DEBUG:boto:Params: {} 2023-10-17T18:50:36.080 DEBUG:boto:Token: None 2023-10-17T18:50:36.081 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cede5cff9603a39-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.086 DEBUG:boto:b'hbmuog-56f1000falsef2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.086 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.087 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.087 DEBUG:boto:path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.088 DEBUG:boto:auth_path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.088 DEBUG:boto:Method: GET 2023-10-17T18:50:36.088 DEBUG:boto:Path: /hbmuog-56/?prefix=f 2023-10-17T18:50:36.089 DEBUG:boto:Data: 2023-10-17T18:50:36.089 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.090 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.091 DEBUG:boto:Params: {} 2023-10-17T18:50:36.091 DEBUG:boto:Token: None 2023-10-17T18:50:36.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f51ffe4a3a69962-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.095 DEBUG:boto:b'hbmuog-56f1000falsef2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.096 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.096 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.097 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.097 DEBUG:boto:Path: /hbmuog-56/f 2023-10-17T18:50:36.097 DEBUG:boto:Data: 2023-10-17T18:50:36.098 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.099 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.099 DEBUG:boto:Params: {} 2023-10-17T18:50:36.099 DEBUG:boto:Token: None 2023-10-17T18:50:36.100 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f 2023-10-17T18:50:36.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyxSj3+wDddvG+IcwPfnhAHQfcI= 2023-10-17T18:50:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyxSj3+wDddvG+IcwPfnhAHQfcI=', 'Content-Length': '0'} 2023-10-17T18:50:36.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027912f3dbb24cc28-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.103 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.104 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.104 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.105 DEBUG:boto:Path: /hbmuog-56/f 2023-10-17T18:50:36.105 DEBUG:boto:Data: 2023-10-17T18:50:36.106 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.106 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.107 DEBUG:boto:Params: {} 2023-10-17T18:50:36.107 DEBUG:boto:Token: None 2023-10-17T18:50:36.108 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f 2023-10-17T18:50:36.108 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyxSj3+wDddvG+IcwPfnhAHQfcI= 2023-10-17T18:50:36.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyxSj3+wDddvG+IcwPfnhAHQfcI=', 'Content-Length': '0'} 2023-10-17T18:50:36.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e51c291fa3551ba-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:50:36.112 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.112 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.112 DEBUG:boto:Method: GET 2023-10-17T18:50:36.113 DEBUG:boto:Path: /hbmuog-56/f 2023-10-17T18:50:36.113 DEBUG:boto:Data: 2023-10-17T18:50:36.114 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.114 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.115 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.115 DEBUG:boto:Params: {} 2023-10-17T18:50:36.116 DEBUG:boto:Token: None 2023-10-17T18:50:36.116 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f 2023-10-17T18:50:36.117 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R4GJLWfM5OKOjlOjx4+Z7FUpffs= 2023-10-17T18:50:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R4GJLWfM5OKOjlOjx4+Z7FUpffs=', 'Content-Length': '0'} 2023-10-17T18:50:36.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab21cd33c3da134-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.120 DEBUG:boto:path=/hbmuog-56/f 2023-10-17T18:50:36.120 DEBUG:boto:auth_path=/hbmuog-56/f 2023-10-17T18:50:36.121 DEBUG:boto:Method: GET 2023-10-17T18:50:36.121 DEBUG:boto:Path: /hbmuog-56/f 2023-10-17T18:50:36.121 DEBUG:boto:Data: 2023-10-17T18:50:36.122 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.122 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.123 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.123 DEBUG:boto:Params: {} 2023-10-17T18:50:36.124 DEBUG:boto:Token: None 2023-10-17T18:50:36.124 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/f 2023-10-17T18:50:36.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R4GJLWfM5OKOjlOjx4+Z7FUpffs= 2023-10-17T18:50:36.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R4GJLWfM5OKOjlOjx4+Z7FUpffs=', 'Content-Length': '0'} 2023-10-17T18:50:36.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076503129bc5875be-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.170 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.171 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.171 DEBUG:boto:path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.172 DEBUG:boto:auth_path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.172 DEBUG:boto:Method: GET 2023-10-17T18:50:36.173 DEBUG:boto:Path: /hbmuog-56/?prefix=f 2023-10-17T18:50:36.173 DEBUG:boto:Data: 2023-10-17T18:50:36.174 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.174 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.175 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.175 DEBUG:boto:Params: {} 2023-10-17T18:50:36.175 DEBUG:boto:Token: None 2023-10-17T18:50:36.176 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.176 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a55cd1764c4f79-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.180 DEBUG:boto:b'hbmuog-56f1000falsef2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.181 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.182 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.182 DEBUG:boto:path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.183 DEBUG:boto:auth_path=/hbmuog-56/?prefix=f 2023-10-17T18:50:36.183 DEBUG:boto:Method: GET 2023-10-17T18:50:36.183 DEBUG:boto:Path: /hbmuog-56/?prefix=f 2023-10-17T18:50:36.184 DEBUG:boto:Data: 2023-10-17T18:50:36.184 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.185 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.186 DEBUG:boto:Params: {} 2023-10-17T18:50:36.186 DEBUG:boto:Token: None 2023-10-17T18:50:36.187 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca06a3594b852270-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.190 DEBUG:boto:b'hbmuog-56f1000falsef2023-10-17T18:48:14.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:50:36.191 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.192 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.192 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.193 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.193 DEBUG:boto:Method: GET 2023-10-17T18:50:36.194 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.194 DEBUG:boto:Data: 2023-10-17T18:50:36.194 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.195 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.195 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.196 DEBUG:boto:Params: {} 2023-10-17T18:50:36.196 DEBUG:boto:Token: None 2023-10-17T18:50:36.197 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.197 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM= 2023-10-17T18:50:36.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM=', 'Content-Length': '0'} 2023-10-17T18:50:36.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d810e200188ac21d-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.242 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.243 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.243 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.244 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.244 DEBUG:boto:Method: GET 2023-10-17T18:50:36.245 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.245 DEBUG:boto:Data: 2023-10-17T18:50:36.246 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.246 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.247 DEBUG:boto:Params: {} 2023-10-17T18:50:36.247 DEBUG:boto:Token: None 2023-10-17T18:50:36.248 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM= 2023-10-17T18:50:36.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM=', 'Content-Length': '0'} 2023-10-17T18:50:36.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c67c90bd1f416f-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.252 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.252 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.253 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.253 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.253 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.254 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.254 DEBUG:boto:Data: 2023-10-17T18:50:36.255 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.255 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.256 DEBUG:boto:Params: {} 2023-10-17T18:50:36.256 DEBUG:boto:Token: None 2023-10-17T18:50:36.257 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jPWf2euiYyDu+GN9P8A4VTpuc54= 2023-10-17T18:50:36.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jPWf2euiYyDu+GN9P8A4VTpuc54=', 'Content-Length': '0'} 2023-10-17T18:50:36.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a7fd5fe53a2338-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.260 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.261 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.261 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.261 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.262 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.262 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.263 DEBUG:boto:Data: 2023-10-17T18:50:36.263 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.264 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.264 DEBUG:boto:Params: {} 2023-10-17T18:50:36.265 DEBUG:boto:Token: None 2023-10-17T18:50:36.265 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.266 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jPWf2euiYyDu+GN9P8A4VTpuc54= 2023-10-17T18:50:36.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jPWf2euiYyDu+GN9P8A4VTpuc54=', 'Content-Length': '0'} 2023-10-17T18:50:36.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f0f933372d54f7d-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:50:36.269 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.270 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.270 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.271 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.271 DEBUG:boto:Method: GET 2023-10-17T18:50:36.271 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.272 DEBUG:boto:Data: 2023-10-17T18:50:36.272 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.273 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.273 DEBUG:boto:Params: {} 2023-10-17T18:50:36.274 DEBUG:boto:Token: None 2023-10-17T18:50:36.274 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM= 2023-10-17T18:50:36.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM=', 'Content-Length': '0'} 2023-10-17T18:50:36.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dddf4c1d669bf04-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.278 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.279 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.279 DEBUG:boto:path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.280 DEBUG:boto:auth_path=/hbmuog-56/g?versionId=null 2023-10-17T18:50:36.280 DEBUG:boto:Method: GET 2023-10-17T18:50:36.280 DEBUG:boto:Path: /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.281 DEBUG:boto:Data: 2023-10-17T18:50:36.281 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.282 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.282 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.283 DEBUG:boto:Params: {} 2023-10-17T18:50:36.283 DEBUG:boto:Token: None 2023-10-17T18:50:36.283 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g?versionId=null 2023-10-17T18:50:36.284 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM= 2023-10-17T18:50:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pgL57/pwxSzuYcfpv9f/pROXEJM=', 'Content-Length': '0'} 2023-10-17T18:50:36.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002588fd57f341845-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.287 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.288 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.288 DEBUG:boto:path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.289 DEBUG:boto:auth_path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.289 DEBUG:boto:Method: GET 2023-10-17T18:50:36.289 DEBUG:boto:Path: /hbmuog-56/?prefix=g 2023-10-17T18:50:36.290 DEBUG:boto:Data: 2023-10-17T18:50:36.290 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.291 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.292 DEBUG:boto:Params: {} 2023-10-17T18:50:36.292 DEBUG:boto:Token: None 2023-10-17T18:50:36.293 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.293 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f990921d1fd9db-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.297 DEBUG:boto:b'hbmuog-56g1000falseg2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.298 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.298 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.299 DEBUG:boto:path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.299 DEBUG:boto:auth_path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.300 DEBUG:boto:Method: GET 2023-10-17T18:50:36.300 DEBUG:boto:Path: /hbmuog-56/?prefix=g 2023-10-17T18:50:36.301 DEBUG:boto:Data: 2023-10-17T18:50:36.301 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.301 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.302 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.302 DEBUG:boto:Params: {} 2023-10-17T18:50:36.303 DEBUG:boto:Token: None 2023-10-17T18:50:36.303 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac4a373bc7eba5c-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.354 DEBUG:boto:b'hbmuog-56g1000falseg2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.358 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.358 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.358 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.359 DEBUG:boto:Path: /hbmuog-56/g 2023-10-17T18:50:36.359 DEBUG:boto:Data: 2023-10-17T18:50:36.360 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.360 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.360 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.361 DEBUG:boto:Params: {} 2023-10-17T18:50:36.361 DEBUG:boto:Token: None 2023-10-17T18:50:36.362 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g 2023-10-17T18:50:36.362 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nEkQznQDMEJ/yWQdVBrhFYd7i5U= 2023-10-17T18:50:36.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nEkQznQDMEJ/yWQdVBrhFYd7i5U=', 'Content-Length': '0'} 2023-10-17T18:50:36.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000395fc7d718f3874b-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.365 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.366 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.366 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.367 DEBUG:boto:Path: /hbmuog-56/g 2023-10-17T18:50:36.367 DEBUG:boto:Data: 2023-10-17T18:50:36.367 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.368 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.368 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.369 DEBUG:boto:Params: {} 2023-10-17T18:50:36.369 DEBUG:boto:Token: None 2023-10-17T18:50:36.370 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g 2023-10-17T18:50:36.370 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nEkQznQDMEJ/yWQdVBrhFYd7i5U= 2023-10-17T18:50:36.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nEkQznQDMEJ/yWQdVBrhFYd7i5U=', 'Content-Length': '0'} 2023-10-17T18:50:36.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace1515cc5c698c1-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:50:36.374 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.374 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.374 DEBUG:boto:Method: GET 2023-10-17T18:50:36.375 DEBUG:boto:Path: /hbmuog-56/g 2023-10-17T18:50:36.375 DEBUG:boto:Data: 2023-10-17T18:50:36.376 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.376 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.377 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.377 DEBUG:boto:Params: {} 2023-10-17T18:50:36.378 DEBUG:boto:Token: None 2023-10-17T18:50:36.378 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g 2023-10-17T18:50:36.379 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7EEGoWBIyWvGyQo5FVvC8QviTss= 2023-10-17T18:50:36.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7EEGoWBIyWvGyQo5FVvC8QviTss=', 'Content-Length': '0'} 2023-10-17T18:50:36.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008110ef94f9808e5a-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.382 DEBUG:boto:path=/hbmuog-56/g 2023-10-17T18:50:36.382 DEBUG:boto:auth_path=/hbmuog-56/g 2023-10-17T18:50:36.383 DEBUG:boto:Method: GET 2023-10-17T18:50:36.383 DEBUG:boto:Path: /hbmuog-56/g 2023-10-17T18:50:36.383 DEBUG:boto:Data: 2023-10-17T18:50:36.384 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.385 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.385 DEBUG:boto:Params: {} 2023-10-17T18:50:36.386 DEBUG:boto:Token: None 2023-10-17T18:50:36.386 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/g 2023-10-17T18:50:36.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7EEGoWBIyWvGyQo5FVvC8QviTss= 2023-10-17T18:50:36.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7EEGoWBIyWvGyQo5FVvC8QviTss=', 'Content-Length': '0'} 2023-10-17T18:50:36.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dead721b1f9aab3-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.390 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.390 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.391 DEBUG:boto:path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.391 DEBUG:boto:auth_path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.392 DEBUG:boto:Method: GET 2023-10-17T18:50:36.392 DEBUG:boto:Path: /hbmuog-56/?prefix=g 2023-10-17T18:50:36.393 DEBUG:boto:Data: 2023-10-17T18:50:36.393 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.393 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.394 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.394 DEBUG:boto:Params: {} 2023-10-17T18:50:36.395 DEBUG:boto:Token: None 2023-10-17T18:50:36.395 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c2a4dd0ca32824-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.400 DEBUG:boto:b'hbmuog-56g1000falseg2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.400 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.401 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.401 DEBUG:boto:path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.402 DEBUG:boto:auth_path=/hbmuog-56/?prefix=g 2023-10-17T18:50:36.402 DEBUG:boto:Method: GET 2023-10-17T18:50:36.403 DEBUG:boto:Path: /hbmuog-56/?prefix=g 2023-10-17T18:50:36.403 DEBUG:boto:Data: 2023-10-17T18:50:36.403 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.404 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.404 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.405 DEBUG:boto:Params: {} 2023-10-17T18:50:36.405 DEBUG:boto:Token: None 2023-10-17T18:50:36.406 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.406 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10ddcaac5f113d3-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.409 DEBUG:boto:b'hbmuog-56g1000falseg2023-10-17T18:48:14.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:50:36.410 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.411 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.411 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.411 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.412 DEBUG:boto:Method: GET 2023-10-17T18:50:36.412 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.413 DEBUG:boto:Data: 2023-10-17T18:50:36.413 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.414 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.415 DEBUG:boto:Params: {} 2023-10-17T18:50:36.415 DEBUG:boto:Token: None 2023-10-17T18:50:36.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0= 2023-10-17T18:50:36.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0=', 'Content-Length': '0'} 2023-10-17T18:50:36.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e74bc490a2c299-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.419 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.420 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.420 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.420 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.421 DEBUG:boto:Method: GET 2023-10-17T18:50:36.421 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.422 DEBUG:boto:Data: 2023-10-17T18:50:36.422 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.423 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.423 DEBUG:boto:Params: {} 2023-10-17T18:50:36.424 DEBUG:boto:Token: None 2023-10-17T18:50:36.424 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0= 2023-10-17T18:50:36.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0=', 'Content-Length': '0'} 2023-10-17T18:50:36.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e1aa243b4bdcf50-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.428 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.428 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.429 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.429 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.430 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.430 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.430 DEBUG:boto:Data: 2023-10-17T18:50:36.431 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.431 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.432 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.432 DEBUG:boto:Params: {} 2023-10-17T18:50:36.432 DEBUG:boto:Token: None 2023-10-17T18:50:36.433 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.433 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hkkKqaPDTKdDUNkOFovSY9WDELk= 2023-10-17T18:50:36.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hkkKqaPDTKdDUNkOFovSY9WDELk=', 'Content-Length': '0'} 2023-10-17T18:50:36.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973c120df49cc8b7-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.436 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.437 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.437 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.437 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.438 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.438 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.439 DEBUG:boto:Data: 2023-10-17T18:50:36.439 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.440 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.440 DEBUG:boto:Params: {} 2023-10-17T18:50:36.441 DEBUG:boto:Token: None 2023-10-17T18:50:36.441 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hkkKqaPDTKdDUNkOFovSY9WDELk= 2023-10-17T18:50:36.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hkkKqaPDTKdDUNkOFovSY9WDELk=', 'Content-Length': '0'} 2023-10-17T18:50:36.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000329694d2adaf7388-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:50:36.445 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.445 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.446 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.446 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.447 DEBUG:boto:Method: GET 2023-10-17T18:50:36.447 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.448 DEBUG:boto:Data: 2023-10-17T18:50:36.448 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.449 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.449 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.449 DEBUG:boto:Params: {} 2023-10-17T18:50:36.450 DEBUG:boto:Token: None 2023-10-17T18:50:36.450 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.451 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0= 2023-10-17T18:50:36.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0=', 'Content-Length': '0'} 2023-10-17T18:50:36.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008898954c9083a544-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.494 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.495 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.495 DEBUG:boto:path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.496 DEBUG:boto:auth_path=/hbmuog-56/h?versionId=null 2023-10-17T18:50:36.496 DEBUG:boto:Method: GET 2023-10-17T18:50:36.497 DEBUG:boto:Path: /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.497 DEBUG:boto:Data: 2023-10-17T18:50:36.497 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.498 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.499 DEBUG:boto:Params: {} 2023-10-17T18:50:36.499 DEBUG:boto:Token: None 2023-10-17T18:50:36.500 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h?versionId=null 2023-10-17T18:50:36.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0= 2023-10-17T18:50:36.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uF18HjBKpTE8xeyj7ChfLh9x/q0=', 'Content-Length': '0'} 2023-10-17T18:50:36.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a7d51822909487-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.503 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.504 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.504 DEBUG:boto:path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.505 DEBUG:boto:auth_path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.505 DEBUG:boto:Method: GET 2023-10-17T18:50:36.506 DEBUG:boto:Path: /hbmuog-56/?prefix=h 2023-10-17T18:50:36.506 DEBUG:boto:Data: 2023-10-17T18:50:36.506 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.507 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.508 DEBUG:boto:Params: {} 2023-10-17T18:50:36.508 DEBUG:boto:Token: None 2023-10-17T18:50:36.509 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007854cfe92eea59ab-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.513 DEBUG:boto:b'hbmuog-56h1000falseh2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.514 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.514 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.515 DEBUG:boto:path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.515 DEBUG:boto:auth_path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.515 DEBUG:boto:Method: GET 2023-10-17T18:50:36.516 DEBUG:boto:Path: /hbmuog-56/?prefix=h 2023-10-17T18:50:36.516 DEBUG:boto:Data: 2023-10-17T18:50:36.517 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.517 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.518 DEBUG:boto:Params: {} 2023-10-17T18:50:36.518 DEBUG:boto:Token: None 2023-10-17T18:50:36.519 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.519 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a7283623e69bf5-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.562 DEBUG:boto:b'hbmuog-56h1000falseh2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.563 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.564 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.564 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.565 DEBUG:boto:Path: /hbmuog-56/h 2023-10-17T18:50:36.565 DEBUG:boto:Data: 2023-10-17T18:50:36.565 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.566 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.566 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.567 DEBUG:boto:Params: {} 2023-10-17T18:50:36.567 DEBUG:boto:Token: None 2023-10-17T18:50:36.567 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h 2023-10-17T18:50:36.568 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:azJveTPWZU46ZgO1cgx1zToBoW0= 2023-10-17T18:50:36.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:azJveTPWZU46ZgO1cgx1zToBoW0=', 'Content-Length': '0'} 2023-10-17T18:50:36.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ddf61011149fe9-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.571 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.572 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.572 DEBUG:boto:Method: HEAD 2023-10-17T18:50:36.572 DEBUG:boto:Path: /hbmuog-56/h 2023-10-17T18:50:36.573 DEBUG:boto:Data: 2023-10-17T18:50:36.573 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.574 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.574 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.574 DEBUG:boto:Params: {} 2023-10-17T18:50:36.575 DEBUG:boto:Token: None 2023-10-17T18:50:36.575 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h 2023-10-17T18:50:36.576 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:azJveTPWZU46ZgO1cgx1zToBoW0= 2023-10-17T18:50:36.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:azJveTPWZU46ZgO1cgx1zToBoW0=', 'Content-Length': '0'} 2023-10-17T18:50:36.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3de727f32e6cafc-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:50:36.579 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.580 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.580 DEBUG:boto:Method: GET 2023-10-17T18:50:36.580 DEBUG:boto:Path: /hbmuog-56/h 2023-10-17T18:50:36.581 DEBUG:boto:Data: 2023-10-17T18:50:36.581 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.582 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.583 DEBUG:boto:Params: {} 2023-10-17T18:50:36.583 DEBUG:boto:Token: None 2023-10-17T18:50:36.583 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h 2023-10-17T18:50:36.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5CyYDJM1ZG96zxwMH/2qpNDizh4= 2023-10-17T18:50:36.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5CyYDJM1ZG96zxwMH/2qpNDizh4=', 'Content-Length': '0'} 2023-10-17T18:50:36.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-56:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4b5ac8433ce725a-00652ed77c-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.587 DEBUG:boto:path=/hbmuog-56/h 2023-10-17T18:50:36.588 DEBUG:boto:auth_path=/hbmuog-56/h 2023-10-17T18:50:36.588 DEBUG:boto:Method: GET 2023-10-17T18:50:36.588 DEBUG:boto:Path: /hbmuog-56/h 2023-10-17T18:50:36.589 DEBUG:boto:Data: 2023-10-17T18:50:36.589 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.590 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.591 DEBUG:boto:Params: {} 2023-10-17T18:50:36.591 DEBUG:boto:Token: None 2023-10-17T18:50:36.591 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/h 2023-10-17T18:50:36.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5CyYDJM1ZG96zxwMH/2qpNDizh4= 2023-10-17T18:50:36.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5CyYDJM1ZG96zxwMH/2qpNDizh4=', 'Content-Length': '0'} 2023-10-17T18:50:36.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:48:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c998242d20955f2-00652ed77c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.595 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.596 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.596 DEBUG:boto:path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.597 DEBUG:boto:auth_path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.597 DEBUG:boto:Method: GET 2023-10-17T18:50:36.597 DEBUG:boto:Path: /hbmuog-56/?prefix=h 2023-10-17T18:50:36.598 DEBUG:boto:Data: 2023-10-17T18:50:36.598 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:36.599 DEBUG:boto:Port: 8001 2023-10-17T18:50:36.600 DEBUG:boto:Params: {} 2023-10-17T18:50:36.600 DEBUG:boto:Token: None 2023-10-17T18:50:36.601 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208015e2f039df9a-00652ed77c-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.646 DEBUG:boto:b'hbmuog-56h1000falseh2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.647 DEBUG:boto:path=/hbmuog-56/ 2023-10-17T18:50:36.648 DEBUG:boto:auth_path=/hbmuog-56/ 2023-10-17T18:50:36.648 DEBUG:boto:path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.648 DEBUG:boto:auth_path=/hbmuog-56/?prefix=h 2023-10-17T18:50:36.649 DEBUG:boto:Method: GET 2023-10-17T18:50:36.649 DEBUG:boto:Path: /hbmuog-56/?prefix=h 2023-10-17T18:50:36.650 DEBUG:boto:Data: 2023-10-17T18:50:36.650 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.651 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.651 DEBUG:boto:Params: {} 2023-10-17T18:50:36.652 DEBUG:boto:Token: None 2023-10-17T18:50:36.652 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-56/ 2023-10-17T18:50:36.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4= 2023-10-17T18:50:36.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L+IVo1BluMYWmfQ7JNLO1s68Hp4=', 'Content-Length': '0'} 2023-10-17T18:50:36.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4920372c9fc532f-00652ed77c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.656 DEBUG:boto:b'hbmuog-56h1000falseh2023-10-17T18:48:14.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:50:36.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-56 zones={a2, a1} 2023-10-17T18:50:36.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-10-17T18:50:36.659 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:36.659 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:36.660 DEBUG:boto:Method: PUT 2023-10-17T18:50:36.660 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:36.661 DEBUG:boto:Data: 2023-10-17T18:50:36.661 DEBUG:boto:Headers: {} 2023-10-17T18:50:36.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:36.662 DEBUG:boto:Port: 8000 2023-10-17T18:50:36.662 DEBUG:boto:Params: {} 2023-10-17T18:50:36.663 DEBUG:boto:Token: None 2023-10-17T18:50:36.663 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:50:36 GMT /hbmuog-57/ 2023-10-17T18:50:36.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dRwvX4eaFcqfwwRi7GknagRT8Bs= 2023-10-17T18:50:36.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dRwvX4eaFcqfwwRi7GknagRT8Bs=', 'Content-Length': '0'} 2023-10-17T18:50:36.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000147f1ed779878777-00652ed77c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:36.683 DEBUG:rgw_multi.tests:created bucket=hbmuog-57 2023-10-17T18:50:36.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:50:36.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:50:36.685 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:50:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:50:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:50:36.784 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:50:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:50:36.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.798 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568636.681147_682.1", 2023-10-17T18:50:36.798 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:50:36.681147Z" 2023-10-17T18:50:36.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.799 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:50:36.799 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:50:36.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.800 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:50:36.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:50:36.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.801 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:50:36.801 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:50:36.801 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:50:36.803 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568636.681147_682.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:50:36.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:50:36.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:50:36.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:50:36.805 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:50:36.856 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:50:36.856 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:50:36.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:36.884+0000 7f7bf9b0a780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:50:36.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:36.885+0000 7f7bf9b0a780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:50:36.899 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:50:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:50:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:50:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:50:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:50:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:50:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:50:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:50:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:50:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:50:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:50:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568636.681147_682.1", 2023-10-17T18:50:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:50:36.903 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:50:36.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:50:36.903 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:50:36.681147Z", 2023-10-17T18:50:36.903 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:50:36.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:50:36.905 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:50:36.905 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:50:36.905 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:50:36.905 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:50:36.906 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:50:36.906 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:50:36.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.907 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:50:36.907 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:50:36.907 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:50:36.907 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:50:36.907 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:50:36.908 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:50:36.908 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:50:36.908 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:50:36.908 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:50:36.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:50:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:50:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:50:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:50:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:50:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:50:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:50:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:50:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:50:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:50:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:50:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:50:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:50:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:50:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:50:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:50:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:50:36Z" 2023-10-17T18:50:36.913 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:50:36.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568636.681147_682.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:50:36.681147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:50:36Z" } 2023-10-17T18:50:36.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568636.681147_682.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:50:36.681147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:50:36.915 DEBUG:rgw_multi.tests:log_status={0: '1_1697568636.681147_682.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:50:36.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568636.681147_682.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:50:36.915 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:50:36.916 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T18:50:36.916 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:50:36.917 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:50:36.915+0000 7f5f26c8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 122928) UID: 0 2023-10-17T18:50:36.917 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:50:36.916+0000 7f5f2aef37c0 -1 shutting down 2023-10-17T18:50:37.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T18:50:43.019 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T18:50:43.020 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.020 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.021 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.021 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.022 DEBUG:boto:Data: 2023-10-17T18:50:43.022 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.023 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.023 DEBUG:boto:Params: {} 2023-10-17T18:50:43.024 DEBUG:boto:Token: None 2023-10-17T18:50:43.024 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c86d4ff5df90153-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.029 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:50:43.029 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:50:43.029 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.030 DEBUG:boto:Path: /hbmuog-57/a 2023-10-17T18:50:43.030 DEBUG:boto:Data: 2023-10-17T18:50:43.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.031 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.031 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.032 DEBUG:boto:Params: {} 2023-10-17T18:50:43.032 DEBUG:boto:Token: None 2023-10-17T18:50:43.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/a 2023-10-17T18:50:43.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sjFOOGHBCGL9l/p6FC037zbEPvE= 2023-10-17T18:50:43.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sjFOOGHBCGL9l/p6FC037zbEPvE='} 2023-10-17T18:50:43.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cda507cae4baaca-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.083 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.084 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.084 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.084 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.085 DEBUG:boto:Data: 2023-10-17T18:50:43.085 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.086 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.086 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.086 DEBUG:boto:Params: {} 2023-10-17T18:50:43.087 DEBUG:boto:Token: None 2023-10-17T18:50:43.087 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.088 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000850ffacb23f6b3b5-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.092 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:50:43.092 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:50:43.092 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.093 DEBUG:boto:Path: /hbmuog-57/b 2023-10-17T18:50:43.093 DEBUG:boto:Data: 2023-10-17T18:50:43.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.094 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.094 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.095 DEBUG:boto:Params: {} 2023-10-17T18:50:43.095 DEBUG:boto:Token: None 2023-10-17T18:50:43.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/b 2023-10-17T18:50:43.096 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7rnuKBjrVKX0zAQXKhoRqY648cc= 2023-10-17T18:50:43.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7rnuKBjrVKX0zAQXKhoRqY648cc='} 2023-10-17T18:50:43.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edde091360d76043-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.104 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.104 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.104 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.105 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.105 DEBUG:boto:Data: 2023-10-17T18:50:43.106 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.106 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.107 DEBUG:boto:Params: {} 2023-10-17T18:50:43.107 DEBUG:boto:Token: None 2023-10-17T18:50:43.108 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.108 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006681242c36d8aa1a-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.112 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:50:43.112 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:50:43.112 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.113 DEBUG:boto:Path: /hbmuog-57/c 2023-10-17T18:50:43.113 DEBUG:boto:Data: 2023-10-17T18:50:43.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.114 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.115 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.115 DEBUG:boto:Params: {} 2023-10-17T18:50:43.115 DEBUG:boto:Token: None 2023-10-17T18:50:43.116 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/c 2023-10-17T18:50:43.116 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qq3V7QMUInSZnYHuZ/aWehq/7d0= 2023-10-17T18:50:43.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qq3V7QMUInSZnYHuZ/aWehq/7d0='} 2023-10-17T18:50:43.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000accaa3cd2467feaf-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.159 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.159 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.160 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.160 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.161 DEBUG:boto:Data: 2023-10-17T18:50:43.161 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.162 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.162 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.162 DEBUG:boto:Params: {} 2023-10-17T18:50:43.163 DEBUG:boto:Token: None 2023-10-17T18:50:43.163 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000076e69b3a601b1a8-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.167 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:50:43.168 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:50:43.168 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.169 DEBUG:boto:Path: /hbmuog-57/d 2023-10-17T18:50:43.169 DEBUG:boto:Data: 2023-10-17T18:50:43.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.170 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.171 DEBUG:boto:Params: {} 2023-10-17T18:50:43.171 DEBUG:boto:Token: None 2023-10-17T18:50:43.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/d 2023-10-17T18:50:43.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1bOb4KvhIs5N8EruCQShpdTEe4A= 2023-10-17T18:50:43.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1bOb4KvhIs5N8EruCQShpdTEe4A='} 2023-10-17T18:50:43.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d21c0d0096c66ba6-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:50:43.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:50:43.180 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:50:43.237 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:50:43.237 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:50:43.282 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:50:43.282 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-57 2023-10-17T18:50:43.286 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 4 2023-10-17T18:50:43.322 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.323 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.323 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.323 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.324 DEBUG:boto:Data: 2023-10-17T18:50:43.324 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.325 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.326 DEBUG:boto:Params: {} 2023-10-17T18:50:43.326 DEBUG:boto:Token: None 2023-10-17T18:50:43.327 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016482f1fe0fb57c1-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.330 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:50:43.331 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:50:43.331 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.332 DEBUG:boto:Path: /hbmuog-57/e 2023-10-17T18:50:43.332 DEBUG:boto:Data: 2023-10-17T18:50:43.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.333 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.333 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.334 DEBUG:boto:Params: {} 2023-10-17T18:50:43.334 DEBUG:boto:Token: None 2023-10-17T18:50:43.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/e 2023-10-17T18:50:43.335 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NpY890YSWt+vU+0URI6eOGtU1FA= 2023-10-17T18:50:43.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NpY890YSWt+vU+0URI6eOGtU1FA='} 2023-10-17T18:50:43.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7978657fb8e9688-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.341 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.342 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.342 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.343 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.343 DEBUG:boto:Data: 2023-10-17T18:50:43.343 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.344 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.345 DEBUG:boto:Params: {} 2023-10-17T18:50:43.345 DEBUG:boto:Token: None 2023-10-17T18:50:43.345 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058760467765f9dc0-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.349 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:50:43.349 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:50:43.350 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.350 DEBUG:boto:Path: /hbmuog-57/f 2023-10-17T18:50:43.351 DEBUG:boto:Data: 2023-10-17T18:50:43.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.352 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.353 DEBUG:boto:Params: {} 2023-10-17T18:50:43.353 DEBUG:boto:Token: None 2023-10-17T18:50:43.354 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/f 2023-10-17T18:50:43.354 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EC8l+7XUXNcgQbpbRCnNDOT2jbI= 2023-10-17T18:50:43.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EC8l+7XUXNcgQbpbRCnNDOT2jbI='} 2023-10-17T18:50:43.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef8fefba674b8346-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.400 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.400 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.400 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.401 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.401 DEBUG:boto:Data: 2023-10-17T18:50:43.402 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.403 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.403 DEBUG:boto:Params: {} 2023-10-17T18:50:43.403 DEBUG:boto:Token: None 2023-10-17T18:50:43.404 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003318967eda1f356b-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.407 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:50:43.408 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:50:43.408 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.409 DEBUG:boto:Path: /hbmuog-57/g 2023-10-17T18:50:43.409 DEBUG:boto:Data: 2023-10-17T18:50:43.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.410 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.411 DEBUG:boto:Params: {} 2023-10-17T18:50:43.411 DEBUG:boto:Token: None 2023-10-17T18:50:43.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/g 2023-10-17T18:50:43.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jn1SMfi/NihkSFEbcHhXXqHp/I8= 2023-10-17T18:50:43.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jn1SMfi/NihkSFEbcHhXXqHp/I8='} 2023-10-17T18:50:43.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000824c75a94a51cf0b-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.418 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:50:43.418 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:50:43.419 DEBUG:boto:Method: HEAD 2023-10-17T18:50:43.419 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:50:43.420 DEBUG:boto:Data: 2023-10-17T18:50:43.420 DEBUG:boto:Headers: {} 2023-10-17T18:50:43.420 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.421 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.421 DEBUG:boto:Params: {} 2023-10-17T18:50:43.422 DEBUG:boto:Token: None 2023-10-17T18:50:43.422 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/ 2023-10-17T18:50:43.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8= 2023-10-17T18:50:43.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SPT4ZruXw8UDjhbOENohHUyK7i8=', 'Content-Length': '0'} 2023-10-17T18:50:43.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db4e76f48b90f19d-00652ed783-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.426 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:50:43.426 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:50:43.427 DEBUG:boto:Method: PUT 2023-10-17T18:50:43.427 DEBUG:boto:Path: /hbmuog-57/h 2023-10-17T18:50:43.427 DEBUG:boto:Data: 2023-10-17T18:50:43.428 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:50:43.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:50:43.429 DEBUG:boto:Port: 8000 2023-10-17T18:50:43.429 DEBUG:boto:Params: {} 2023-10-17T18:50:43.429 DEBUG:boto:Token: None 2023-10-17T18:50:43.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:50:43 GMT /hbmuog-57/h 2023-10-17T18:50:43.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kt2vTPq+fXu8rCC7F2yYr4y8AcM= 2023-10-17T18:50:43.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:50:43 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kt2vTPq+fXu8rCC7F2yYr4y8AcM='} 2023-10-17T18:50:43.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T18:50:43.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a820ba0ad8abe02d-00652ed783-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:50:43.479 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T18:50:43.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T18:50:43.483 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T18:50:43.483 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:43.505 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:50:43.505 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:50:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:50:43.507 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T18:50:44.508 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:50:44.533 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:50:44.533 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:50:44.544 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 17833 0 --:--:-- --:--:-- --:--:-- 17833 2023-10-17T18:50:44.545 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:50:44.546 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-57 2023-10-17T18:50:44.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:50:44.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:50:44.547 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:50:44.644 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:50:44.644 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:50:44.682 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:44.680+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:50:49.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:49.682+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:50:54.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:54.683+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:50:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:50:59.683+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:04.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:04.683+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:09.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:09.684+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:14.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:14.684+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:19.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:19.685+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:24.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:24.686+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:29.687+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:34.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:34.688+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:39.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:39.687+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:44.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:44.687+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:49.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:49.688+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:54.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:54.689+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:51:59.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:51:59.690+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:04.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:04.691+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:09.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:09.691+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:14.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:14.691+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:19.692+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:24.692+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:29.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:29.693+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:34.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:34.693+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:39.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:39.695+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:44.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:44.695+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:49.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:49.695+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:54.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:54.696+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:52:59.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:52:59.696+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:53:04.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:04.697+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:53:09.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:09.698+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:53:14.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:14.699+0000 7f57bb77e780 1 waiting to reach incremental sync.. 2023-10-17T18:53:19.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:19.700+0000 7f57bb77e780 1 bucket sync caught up with source: 2023-10-17T18:53:19.701 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000003.39.6, 00000000002.32.6, 00000000002.30.6] 2023-10-17T18:53:19.702 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000003.39.6, 00000000002.32.6, 00000000002.30.6] 2023-10-17T18:53:19.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:19.700+0000 7f57bb77e780 0 bucket checkpoint complete 2023-10-17T18:53:19.719 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-57 2023-10-17T18:53:19.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:19.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:19.721 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:53:19.785 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:19.786 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:19.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:19.824+0000 7f871adb0780 1 waiting for incremental sync to catch up: 2023-10-17T18:53:19.825 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000001.231.6] 2023-10-17T18:53:19.825 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000002.236.6] 2023-10-17T18:53:24.826 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:24.825+0000 7f871adb0780 1 waiting for incremental sync to catch up: 2023-10-17T18:53:24.826 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000001.231.6] 2023-10-17T18:53:24.826 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000002.236.6] 2023-10-17T18:53:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:29.826+0000 7f871adb0780 1 waiting for incremental sync to catch up: 2023-10-17T18:53:29.830 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000001.231.6] 2023-10-17T18:53:29.830 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000002.236.6] 2023-10-17T18:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:34.827+0000 7f871adb0780 1 bucket sync caught up with source: 2023-10-17T18:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000002.236.6] 2023-10-17T18:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000002.60.6, , , 00000000001.121.6, 00000000001.154.6, , , 00000000002.198.6, , 00000000002.236.6] 2023-10-17T18:53:34.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:34.827+0000 7f871adb0780 0 bucket checkpoint complete 2023-10-17T18:53:34.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-57 zones={a2, a1} 2023-10-17T18:53:34.847 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.848 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.848 DEBUG:boto:Method: HEAD 2023-10-17T18:53:34.849 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:53:34.849 DEBUG:boto:Data: 2023-10-17T18:53:34.849 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.850 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.851 DEBUG:boto:Params: {} 2023-10-17T18:53:34.852 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:53:34.852 DEBUG:boto:Token: None 2023-10-17T18:53:34.853 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/ 2023-10-17T18:53:34.854 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WdSkGWx2n1C3Lnk+sz9YxmjWfpI= 2023-10-17T18:53:34.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WdSkGWx2n1C3Lnk+sz9YxmjWfpI=', 'Content-Length': '0'} 2023-10-17T18:53:34.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf496ef4df044338-00652ed82e-1a76-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.861 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.861 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.862 DEBUG:boto:Method: HEAD 2023-10-17T18:53:34.862 DEBUG:boto:Path: /hbmuog-57/ 2023-10-17T18:53:34.863 DEBUG:boto:Data: 2023-10-17T18:53:34.863 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.864 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:34.864 DEBUG:boto:Port: 8000 2023-10-17T18:53:34.864 DEBUG:boto:Params: {} 2023-10-17T18:53:34.865 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:53:34.866 DEBUG:boto:Token: None 2023-10-17T18:53:34.866 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/ 2023-10-17T18:53:34.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WdSkGWx2n1C3Lnk+sz9YxmjWfpI= 2023-10-17T18:53:34.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WdSkGWx2n1C3Lnk+sz9YxmjWfpI=', 'Content-Length': '0'} 2023-10-17T18:53:34.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c674d83f288491f-00652ed82e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:53:34.872 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.873 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.873 DEBUG:boto:path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.873 DEBUG:boto:auth_path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.874 DEBUG:boto:Method: GET 2023-10-17T18:53:34.874 DEBUG:boto:Path: /hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.875 DEBUG:boto:Data: 2023-10-17T18:53:34.875 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.876 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.876 DEBUG:boto:Params: {} 2023-10-17T18:53:34.877 DEBUG:boto:Token: None 2023-10-17T18:53:34.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/?versions 2023-10-17T18:53:34.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE= 2023-10-17T18:53:34.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE=', 'Content-Length': '0'} 2023-10-17T18:53:34.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a38857320a22170-00652ed82e-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.881 DEBUG:boto:b'hbmuog-57999falseanulltrue2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:34.883 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:53:34.883 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:53:34.884 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:53:34.884 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:53:34.884 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:53:34.885 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:53:34.885 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:53:34.886 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:53:34.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:53:34.887 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.887 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.887 DEBUG:boto:path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.888 DEBUG:boto:auth_path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.888 DEBUG:boto:Method: GET 2023-10-17T18:53:34.889 DEBUG:boto:Path: /hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.889 DEBUG:boto:Data: 2023-10-17T18:53:34.890 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:34.891 DEBUG:boto:Port: 8000 2023-10-17T18:53:34.891 DEBUG:boto:Params: {} 2023-10-17T18:53:34.891 DEBUG:boto:Token: None 2023-10-17T18:53:34.892 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/?versions 2023-10-17T18:53:34.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE= 2023-10-17T18:53:34.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE=', 'Content-Length': '0'} 2023-10-17T18:53:34.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15fff6cc578142c-00652ed82e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.896 DEBUG:boto:b'hbmuog-57999falseanulltrue2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:34.897 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-17T18:53:34.897 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-17T18:53:34.898 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-17T18:53:34.898 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-17T18:53:34.899 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-17T18:53:34.899 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-17T18:53:34.899 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-17T18:53:34.900 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-17T18:53:34.900 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.901 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.901 DEBUG:boto:path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.902 DEBUG:boto:auth_path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.902 DEBUG:boto:Method: GET 2023-10-17T18:53:34.903 DEBUG:boto:Path: /hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.903 DEBUG:boto:Data: 2023-10-17T18:53:34.903 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.904 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.904 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.905 DEBUG:boto:Params: {} 2023-10-17T18:53:34.905 DEBUG:boto:Token: None 2023-10-17T18:53:34.906 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/?versions 2023-10-17T18:53:34.906 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE= 2023-10-17T18:53:34.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE=', 'Content-Length': '0'} 2023-10-17T18:53:34.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b02e29f98cd59086-00652ed82e-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.910 DEBUG:boto:b'hbmuog-57999falseanulltrue2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:34.911 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:34.912 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:34.912 DEBUG:boto:path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.913 DEBUG:boto:auth_path=/hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.913 DEBUG:boto:Method: GET 2023-10-17T18:53:34.914 DEBUG:boto:Path: /hbmuog-57/?versions&max-keys=999 2023-10-17T18:53:34.914 DEBUG:boto:Data: 2023-10-17T18:53:34.915 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:34.915 DEBUG:boto:Port: 8000 2023-10-17T18:53:34.916 DEBUG:boto:Params: {} 2023-10-17T18:53:34.916 DEBUG:boto:Token: None 2023-10-17T18:53:34.917 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/?versions 2023-10-17T18:53:34.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE= 2023-10-17T18:53:34.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XvCfy3IdrctuztPADMmFxFisxnE=', 'Content-Length': '0'} 2023-10-17T18:53:34.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbedb286f52da61-00652ed82e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.921 DEBUG:boto:b'hbmuog-57999falseanulltrue2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:34.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:53:34.922 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:34.923 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:34.923 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.924 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.924 DEBUG:boto:Method: GET 2023-10-17T18:53:34.925 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.925 DEBUG:boto:Data: 2023-10-17T18:53:34.926 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.926 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.927 DEBUG:boto:Params: {} 2023-10-17T18:53:34.927 DEBUG:boto:Token: None 2023-10-17T18:53:34.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0= 2023-10-17T18:53:34.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0=', 'Content-Length': '0'} 2023-10-17T18:53:34.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7105ceaea81dda-00652ed82e-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.933 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:34.933 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:34.934 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.934 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.934 DEBUG:boto:Method: GET 2023-10-17T18:53:34.935 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.935 DEBUG:boto:Data: 2023-10-17T18:53:34.936 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.936 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:34.937 DEBUG:boto:Port: 8000 2023-10-17T18:53:34.937 DEBUG:boto:Params: {} 2023-10-17T18:53:34.938 DEBUG:boto:Token: None 2023-10-17T18:53:34.938 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0= 2023-10-17T18:53:34.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0=', 'Content-Length': '0'} 2023-10-17T18:53:34.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad88495ce72e8cc-00652ed82e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.942 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:34.943 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:34.943 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.943 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.944 DEBUG:boto:Method: HEAD 2023-10-17T18:53:34.944 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.945 DEBUG:boto:Data: 2023-10-17T18:53:34.945 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.946 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.946 DEBUG:boto:Params: {} 2023-10-17T18:53:34.947 DEBUG:boto:Token: None 2023-10-17T18:53:34.947 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/6w5xWa3TRSolCxHF3+JDxA/gPo= 2023-10-17T18:53:34.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/6w5xWa3TRSolCxHF3+JDxA/gPo=', 'Content-Length': '0'} 2023-10-17T18:53:34.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ace85c74e28d3d9-00652ed82e-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.951 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:34.951 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:34.952 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.952 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.953 DEBUG:boto:Method: HEAD 2023-10-17T18:53:34.953 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.954 DEBUG:boto:Data: 2023-10-17T18:53:34.954 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.955 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:34.955 DEBUG:boto:Port: 8000 2023-10-17T18:53:34.955 DEBUG:boto:Params: {} 2023-10-17T18:53:34.956 DEBUG:boto:Token: None 2023-10-17T18:53:34.957 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.957 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/6w5xWa3TRSolCxHF3+JDxA/gPo= 2023-10-17T18:53:34.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/6w5xWa3TRSolCxHF3+JDxA/gPo=', 'Content-Length': '0'} 2023-10-17T18:53:34.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7877471ddc1cefd-00652ed82e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:34.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:53:34.961 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:34.961 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:34.962 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.962 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:34.963 DEBUG:boto:Method: GET 2023-10-17T18:53:34.963 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.964 DEBUG:boto:Data: 2023-10-17T18:53:34.964 DEBUG:boto:Headers: {} 2023-10-17T18:53:34.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:34.965 DEBUG:boto:Port: 8001 2023-10-17T18:53:34.965 DEBUG:boto:Params: {} 2023-10-17T18:53:34.966 DEBUG:boto:Token: None 2023-10-17T18:53:34.967 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:34 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:34.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0= 2023-10-17T18:53:34.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0XcCHKALDDHcMH7oqPLm96Wojd0=', 'Content-Length': '0'} 2023-10-17T18:53:34.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002719bdb07982d1ec-00652ed82e-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.010 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:35.011 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:35.011 DEBUG:boto:path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:35.012 DEBUG:boto:auth_path=/hbmuog-57/a?versionId=null 2023-10-17T18:53:35.012 DEBUG:boto:Method: GET 2023-10-17T18:53:35.013 DEBUG:boto:Path: /hbmuog-57/a?versionId=null 2023-10-17T18:53:35.013 DEBUG:boto:Data: 2023-10-17T18:53:35.014 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.014 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.015 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.015 DEBUG:boto:Params: {} 2023-10-17T18:53:35.016 DEBUG:boto:Token: None 2023-10-17T18:53:35.016 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/a?versionId=null 2023-10-17T18:53:35.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:POulXuH+hoAStTF9jDtKZgvEfGM= 2023-10-17T18:53:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:POulXuH+hoAStTF9jDtKZgvEfGM=', 'Content-Length': '0'} 2023-10-17T18:53:35.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf296f6c163ddfe-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.020 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.021 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.021 DEBUG:boto:path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.022 DEBUG:boto:auth_path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.022 DEBUG:boto:Method: GET 2023-10-17T18:53:35.022 DEBUG:boto:Path: /hbmuog-57/?prefix=a 2023-10-17T18:53:35.023 DEBUG:boto:Data: 2023-10-17T18:53:35.023 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.024 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.024 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.025 DEBUG:boto:Params: {} 2023-10-17T18:53:35.025 DEBUG:boto:Token: None 2023-10-17T18:53:35.026 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.026 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e2c1d47ebc19f2-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.030 DEBUG:boto:b'hbmuog-57a1000falsea2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.031 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.031 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.032 DEBUG:boto:path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.032 DEBUG:boto:auth_path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.033 DEBUG:boto:Method: GET 2023-10-17T18:53:35.033 DEBUG:boto:Path: /hbmuog-57/?prefix=a 2023-10-17T18:53:35.034 DEBUG:boto:Data: 2023-10-17T18:53:35.034 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.035 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.035 DEBUG:boto:Params: {} 2023-10-17T18:53:35.036 DEBUG:boto:Token: None 2023-10-17T18:53:35.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb0b5acc20edac5-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.086 DEBUG:boto:b'hbmuog-57a1000falsea2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.087 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:35.088 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:35.088 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.088 DEBUG:boto:Path: /hbmuog-57/a 2023-10-17T18:53:35.089 DEBUG:boto:Data: 2023-10-17T18:53:35.089 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.089 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.090 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.090 DEBUG:boto:Params: {} 2023-10-17T18:53:35.091 DEBUG:boto:Token: None 2023-10-17T18:53:35.091 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/a 2023-10-17T18:53:35.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sIF6ojx6ON9y2XOwrlVKe82EkKQ= 2023-10-17T18:53:35.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sIF6ojx6ON9y2XOwrlVKe82EkKQ=', 'Content-Length': '0'} 2023-10-17T18:53:35.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e035b7fc4fc3de-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.095 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:35.095 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:35.096 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.096 DEBUG:boto:Path: /hbmuog-57/a 2023-10-17T18:53:35.096 DEBUG:boto:Data: 2023-10-17T18:53:35.097 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.097 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.098 DEBUG:boto:Params: {} 2023-10-17T18:53:35.098 DEBUG:boto:Token: None 2023-10-17T18:53:35.099 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/a 2023-10-17T18:53:35.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sIF6ojx6ON9y2XOwrlVKe82EkKQ= 2023-10-17T18:53:35.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sIF6ojx6ON9y2XOwrlVKe82EkKQ=', 'Content-Length': '0'} 2023-10-17T18:53:35.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007762b7f506ad4e85-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-17T18:53:35.103 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:35.103 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:35.104 DEBUG:boto:Method: GET 2023-10-17T18:53:35.104 DEBUG:boto:Path: /hbmuog-57/a 2023-10-17T18:53:35.104 DEBUG:boto:Data: 2023-10-17T18:53:35.105 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.105 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.106 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.106 DEBUG:boto:Params: {} 2023-10-17T18:53:35.107 DEBUG:boto:Token: None 2023-10-17T18:53:35.107 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/a 2023-10-17T18:53:35.107 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:00zjNvsbEilWl2in/rbnTdS8kUg= 2023-10-17T18:53:35.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:00zjNvsbEilWl2in/rbnTdS8kUg=', 'Content-Length': '0'} 2023-10-17T18:53:35.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9707ef98bb9c7e-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.110 DEBUG:boto:path=/hbmuog-57/a 2023-10-17T18:53:35.111 DEBUG:boto:auth_path=/hbmuog-57/a 2023-10-17T18:53:35.111 DEBUG:boto:Method: GET 2023-10-17T18:53:35.112 DEBUG:boto:Path: /hbmuog-57/a 2023-10-17T18:53:35.112 DEBUG:boto:Data: 2023-10-17T18:53:35.112 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.113 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.114 DEBUG:boto:Params: {} 2023-10-17T18:53:35.114 DEBUG:boto:Token: None 2023-10-17T18:53:35.115 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/a 2023-10-17T18:53:35.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:00zjNvsbEilWl2in/rbnTdS8kUg= 2023-10-17T18:53:35.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:00zjNvsbEilWl2in/rbnTdS8kUg=', 'Content-Length': '0'} 2023-10-17T18:53:35.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c42a623e64a29b2-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.118 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.118 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.119 DEBUG:boto:path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.119 DEBUG:boto:auth_path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.120 DEBUG:boto:Method: GET 2023-10-17T18:53:35.120 DEBUG:boto:Path: /hbmuog-57/?prefix=a 2023-10-17T18:53:35.121 DEBUG:boto:Data: 2023-10-17T18:53:35.121 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.121 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.122 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.122 DEBUG:boto:Params: {} 2023-10-17T18:53:35.123 DEBUG:boto:Token: None 2023-10-17T18:53:35.123 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.124 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a8835a1dedaff0-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.127 DEBUG:boto:b'hbmuog-57a1000falsea2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.128 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.128 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.129 DEBUG:boto:path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.129 DEBUG:boto:auth_path=/hbmuog-57/?prefix=a 2023-10-17T18:53:35.129 DEBUG:boto:Method: GET 2023-10-17T18:53:35.130 DEBUG:boto:Path: /hbmuog-57/?prefix=a 2023-10-17T18:53:35.130 DEBUG:boto:Data: 2023-10-17T18:53:35.134 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.134 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.135 DEBUG:boto:Params: {} 2023-10-17T18:53:35.135 DEBUG:boto:Token: None 2023-10-17T18:53:35.136 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000612ef801a07e20dc-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.139 DEBUG:boto:b'hbmuog-57a1000falsea2023-10-17T18:50:43.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:53:35.140 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.141 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.141 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.142 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.142 DEBUG:boto:Method: GET 2023-10-17T18:53:35.143 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.143 DEBUG:boto:Data: 2023-10-17T18:53:35.143 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.144 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.144 DEBUG:boto:Params: {} 2023-10-17T18:53:35.145 DEBUG:boto:Token: None 2023-10-17T18:53:35.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM= 2023-10-17T18:53:35.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM=', 'Content-Length': '0'} 2023-10-17T18:53:35.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9acb77f8ddecd0-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.190 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.191 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.191 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.192 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.192 DEBUG:boto:Method: GET 2023-10-17T18:53:35.193 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.193 DEBUG:boto:Data: 2023-10-17T18:53:35.193 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.194 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.194 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.195 DEBUG:boto:Params: {} 2023-10-17T18:53:35.195 DEBUG:boto:Token: None 2023-10-17T18:53:35.196 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM= 2023-10-17T18:53:35.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM=', 'Content-Length': '0'} 2023-10-17T18:53:35.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c16a15993460e6-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.199 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.200 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.200 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.201 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.201 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.201 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.202 DEBUG:boto:Data: 2023-10-17T18:53:35.202 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.203 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.203 DEBUG:boto:Params: {} 2023-10-17T18:53:35.204 DEBUG:boto:Token: None 2023-10-17T18:53:35.204 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngB6JY6MsoXV1LX95G38SAxBpgA= 2023-10-17T18:53:35.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngB6JY6MsoXV1LX95G38SAxBpgA=', 'Content-Length': '0'} 2023-10-17T18:53:35.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498dac5bc227f6fd-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.207 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.208 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.208 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.208 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.209 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.209 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.210 DEBUG:boto:Data: 2023-10-17T18:53:35.210 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.210 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.211 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.211 DEBUG:boto:Params: {} 2023-10-17T18:53:35.212 DEBUG:boto:Token: None 2023-10-17T18:53:35.212 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngB6JY6MsoXV1LX95G38SAxBpgA= 2023-10-17T18:53:35.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngB6JY6MsoXV1LX95G38SAxBpgA=', 'Content-Length': '0'} 2023-10-17T18:53:35.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07b61cc86b89c6a-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:53:35.216 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.216 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.216 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.217 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.217 DEBUG:boto:Method: GET 2023-10-17T18:53:35.218 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.218 DEBUG:boto:Data: 2023-10-17T18:53:35.218 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.219 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.219 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.220 DEBUG:boto:Params: {} 2023-10-17T18:53:35.220 DEBUG:boto:Token: None 2023-10-17T18:53:35.220 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.221 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM= 2023-10-17T18:53:35.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM=', 'Content-Length': '0'} 2023-10-17T18:53:35.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5f3fa53b1a761e-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.224 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.224 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.225 DEBUG:boto:path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.225 DEBUG:boto:auth_path=/hbmuog-57/b?versionId=null 2023-10-17T18:53:35.225 DEBUG:boto:Method: GET 2023-10-17T18:53:35.226 DEBUG:boto:Path: /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.226 DEBUG:boto:Data: 2023-10-17T18:53:35.227 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.227 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.228 DEBUG:boto:Params: {} 2023-10-17T18:53:35.228 DEBUG:boto:Token: None 2023-10-17T18:53:35.229 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b?versionId=null 2023-10-17T18:53:35.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM= 2023-10-17T18:53:35.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:adDIZMWquyMnovruDraxr7Mc7AM=', 'Content-Length': '0'} 2023-10-17T18:53:35.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ce57102c3515fd-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.274 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.275 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.275 DEBUG:boto:path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.275 DEBUG:boto:auth_path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.276 DEBUG:boto:Method: GET 2023-10-17T18:53:35.276 DEBUG:boto:Path: /hbmuog-57/?prefix=b 2023-10-17T18:53:35.277 DEBUG:boto:Data: 2023-10-17T18:53:35.277 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.277 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.278 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.278 DEBUG:boto:Params: {} 2023-10-17T18:53:35.279 DEBUG:boto:Token: None 2023-10-17T18:53:35.279 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.279 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49d6ce9a43fae09-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.283 DEBUG:boto:b'hbmuog-57b1000falseb2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.284 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.284 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.285 DEBUG:boto:path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.285 DEBUG:boto:auth_path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.285 DEBUG:boto:Method: GET 2023-10-17T18:53:35.286 DEBUG:boto:Path: /hbmuog-57/?prefix=b 2023-10-17T18:53:35.286 DEBUG:boto:Data: 2023-10-17T18:53:35.287 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.287 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.288 DEBUG:boto:Params: {} 2023-10-17T18:53:35.288 DEBUG:boto:Token: None 2023-10-17T18:53:35.289 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e23886fc516cee1-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.292 DEBUG:boto:b'hbmuog-57b1000falseb2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.293 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.293 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.293 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.294 DEBUG:boto:Path: /hbmuog-57/b 2023-10-17T18:53:35.294 DEBUG:boto:Data: 2023-10-17T18:53:35.294 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.295 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.296 DEBUG:boto:Params: {} 2023-10-17T18:53:35.296 DEBUG:boto:Token: None 2023-10-17T18:53:35.296 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b 2023-10-17T18:53:35.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0pDaQrDksXfqNbEMGsy8IkZZbgM= 2023-10-17T18:53:35.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0pDaQrDksXfqNbEMGsy8IkZZbgM=', 'Content-Length': '0'} 2023-10-17T18:53:35.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000927d75da0b9500d1-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.300 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.300 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.300 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.301 DEBUG:boto:Path: /hbmuog-57/b 2023-10-17T18:53:35.301 DEBUG:boto:Data: 2023-10-17T18:53:35.302 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.302 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.303 DEBUG:boto:Params: {} 2023-10-17T18:53:35.303 DEBUG:boto:Token: None 2023-10-17T18:53:35.303 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b 2023-10-17T18:53:35.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0pDaQrDksXfqNbEMGsy8IkZZbgM= 2023-10-17T18:53:35.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0pDaQrDksXfqNbEMGsy8IkZZbgM=', 'Content-Length': '0'} 2023-10-17T18:53:35.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248ef225b73bd822-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-17T18:53:35.307 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.307 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.308 DEBUG:boto:Method: GET 2023-10-17T18:53:35.308 DEBUG:boto:Path: /hbmuog-57/b 2023-10-17T18:53:35.308 DEBUG:boto:Data: 2023-10-17T18:53:35.309 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.310 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.310 DEBUG:boto:Params: {} 2023-10-17T18:53:35.310 DEBUG:boto:Token: None 2023-10-17T18:53:35.311 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b 2023-10-17T18:53:35.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HmgrjhSmXsLLm1qMmTY/tTIlLGs= 2023-10-17T18:53:35.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HmgrjhSmXsLLm1qMmTY/tTIlLGs=', 'Content-Length': '0'} 2023-10-17T18:53:35.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd0391e176e9047-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.354 DEBUG:boto:path=/hbmuog-57/b 2023-10-17T18:53:35.355 DEBUG:boto:auth_path=/hbmuog-57/b 2023-10-17T18:53:35.355 DEBUG:boto:Method: GET 2023-10-17T18:53:35.355 DEBUG:boto:Path: /hbmuog-57/b 2023-10-17T18:53:35.356 DEBUG:boto:Data: 2023-10-17T18:53:35.356 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.357 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.357 DEBUG:boto:Params: {} 2023-10-17T18:53:35.358 DEBUG:boto:Token: None 2023-10-17T18:53:35.358 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/b 2023-10-17T18:53:35.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HmgrjhSmXsLLm1qMmTY/tTIlLGs= 2023-10-17T18:53:35.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HmgrjhSmXsLLm1qMmTY/tTIlLGs=', 'Content-Length': '0'} 2023-10-17T18:53:35.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001433d7055e044e76-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.362 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.362 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.363 DEBUG:boto:path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.363 DEBUG:boto:auth_path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.363 DEBUG:boto:Method: GET 2023-10-17T18:53:35.364 DEBUG:boto:Path: /hbmuog-57/?prefix=b 2023-10-17T18:53:35.364 DEBUG:boto:Data: 2023-10-17T18:53:35.364 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.365 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.366 DEBUG:boto:Params: {} 2023-10-17T18:53:35.366 DEBUG:boto:Token: None 2023-10-17T18:53:35.367 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc4f8569a07d494-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.371 DEBUG:boto:b'hbmuog-57b1000falseb2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.371 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.372 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.372 DEBUG:boto:path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.373 DEBUG:boto:auth_path=/hbmuog-57/?prefix=b 2023-10-17T18:53:35.373 DEBUG:boto:Method: GET 2023-10-17T18:53:35.373 DEBUG:boto:Path: /hbmuog-57/?prefix=b 2023-10-17T18:53:35.374 DEBUG:boto:Data: 2023-10-17T18:53:35.374 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.375 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.375 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.375 DEBUG:boto:Params: {} 2023-10-17T18:53:35.376 DEBUG:boto:Token: None 2023-10-17T18:53:35.376 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b1f61e62632694-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.380 DEBUG:boto:b'hbmuog-57b1000falseb2023-10-17T18:50:43.098Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:53:35.381 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.381 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.382 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.382 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.382 DEBUG:boto:Method: GET 2023-10-17T18:53:35.383 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.383 DEBUG:boto:Data: 2023-10-17T18:53:35.384 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.384 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.385 DEBUG:boto:Params: {} 2023-10-17T18:53:35.385 DEBUG:boto:Token: None 2023-10-17T18:53:35.386 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.386 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I= 2023-10-17T18:53:35.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I=', 'Content-Length': '0'} 2023-10-17T18:53:35.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776b359ae16d928a-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.389 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.390 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.390 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.390 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.391 DEBUG:boto:Method: GET 2023-10-17T18:53:35.391 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.392 DEBUG:boto:Data: 2023-10-17T18:53:35.392 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.392 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.393 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.393 DEBUG:boto:Params: {} 2023-10-17T18:53:35.393 DEBUG:boto:Token: None 2023-10-17T18:53:35.394 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.394 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I= 2023-10-17T18:53:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I=', 'Content-Length': '0'} 2023-10-17T18:53:35.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e5528fcb8a4b48-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.397 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.398 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.398 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.399 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.399 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.399 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.400 DEBUG:boto:Data: 2023-10-17T18:53:35.400 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.401 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.401 DEBUG:boto:Params: {} 2023-10-17T18:53:35.402 DEBUG:boto:Token: None 2023-10-17T18:53:35.402 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zf0p8vc541qDWPNQ2RerPdPOurs= 2023-10-17T18:53:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zf0p8vc541qDWPNQ2RerPdPOurs=', 'Content-Length': '0'} 2023-10-17T18:53:35.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7cb613cd3bac4f-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.405 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.406 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.406 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.407 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.407 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.407 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.408 DEBUG:boto:Data: 2023-10-17T18:53:35.408 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.409 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.409 DEBUG:boto:Params: {} 2023-10-17T18:53:35.410 DEBUG:boto:Token: None 2023-10-17T18:53:35.410 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zf0p8vc541qDWPNQ2RerPdPOurs= 2023-10-17T18:53:35.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zf0p8vc541qDWPNQ2RerPdPOurs=', 'Content-Length': '0'} 2023-10-17T18:53:35.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01397a7cc473e4d-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:53:35.414 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.414 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.415 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.415 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.415 DEBUG:boto:Method: GET 2023-10-17T18:53:35.416 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.416 DEBUG:boto:Data: 2023-10-17T18:53:35.416 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.417 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.418 DEBUG:boto:Params: {} 2023-10-17T18:53:35.418 DEBUG:boto:Token: None 2023-10-17T18:53:35.419 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.419 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I= 2023-10-17T18:53:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I=', 'Content-Length': '0'} 2023-10-17T18:53:35.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32c244321542d76-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.422 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.422 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.423 DEBUG:boto:path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.423 DEBUG:boto:auth_path=/hbmuog-57/c?versionId=null 2023-10-17T18:53:35.423 DEBUG:boto:Method: GET 2023-10-17T18:53:35.424 DEBUG:boto:Path: /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.424 DEBUG:boto:Data: 2023-10-17T18:53:35.425 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.425 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.425 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.426 DEBUG:boto:Params: {} 2023-10-17T18:53:35.426 DEBUG:boto:Token: None 2023-10-17T18:53:35.427 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c?versionId=null 2023-10-17T18:53:35.427 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I= 2023-10-17T18:53:35.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sm0VDmT4AoQPzeusGzXyhydGv1I=', 'Content-Length': '0'} 2023-10-17T18:53:35.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135bfbb02a622661-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.430 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.431 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.431 DEBUG:boto:path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.432 DEBUG:boto:auth_path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.432 DEBUG:boto:Method: GET 2023-10-17T18:53:35.432 DEBUG:boto:Path: /hbmuog-57/?prefix=c 2023-10-17T18:53:35.433 DEBUG:boto:Data: 2023-10-17T18:53:35.433 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.433 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.434 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.434 DEBUG:boto:Params: {} 2023-10-17T18:53:35.435 DEBUG:boto:Token: None 2023-10-17T18:53:35.435 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.436 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc18827189f5a21-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.439 DEBUG:boto:b'hbmuog-57c1000falsec2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.440 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.440 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.441 DEBUG:boto:path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.441 DEBUG:boto:auth_path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.441 DEBUG:boto:Method: GET 2023-10-17T18:53:35.442 DEBUG:boto:Path: /hbmuog-57/?prefix=c 2023-10-17T18:53:35.442 DEBUG:boto:Data: 2023-10-17T18:53:35.443 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.443 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.443 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.444 DEBUG:boto:Params: {} 2023-10-17T18:53:35.444 DEBUG:boto:Token: None 2023-10-17T18:53:35.445 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ebd7317b4a9a2a7-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.448 DEBUG:boto:b'hbmuog-57c1000falsec2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.449 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.449 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.449 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.450 DEBUG:boto:Path: /hbmuog-57/c 2023-10-17T18:53:35.450 DEBUG:boto:Data: 2023-10-17T18:53:35.451 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.451 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.452 DEBUG:boto:Params: {} 2023-10-17T18:53:35.452 DEBUG:boto:Token: None 2023-10-17T18:53:35.452 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c 2023-10-17T18:53:35.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0FpdtByAPVUCLnaknAw01qcjf84= 2023-10-17T18:53:35.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0FpdtByAPVUCLnaknAw01qcjf84=', 'Content-Length': '0'} 2023-10-17T18:53:35.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007515cbb3ff8c835-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.455 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.456 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.456 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.457 DEBUG:boto:Path: /hbmuog-57/c 2023-10-17T18:53:35.457 DEBUG:boto:Data: 2023-10-17T18:53:35.457 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.458 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.458 DEBUG:boto:Params: {} 2023-10-17T18:53:35.459 DEBUG:boto:Token: None 2023-10-17T18:53:35.459 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c 2023-10-17T18:53:35.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0FpdtByAPVUCLnaknAw01qcjf84= 2023-10-17T18:53:35.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0FpdtByAPVUCLnaknAw01qcjf84=', 'Content-Length': '0'} 2023-10-17T18:53:35.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0946b09ddaf7f25-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-17T18:53:35.463 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.463 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.464 DEBUG:boto:Method: GET 2023-10-17T18:53:35.464 DEBUG:boto:Path: /hbmuog-57/c 2023-10-17T18:53:35.464 DEBUG:boto:Data: 2023-10-17T18:53:35.465 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.466 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.466 DEBUG:boto:Params: {} 2023-10-17T18:53:35.466 DEBUG:boto:Token: None 2023-10-17T18:53:35.467 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c 2023-10-17T18:53:35.467 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P2+aUSk4gHu/OiZecJOHsnb7bN8= 2023-10-17T18:53:35.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P2+aUSk4gHu/OiZecJOHsnb7bN8=', 'Content-Length': '0'} 2023-10-17T18:53:35.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f82cca0ca875493-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.510 DEBUG:boto:path=/hbmuog-57/c 2023-10-17T18:53:35.511 DEBUG:boto:auth_path=/hbmuog-57/c 2023-10-17T18:53:35.511 DEBUG:boto:Method: GET 2023-10-17T18:53:35.511 DEBUG:boto:Path: /hbmuog-57/c 2023-10-17T18:53:35.512 DEBUG:boto:Data: 2023-10-17T18:53:35.512 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.513 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.513 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.513 DEBUG:boto:Params: {} 2023-10-17T18:53:35.514 DEBUG:boto:Token: None 2023-10-17T18:53:35.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/c 2023-10-17T18:53:35.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P2+aUSk4gHu/OiZecJOHsnb7bN8= 2023-10-17T18:53:35.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P2+aUSk4gHu/OiZecJOHsnb7bN8=', 'Content-Length': '0'} 2023-10-17T18:53:35.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e828656bc8fd94-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.518 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.519 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.519 DEBUG:boto:path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.519 DEBUG:boto:auth_path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.520 DEBUG:boto:Method: GET 2023-10-17T18:53:35.520 DEBUG:boto:Path: /hbmuog-57/?prefix=c 2023-10-17T18:53:35.521 DEBUG:boto:Data: 2023-10-17T18:53:35.521 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.522 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.523 DEBUG:boto:Params: {} 2023-10-17T18:53:35.523 DEBUG:boto:Token: None 2023-10-17T18:53:35.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b0bcec21c4bca8-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.528 DEBUG:boto:b'hbmuog-57c1000falsec2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.529 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.529 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.529 DEBUG:boto:path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.530 DEBUG:boto:auth_path=/hbmuog-57/?prefix=c 2023-10-17T18:53:35.530 DEBUG:boto:Method: GET 2023-10-17T18:53:35.531 DEBUG:boto:Path: /hbmuog-57/?prefix=c 2023-10-17T18:53:35.531 DEBUG:boto:Data: 2023-10-17T18:53:35.532 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.532 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.533 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.533 DEBUG:boto:Params: {} 2023-10-17T18:53:35.534 DEBUG:boto:Token: None 2023-10-17T18:53:35.534 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.535 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49f355f0e57c082-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.538 DEBUG:boto:b'hbmuog-57c1000falsec2023-10-17T18:50:43.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:53:35.539 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.540 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.541 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.541 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.541 DEBUG:boto:Method: GET 2023-10-17T18:53:35.542 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.542 DEBUG:boto:Data: 2023-10-17T18:53:35.543 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.543 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.544 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.544 DEBUG:boto:Params: {} 2023-10-17T18:53:35.544 DEBUG:boto:Token: None 2023-10-17T18:53:35.545 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.545 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0= 2023-10-17T18:53:35.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0=', 'Content-Length': '0'} 2023-10-17T18:53:35.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004083c0731191c303-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.549 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.549 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.549 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.550 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.550 DEBUG:boto:Method: GET 2023-10-17T18:53:35.551 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.551 DEBUG:boto:Data: 2023-10-17T18:53:35.552 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.552 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.553 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.553 DEBUG:boto:Params: {} 2023-10-17T18:53:35.553 DEBUG:boto:Token: None 2023-10-17T18:53:35.554 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.554 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0= 2023-10-17T18:53:35.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0=', 'Content-Length': '0'} 2023-10-17T18:53:35.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c5841fc5b48adce-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.558 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.558 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.559 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.559 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.559 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.560 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.560 DEBUG:boto:Data: 2023-10-17T18:53:35.561 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.561 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.562 DEBUG:boto:Params: {} 2023-10-17T18:53:35.562 DEBUG:boto:Token: None 2023-10-17T18:53:35.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4H1jZzVycuikUydN+2WJFx5SKtk= 2023-10-17T18:53:35.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4H1jZzVycuikUydN+2WJFx5SKtk=', 'Content-Length': '0'} 2023-10-17T18:53:35.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564cbf88b833d7b8-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.566 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.567 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.567 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.568 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.568 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.568 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.569 DEBUG:boto:Data: 2023-10-17T18:53:35.569 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.570 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.570 DEBUG:boto:Params: {} 2023-10-17T18:53:35.571 DEBUG:boto:Token: None 2023-10-17T18:53:35.572 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4H1jZzVycuikUydN+2WJFx5SKtk= 2023-10-17T18:53:35.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4H1jZzVycuikUydN+2WJFx5SKtk=', 'Content-Length': '0'} 2023-10-17T18:53:35.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc460b9a8432dc97-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:53:35.575 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.576 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.576 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.577 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.577 DEBUG:boto:Method: GET 2023-10-17T18:53:35.577 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.578 DEBUG:boto:Data: 2023-10-17T18:53:35.578 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.579 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.580 DEBUG:boto:Params: {} 2023-10-17T18:53:35.580 DEBUG:boto:Token: None 2023-10-17T18:53:35.581 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0= 2023-10-17T18:53:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0=', 'Content-Length': '0'} 2023-10-17T18:53:35.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de490b79a20033c3-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.584 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.585 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.585 DEBUG:boto:path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.586 DEBUG:boto:auth_path=/hbmuog-57/d?versionId=null 2023-10-17T18:53:35.586 DEBUG:boto:Method: GET 2023-10-17T18:53:35.586 DEBUG:boto:Path: /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.587 DEBUG:boto:Data: 2023-10-17T18:53:35.587 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.588 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.589 DEBUG:boto:Params: {} 2023-10-17T18:53:35.589 DEBUG:boto:Token: None 2023-10-17T18:53:35.589 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d?versionId=null 2023-10-17T18:53:35.590 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0= 2023-10-17T18:53:35.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8xtPdqIWTQCPzEpF6kltEdnjtZ0=', 'Content-Length': '0'} 2023-10-17T18:53:35.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfd87147568bbada-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.638 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.639 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.639 DEBUG:boto:path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.640 DEBUG:boto:auth_path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.640 DEBUG:boto:Method: GET 2023-10-17T18:53:35.641 DEBUG:boto:Path: /hbmuog-57/?prefix=d 2023-10-17T18:53:35.641 DEBUG:boto:Data: 2023-10-17T18:53:35.642 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.642 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.643 DEBUG:boto:Params: {} 2023-10-17T18:53:35.643 DEBUG:boto:Token: None 2023-10-17T18:53:35.644 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef408a932fa536e-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.648 DEBUG:boto:b'hbmuog-57d1000falsed2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.649 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.649 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.650 DEBUG:boto:path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.650 DEBUG:boto:auth_path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.651 DEBUG:boto:Method: GET 2023-10-17T18:53:35.651 DEBUG:boto:Path: /hbmuog-57/?prefix=d 2023-10-17T18:53:35.652 DEBUG:boto:Data: 2023-10-17T18:53:35.652 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.653 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.653 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.653 DEBUG:boto:Params: {} 2023-10-17T18:53:35.654 DEBUG:boto:Token: None 2023-10-17T18:53:35.654 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b1eb85b0f5b9c4-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.658 DEBUG:boto:b'hbmuog-57d1000falsed2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.659 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.659 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.660 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.660 DEBUG:boto:Path: /hbmuog-57/d 2023-10-17T18:53:35.660 DEBUG:boto:Data: 2023-10-17T18:53:35.661 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.661 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.662 DEBUG:boto:Params: {} 2023-10-17T18:53:35.662 DEBUG:boto:Token: None 2023-10-17T18:53:35.663 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d 2023-10-17T18:53:35.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NGPbUAI/7sYR0KB+QgydV7iMFZU= 2023-10-17T18:53:35.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NGPbUAI/7sYR0KB+QgydV7iMFZU=', 'Content-Length': '0'} 2023-10-17T18:53:35.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029412f52c80f92f5-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.666 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.667 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.667 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.668 DEBUG:boto:Path: /hbmuog-57/d 2023-10-17T18:53:35.668 DEBUG:boto:Data: 2023-10-17T18:53:35.668 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.669 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.669 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.670 DEBUG:boto:Params: {} 2023-10-17T18:53:35.670 DEBUG:boto:Token: None 2023-10-17T18:53:35.671 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d 2023-10-17T18:53:35.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NGPbUAI/7sYR0KB+QgydV7iMFZU= 2023-10-17T18:53:35.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NGPbUAI/7sYR0KB+QgydV7iMFZU=', 'Content-Length': '0'} 2023-10-17T18:53:35.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2ae584c3db2d00-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-17T18:53:35.675 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.675 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.675 DEBUG:boto:Method: GET 2023-10-17T18:53:35.676 DEBUG:boto:Path: /hbmuog-57/d 2023-10-17T18:53:35.676 DEBUG:boto:Data: 2023-10-17T18:53:35.677 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.678 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.678 DEBUG:boto:Params: {} 2023-10-17T18:53:35.679 DEBUG:boto:Token: None 2023-10-17T18:53:35.679 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d 2023-10-17T18:53:35.680 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:33vSzhCfPOTkRmnOjNAHNjHB6D0= 2023-10-17T18:53:35.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:33vSzhCfPOTkRmnOjNAHNjHB6D0=', 'Content-Length': '0'} 2023-10-17T18:53:35.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b861eed8e98ae00b-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.722 DEBUG:boto:path=/hbmuog-57/d 2023-10-17T18:53:35.723 DEBUG:boto:auth_path=/hbmuog-57/d 2023-10-17T18:53:35.723 DEBUG:boto:Method: GET 2023-10-17T18:53:35.724 DEBUG:boto:Path: /hbmuog-57/d 2023-10-17T18:53:35.724 DEBUG:boto:Data: 2023-10-17T18:53:35.725 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.725 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.726 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.726 DEBUG:boto:Params: {} 2023-10-17T18:53:35.727 DEBUG:boto:Token: None 2023-10-17T18:53:35.727 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/d 2023-10-17T18:53:35.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:33vSzhCfPOTkRmnOjNAHNjHB6D0= 2023-10-17T18:53:35.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:33vSzhCfPOTkRmnOjNAHNjHB6D0=', 'Content-Length': '0'} 2023-10-17T18:53:35.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d0bad14bf09ef6-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.731 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.732 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.732 DEBUG:boto:path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.732 DEBUG:boto:auth_path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.733 DEBUG:boto:Method: GET 2023-10-17T18:53:35.733 DEBUG:boto:Path: /hbmuog-57/?prefix=d 2023-10-17T18:53:35.734 DEBUG:boto:Data: 2023-10-17T18:53:35.734 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.735 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.735 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.736 DEBUG:boto:Params: {} 2023-10-17T18:53:35.736 DEBUG:boto:Token: None 2023-10-17T18:53:35.737 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.737 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147b243cef24074a-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.741 DEBUG:boto:b'hbmuog-57d1000falsed2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.741 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.742 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.742 DEBUG:boto:path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.743 DEBUG:boto:auth_path=/hbmuog-57/?prefix=d 2023-10-17T18:53:35.743 DEBUG:boto:Method: GET 2023-10-17T18:53:35.744 DEBUG:boto:Path: /hbmuog-57/?prefix=d 2023-10-17T18:53:35.744 DEBUG:boto:Data: 2023-10-17T18:53:35.745 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.745 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.745 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.746 DEBUG:boto:Params: {} 2023-10-17T18:53:35.746 DEBUG:boto:Token: None 2023-10-17T18:53:35.747 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.747 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd9927d7cdac6fd1-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.750 DEBUG:boto:b'hbmuog-57d1000falsed2023-10-17T18:50:43.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:53:35.752 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.752 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.752 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.753 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.753 DEBUG:boto:Method: GET 2023-10-17T18:53:35.754 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.754 DEBUG:boto:Data: 2023-10-17T18:53:35.755 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.755 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.756 DEBUG:boto:Params: {} 2023-10-17T18:53:35.756 DEBUG:boto:Token: None 2023-10-17T18:53:35.757 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw= 2023-10-17T18:53:35.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw=', 'Content-Length': '0'} 2023-10-17T18:53:35.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84a70b2bd33ab80-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.761 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.761 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.762 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.762 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.763 DEBUG:boto:Method: GET 2023-10-17T18:53:35.763 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.764 DEBUG:boto:Data: 2023-10-17T18:53:35.764 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.764 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.765 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.765 DEBUG:boto:Params: {} 2023-10-17T18:53:35.766 DEBUG:boto:Token: None 2023-10-17T18:53:35.766 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw= 2023-10-17T18:53:35.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw=', 'Content-Length': '0'} 2023-10-17T18:53:35.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b1092774de9f5f-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.770 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.771 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.771 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.772 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.772 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.772 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.773 DEBUG:boto:Data: 2023-10-17T18:53:35.773 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.774 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.774 DEBUG:boto:Params: {} 2023-10-17T18:53:35.775 DEBUG:boto:Token: None 2023-10-17T18:53:35.775 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pnqSQGwthmhg3VCGAJGJO7308wQ= 2023-10-17T18:53:35.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnqSQGwthmhg3VCGAJGJO7308wQ=', 'Content-Length': '0'} 2023-10-17T18:53:35.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833233521cc81dd1-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.779 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.779 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.780 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.780 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.780 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.781 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.781 DEBUG:boto:Data: 2023-10-17T18:53:35.782 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.782 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.783 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.783 DEBUG:boto:Params: {} 2023-10-17T18:53:35.783 DEBUG:boto:Token: None 2023-10-17T18:53:35.784 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pnqSQGwthmhg3VCGAJGJO7308wQ= 2023-10-17T18:53:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnqSQGwthmhg3VCGAJGJO7308wQ=', 'Content-Length': '0'} 2023-10-17T18:53:35.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007545dbfca5c58d05-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:53:35.788 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.789 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.789 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.789 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.790 DEBUG:boto:Method: GET 2023-10-17T18:53:35.790 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.791 DEBUG:boto:Data: 2023-10-17T18:53:35.791 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.792 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.792 DEBUG:boto:Params: {} 2023-10-17T18:53:35.793 DEBUG:boto:Token: None 2023-10-17T18:53:35.793 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw= 2023-10-17T18:53:35.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw=', 'Content-Length': '0'} 2023-10-17T18:53:35.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fd7467fbd5e5ac-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.797 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.798 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.798 DEBUG:boto:path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.799 DEBUG:boto:auth_path=/hbmuog-57/e?versionId=null 2023-10-17T18:53:35.799 DEBUG:boto:Method: GET 2023-10-17T18:53:35.800 DEBUG:boto:Path: /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.800 DEBUG:boto:Data: 2023-10-17T18:53:35.800 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.801 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.802 DEBUG:boto:Params: {} 2023-10-17T18:53:35.802 DEBUG:boto:Token: None 2023-10-17T18:53:35.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e?versionId=null 2023-10-17T18:53:35.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw= 2023-10-17T18:53:35.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i9ze9CH2Yp8HmopJ8P3EKSKvpRw=', 'Content-Length': '0'} 2023-10-17T18:53:35.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004b8f11480d373eb-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.846 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.847 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.847 DEBUG:boto:path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.848 DEBUG:boto:auth_path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.848 DEBUG:boto:Method: GET 2023-10-17T18:53:35.848 DEBUG:boto:Path: /hbmuog-57/?prefix=e 2023-10-17T18:53:35.849 DEBUG:boto:Data: 2023-10-17T18:53:35.849 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.850 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.851 DEBUG:boto:Params: {} 2023-10-17T18:53:35.851 DEBUG:boto:Token: None 2023-10-17T18:53:35.851 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac2b269fd947fcc-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.856 DEBUG:boto:b'hbmuog-57e1000falsee2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.856 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.857 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.857 DEBUG:boto:path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.857 DEBUG:boto:auth_path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.858 DEBUG:boto:Method: GET 2023-10-17T18:53:35.858 DEBUG:boto:Path: /hbmuog-57/?prefix=e 2023-10-17T18:53:35.859 DEBUG:boto:Data: 2023-10-17T18:53:35.859 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.859 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.860 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.860 DEBUG:boto:Params: {} 2023-10-17T18:53:35.861 DEBUG:boto:Token: None 2023-10-17T18:53:35.861 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea8d94c44025d4e-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.864 DEBUG:boto:b'hbmuog-57e1000falsee2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.865 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.865 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.866 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.866 DEBUG:boto:Path: /hbmuog-57/e 2023-10-17T18:53:35.866 DEBUG:boto:Data: 2023-10-17T18:53:35.867 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.868 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.868 DEBUG:boto:Params: {} 2023-10-17T18:53:35.868 DEBUG:boto:Token: None 2023-10-17T18:53:35.869 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e 2023-10-17T18:53:35.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZoYR8MJva8f7f0IdQX0Hvt87GXo= 2023-10-17T18:53:35.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZoYR8MJva8f7f0IdQX0Hvt87GXo=', 'Content-Length': '0'} 2023-10-17T18:53:35.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c9b508edf3a147-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.872 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.873 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.873 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.873 DEBUG:boto:Path: /hbmuog-57/e 2023-10-17T18:53:35.874 DEBUG:boto:Data: 2023-10-17T18:53:35.874 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.874 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.875 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.875 DEBUG:boto:Params: {} 2023-10-17T18:53:35.876 DEBUG:boto:Token: None 2023-10-17T18:53:35.876 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e 2023-10-17T18:53:35.876 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZoYR8MJva8f7f0IdQX0Hvt87GXo= 2023-10-17T18:53:35.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZoYR8MJva8f7f0IdQX0Hvt87GXo=', 'Content-Length': '0'} 2023-10-17T18:53:35.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b8aa12c66c4e34-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-17T18:53:35.880 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.880 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.880 DEBUG:boto:Method: GET 2023-10-17T18:53:35.881 DEBUG:boto:Path: /hbmuog-57/e 2023-10-17T18:53:35.881 DEBUG:boto:Data: 2023-10-17T18:53:35.882 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.882 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.883 DEBUG:boto:Params: {} 2023-10-17T18:53:35.883 DEBUG:boto:Token: None 2023-10-17T18:53:35.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e 2023-10-17T18:53:35.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:obyHr6KLd0KtbPF+5s1mkiniwZY= 2023-10-17T18:53:35.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:obyHr6KLd0KtbPF+5s1mkiniwZY=', 'Content-Length': '0'} 2023-10-17T18:53:35.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d8829a276c3d2c-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.930 DEBUG:boto:path=/hbmuog-57/e 2023-10-17T18:53:35.931 DEBUG:boto:auth_path=/hbmuog-57/e 2023-10-17T18:53:35.931 DEBUG:boto:Method: GET 2023-10-17T18:53:35.932 DEBUG:boto:Path: /hbmuog-57/e 2023-10-17T18:53:35.932 DEBUG:boto:Data: 2023-10-17T18:53:35.932 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.933 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.933 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.933 DEBUG:boto:Params: {} 2023-10-17T18:53:35.934 DEBUG:boto:Token: None 2023-10-17T18:53:35.934 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/e 2023-10-17T18:53:35.935 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:obyHr6KLd0KtbPF+5s1mkiniwZY= 2023-10-17T18:53:35.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:obyHr6KLd0KtbPF+5s1mkiniwZY=', 'Content-Length': '0'} 2023-10-17T18:53:35.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdd75f0b95e873fb-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.938 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.938 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.939 DEBUG:boto:path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.939 DEBUG:boto:auth_path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.940 DEBUG:boto:Method: GET 2023-10-17T18:53:35.940 DEBUG:boto:Path: /hbmuog-57/?prefix=e 2023-10-17T18:53:35.940 DEBUG:boto:Data: 2023-10-17T18:53:35.941 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.941 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.941 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.942 DEBUG:boto:Params: {} 2023-10-17T18:53:35.942 DEBUG:boto:Token: None 2023-10-17T18:53:35.943 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.943 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b85294e58d35ab-00652ed82f-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.947 DEBUG:boto:b'hbmuog-57e1000falsee2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.948 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:35.948 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:35.948 DEBUG:boto:path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.949 DEBUG:boto:auth_path=/hbmuog-57/?prefix=e 2023-10-17T18:53:35.949 DEBUG:boto:Method: GET 2023-10-17T18:53:35.950 DEBUG:boto:Path: /hbmuog-57/?prefix=e 2023-10-17T18:53:35.950 DEBUG:boto:Data: 2023-10-17T18:53:35.950 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.951 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.951 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.952 DEBUG:boto:Params: {} 2023-10-17T18:53:35.952 DEBUG:boto:Token: None 2023-10-17T18:53:35.952 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/ 2023-10-17T18:53:35.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58= 2023-10-17T18:53:35.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BP41l7jyi3DtOGCcDAc/flovF58=', 'Content-Length': '0'} 2023-10-17T18:53:35.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cf79ce374dd19ce-00652ed82f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.956 DEBUG:boto:b'hbmuog-57e1000falsee2023-10-17T18:50:43.337Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:35.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:53:35.957 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:35.957 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:35.957 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.958 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.958 DEBUG:boto:Method: GET 2023-10-17T18:53:35.959 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.959 DEBUG:boto:Data: 2023-10-17T18:53:35.959 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.960 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.961 DEBUG:boto:Params: {} 2023-10-17T18:53:35.961 DEBUG:boto:Token: None 2023-10-17T18:53:35.962 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8= 2023-10-17T18:53:35.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8=', 'Content-Length': '0'} 2023-10-17T18:53:35.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a49552bc001bf5-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.967 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:35.968 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:35.968 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.968 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.969 DEBUG:boto:Method: GET 2023-10-17T18:53:35.969 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.970 DEBUG:boto:Data: 2023-10-17T18:53:35.970 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.971 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.971 DEBUG:boto:Params: {} 2023-10-17T18:53:35.972 DEBUG:boto:Token: None 2023-10-17T18:53:35.972 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8= 2023-10-17T18:53:35.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8=', 'Content-Length': '0'} 2023-10-17T18:53:35.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b97080202f625a-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.975 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:35.976 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:35.976 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.977 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.977 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.977 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.978 DEBUG:boto:Data: 2023-10-17T18:53:35.978 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.978 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.979 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.979 DEBUG:boto:Params: {} 2023-10-17T18:53:35.980 DEBUG:boto:Token: None 2023-10-17T18:53:35.980 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.981 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:491uq54MdiZUbuGTkqmDguPSp4I= 2023-10-17T18:53:35.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:491uq54MdiZUbuGTkqmDguPSp4I=', 'Content-Length': '0'} 2023-10-17T18:53:35.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228b7e1fdddd3a55-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.983 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:35.984 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:35.984 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.985 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.985 DEBUG:boto:Method: HEAD 2023-10-17T18:53:35.985 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.986 DEBUG:boto:Data: 2023-10-17T18:53:35.986 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:35.987 DEBUG:boto:Port: 8000 2023-10-17T18:53:35.987 DEBUG:boto:Params: {} 2023-10-17T18:53:35.988 DEBUG:boto:Token: None 2023-10-17T18:53:35.988 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:491uq54MdiZUbuGTkqmDguPSp4I= 2023-10-17T18:53:35.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:491uq54MdiZUbuGTkqmDguPSp4I=', 'Content-Length': '0'} 2023-10-17T18:53:35.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b60917cb45d10ce-00652ed82f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:35.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:53:35.992 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:35.992 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:35.992 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.993 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:35.993 DEBUG:boto:Method: GET 2023-10-17T18:53:35.994 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.994 DEBUG:boto:Data: 2023-10-17T18:53:35.994 DEBUG:boto:Headers: {} 2023-10-17T18:53:35.995 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:35.995 DEBUG:boto:Port: 8001 2023-10-17T18:53:35.996 DEBUG:boto:Params: {} 2023-10-17T18:53:35.996 DEBUG:boto:Token: None 2023-10-17T18:53:35.997 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:35 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:35.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8= 2023-10-17T18:53:35.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pQm8HKgtBR4WAUhluu1nPUVori8=', 'Content-Length': '0'} 2023-10-17T18:53:36.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d74a2723300f43-00652ed82f-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.042 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:36.043 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:36.043 DEBUG:boto:path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:36.043 DEBUG:boto:auth_path=/hbmuog-57/f?versionId=null 2023-10-17T18:53:36.044 DEBUG:boto:Method: GET 2023-10-17T18:53:36.044 DEBUG:boto:Path: /hbmuog-57/f?versionId=null 2023-10-17T18:53:36.045 DEBUG:boto:Data: 2023-10-17T18:53:36.045 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.046 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.046 DEBUG:boto:Params: {} 2023-10-17T18:53:36.047 DEBUG:boto:Token: None 2023-10-17T18:53:36.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/f?versionId=null 2023-10-17T18:53:36.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r31vT8NZ+xbzPycdls016ClSVQg= 2023-10-17T18:53:36.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r31vT8NZ+xbzPycdls016ClSVQg=', 'Content-Length': '0'} 2023-10-17T18:53:36.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30fbc6feffcdf4c-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.051 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.051 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.052 DEBUG:boto:path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.052 DEBUG:boto:auth_path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.052 DEBUG:boto:Method: GET 2023-10-17T18:53:36.053 DEBUG:boto:Path: /hbmuog-57/?prefix=f 2023-10-17T18:53:36.053 DEBUG:boto:Data: 2023-10-17T18:53:36.054 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.055 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.055 DEBUG:boto:Params: {} 2023-10-17T18:53:36.055 DEBUG:boto:Token: None 2023-10-17T18:53:36.056 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af20e54af4da6981-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.060 DEBUG:boto:b'hbmuog-57f1000falsef2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.060 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.061 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.061 DEBUG:boto:path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.061 DEBUG:boto:auth_path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.062 DEBUG:boto:Method: GET 2023-10-17T18:53:36.062 DEBUG:boto:Path: /hbmuog-57/?prefix=f 2023-10-17T18:53:36.063 DEBUG:boto:Data: 2023-10-17T18:53:36.063 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.063 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.064 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.064 DEBUG:boto:Params: {} 2023-10-17T18:53:36.065 DEBUG:boto:Token: None 2023-10-17T18:53:36.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bef6dbd185f162d-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.069 DEBUG:boto:b'hbmuog-57f1000falsef2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.069 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:36.070 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:36.070 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.070 DEBUG:boto:Path: /hbmuog-57/f 2023-10-17T18:53:36.071 DEBUG:boto:Data: 2023-10-17T18:53:36.071 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.072 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.072 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.072 DEBUG:boto:Params: {} 2023-10-17T18:53:36.073 DEBUG:boto:Token: None 2023-10-17T18:53:36.073 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/f 2023-10-17T18:53:36.074 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dO+HtGD/y1RVUgGpyA79P+OpQW4= 2023-10-17T18:53:36.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dO+HtGD/y1RVUgGpyA79P+OpQW4=', 'Content-Length': '0'} 2023-10-17T18:53:36.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad832532c3f45743-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.077 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:36.077 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:36.078 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.078 DEBUG:boto:Path: /hbmuog-57/f 2023-10-17T18:53:36.078 DEBUG:boto:Data: 2023-10-17T18:53:36.079 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.080 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.080 DEBUG:boto:Params: {} 2023-10-17T18:53:36.081 DEBUG:boto:Token: None 2023-10-17T18:53:36.081 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/f 2023-10-17T18:53:36.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dO+HtGD/y1RVUgGpyA79P+OpQW4= 2023-10-17T18:53:36.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dO+HtGD/y1RVUgGpyA79P+OpQW4=', 'Content-Length': '0'} 2023-10-17T18:53:36.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077620d8b2f480f86-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-17T18:53:36.085 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:36.085 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:36.085 DEBUG:boto:Method: GET 2023-10-17T18:53:36.086 DEBUG:boto:Path: /hbmuog-57/f 2023-10-17T18:53:36.086 DEBUG:boto:Data: 2023-10-17T18:53:36.087 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.087 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.087 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.088 DEBUG:boto:Params: {} 2023-10-17T18:53:36.088 DEBUG:boto:Token: None 2023-10-17T18:53:36.089 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/f 2023-10-17T18:53:36.089 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VAYFppm6C8bVlanGvjLZp1p1ID0= 2023-10-17T18:53:36.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VAYFppm6C8bVlanGvjLZp1p1ID0=', 'Content-Length': '0'} 2023-10-17T18:53:36.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb9db4fdcf6b9a9-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.092 DEBUG:boto:path=/hbmuog-57/f 2023-10-17T18:53:36.093 DEBUG:boto:auth_path=/hbmuog-57/f 2023-10-17T18:53:36.093 DEBUG:boto:Method: GET 2023-10-17T18:53:36.094 DEBUG:boto:Path: /hbmuog-57/f 2023-10-17T18:53:36.094 DEBUG:boto:Data: 2023-10-17T18:53:36.094 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.095 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.095 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.096 DEBUG:boto:Params: {} 2023-10-17T18:53:36.096 DEBUG:boto:Token: None 2023-10-17T18:53:36.097 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/f 2023-10-17T18:53:36.097 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VAYFppm6C8bVlanGvjLZp1p1ID0= 2023-10-17T18:53:36.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VAYFppm6C8bVlanGvjLZp1p1ID0=', 'Content-Length': '0'} 2023-10-17T18:53:36.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4473db9913190d2-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.100 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.101 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.101 DEBUG:boto:path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.102 DEBUG:boto:auth_path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.102 DEBUG:boto:Method: GET 2023-10-17T18:53:36.102 DEBUG:boto:Path: /hbmuog-57/?prefix=f 2023-10-17T18:53:36.103 DEBUG:boto:Data: 2023-10-17T18:53:36.103 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.104 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.104 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.104 DEBUG:boto:Params: {} 2023-10-17T18:53:36.105 DEBUG:boto:Token: None 2023-10-17T18:53:36.105 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.106 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ead6a9d63af95ce0-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.109 DEBUG:boto:b'hbmuog-57f1000falsef2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.110 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.110 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.111 DEBUG:boto:path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.111 DEBUG:boto:auth_path=/hbmuog-57/?prefix=f 2023-10-17T18:53:36.112 DEBUG:boto:Method: GET 2023-10-17T18:53:36.112 DEBUG:boto:Path: /hbmuog-57/?prefix=f 2023-10-17T18:53:36.112 DEBUG:boto:Data: 2023-10-17T18:53:36.113 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.114 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.114 DEBUG:boto:Params: {} 2023-10-17T18:53:36.115 DEBUG:boto:Token: None 2023-10-17T18:53:36.115 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.116 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7385b1b8cecd5b2-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.119 DEBUG:boto:b'hbmuog-57f1000falsef2023-10-17T18:50:43.356Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:53:36.120 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.120 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.121 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.121 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.121 DEBUG:boto:Method: GET 2023-10-17T18:53:36.122 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.122 DEBUG:boto:Data: 2023-10-17T18:53:36.123 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.123 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.124 DEBUG:boto:Params: {} 2023-10-17T18:53:36.124 DEBUG:boto:Token: None 2023-10-17T18:53:36.125 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0= 2023-10-17T18:53:36.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0=', 'Content-Length': '0'} 2023-10-17T18:53:36.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f46eed0b6c1539-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.130 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.130 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.130 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.131 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.131 DEBUG:boto:Method: GET 2023-10-17T18:53:36.132 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.132 DEBUG:boto:Data: 2023-10-17T18:53:36.132 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.133 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.133 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.134 DEBUG:boto:Params: {} 2023-10-17T18:53:36.134 DEBUG:boto:Token: None 2023-10-17T18:53:36.135 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.135 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0= 2023-10-17T18:53:36.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0=', 'Content-Length': '0'} 2023-10-17T18:53:36.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0354607323f9b8c-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.138 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.139 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.139 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.139 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.140 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.140 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.141 DEBUG:boto:Data: 2023-10-17T18:53:36.141 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.141 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.142 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.142 DEBUG:boto:Params: {} 2023-10-17T18:53:36.143 DEBUG:boto:Token: None 2023-10-17T18:53:36.143 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.143 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bNeipI84rVMylTb7YUpf4lRsLDo= 2023-10-17T18:53:36.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bNeipI84rVMylTb7YUpf4lRsLDo=', 'Content-Length': '0'} 2023-10-17T18:53:36.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae102a2896499c0-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.147 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.148 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.148 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.148 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.149 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.149 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.149 DEBUG:boto:Data: 2023-10-17T18:53:36.150 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.150 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.151 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.151 DEBUG:boto:Params: {} 2023-10-17T18:53:36.152 DEBUG:boto:Token: None 2023-10-17T18:53:36.152 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.153 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bNeipI84rVMylTb7YUpf4lRsLDo= 2023-10-17T18:53:36.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bNeipI84rVMylTb7YUpf4lRsLDo=', 'Content-Length': '0'} 2023-10-17T18:53:36.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf8dc5822b37913-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:53:36.156 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.157 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.157 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.157 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.158 DEBUG:boto:Method: GET 2023-10-17T18:53:36.158 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.159 DEBUG:boto:Data: 2023-10-17T18:53:36.159 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.160 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.161 DEBUG:boto:Params: {} 2023-10-17T18:53:36.161 DEBUG:boto:Token: None 2023-10-17T18:53:36.162 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.162 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0= 2023-10-17T18:53:36.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0=', 'Content-Length': '0'} 2023-10-17T18:53:36.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ec6debf09b5baf-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.165 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.166 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.166 DEBUG:boto:path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.166 DEBUG:boto:auth_path=/hbmuog-57/g?versionId=null 2023-10-17T18:53:36.167 DEBUG:boto:Method: GET 2023-10-17T18:53:36.167 DEBUG:boto:Path: /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.169 DEBUG:boto:Data: 2023-10-17T18:53:36.169 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.170 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.171 DEBUG:boto:Params: {} 2023-10-17T18:53:36.171 DEBUG:boto:Token: None 2023-10-17T18:53:36.172 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g?versionId=null 2023-10-17T18:53:36.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0= 2023-10-17T18:53:36.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OzPo6M4ql937/UcCBBqcr1Rmnc0=', 'Content-Length': '0'} 2023-10-17T18:53:36.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046800d0367addc6c-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.218 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.219 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.219 DEBUG:boto:path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.220 DEBUG:boto:auth_path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.220 DEBUG:boto:Method: GET 2023-10-17T18:53:36.221 DEBUG:boto:Path: /hbmuog-57/?prefix=g 2023-10-17T18:53:36.221 DEBUG:boto:Data: 2023-10-17T18:53:36.222 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.223 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.223 DEBUG:boto:Params: {} 2023-10-17T18:53:36.224 DEBUG:boto:Token: None 2023-10-17T18:53:36.224 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf88ce5bbf6400d-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.228 DEBUG:boto:b'hbmuog-57g1000falseg2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.229 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.229 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.230 DEBUG:boto:path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.230 DEBUG:boto:auth_path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.231 DEBUG:boto:Method: GET 2023-10-17T18:53:36.231 DEBUG:boto:Path: /hbmuog-57/?prefix=g 2023-10-17T18:53:36.232 DEBUG:boto:Data: 2023-10-17T18:53:36.232 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.233 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.233 DEBUG:boto:Params: {} 2023-10-17T18:53:36.234 DEBUG:boto:Token: None 2023-10-17T18:53:36.234 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1498f4b239a33d7-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.238 DEBUG:boto:b'hbmuog-57g1000falseg2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.239 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.239 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.240 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.240 DEBUG:boto:Path: /hbmuog-57/g 2023-10-17T18:53:36.240 DEBUG:boto:Data: 2023-10-17T18:53:36.241 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.242 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.242 DEBUG:boto:Params: {} 2023-10-17T18:53:36.243 DEBUG:boto:Token: None 2023-10-17T18:53:36.243 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g 2023-10-17T18:53:36.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSkbALJ/QiwmaE50WQKktWdikrc= 2023-10-17T18:53:36.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSkbALJ/QiwmaE50WQKktWdikrc=', 'Content-Length': '0'} 2023-10-17T18:53:36.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c18acd9dfaa77c-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.246 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.247 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.247 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.248 DEBUG:boto:Path: /hbmuog-57/g 2023-10-17T18:53:36.248 DEBUG:boto:Data: 2023-10-17T18:53:36.248 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.249 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.250 DEBUG:boto:Params: {} 2023-10-17T18:53:36.250 DEBUG:boto:Token: None 2023-10-17T18:53:36.251 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g 2023-10-17T18:53:36.251 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cSkbALJ/QiwmaE50WQKktWdikrc= 2023-10-17T18:53:36.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cSkbALJ/QiwmaE50WQKktWdikrc=', 'Content-Length': '0'} 2023-10-17T18:53:36.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b104e2f5ab8ffe3a-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-17T18:53:36.255 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.255 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.256 DEBUG:boto:Method: GET 2023-10-17T18:53:36.256 DEBUG:boto:Path: /hbmuog-57/g 2023-10-17T18:53:36.256 DEBUG:boto:Data: 2023-10-17T18:53:36.257 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.258 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.258 DEBUG:boto:Params: {} 2023-10-17T18:53:36.258 DEBUG:boto:Token: None 2023-10-17T18:53:36.259 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g 2023-10-17T18:53:36.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7SDJQSYyh8JFvDXNiDQVQCg07hg= 2023-10-17T18:53:36.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7SDJQSYyh8JFvDXNiDQVQCg07hg=', 'Content-Length': '0'} 2023-10-17T18:53:36.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f9949db9322bf7-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.306 DEBUG:boto:path=/hbmuog-57/g 2023-10-17T18:53:36.307 DEBUG:boto:auth_path=/hbmuog-57/g 2023-10-17T18:53:36.307 DEBUG:boto:Method: GET 2023-10-17T18:53:36.308 DEBUG:boto:Path: /hbmuog-57/g 2023-10-17T18:53:36.308 DEBUG:boto:Data: 2023-10-17T18:53:36.309 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.309 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.310 DEBUG:boto:Params: {} 2023-10-17T18:53:36.310 DEBUG:boto:Token: None 2023-10-17T18:53:36.311 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/g 2023-10-17T18:53:36.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7SDJQSYyh8JFvDXNiDQVQCg07hg= 2023-10-17T18:53:36.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7SDJQSYyh8JFvDXNiDQVQCg07hg=', 'Content-Length': '0'} 2023-10-17T18:53:36.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ddad0dd7a241593-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.315 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.315 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.316 DEBUG:boto:path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.316 DEBUG:boto:auth_path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.316 DEBUG:boto:Method: GET 2023-10-17T18:53:36.317 DEBUG:boto:Path: /hbmuog-57/?prefix=g 2023-10-17T18:53:36.317 DEBUG:boto:Data: 2023-10-17T18:53:36.318 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.318 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.319 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.319 DEBUG:boto:Params: {} 2023-10-17T18:53:36.319 DEBUG:boto:Token: None 2023-10-17T18:53:36.320 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b632e20b346401e7-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.324 DEBUG:boto:b'hbmuog-57g1000falseg2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.325 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.325 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.325 DEBUG:boto:path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.326 DEBUG:boto:auth_path=/hbmuog-57/?prefix=g 2023-10-17T18:53:36.326 DEBUG:boto:Method: GET 2023-10-17T18:53:36.327 DEBUG:boto:Path: /hbmuog-57/?prefix=g 2023-10-17T18:53:36.327 DEBUG:boto:Data: 2023-10-17T18:53:36.328 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.328 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.328 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.329 DEBUG:boto:Params: {} 2023-10-17T18:53:36.329 DEBUG:boto:Token: None 2023-10-17T18:53:36.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.330 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ee52409cfaf42e2-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.334 DEBUG:boto:b'hbmuog-57g1000falseg2023-10-17T18:50:43.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:53:36.335 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.335 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.336 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.336 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.336 DEBUG:boto:Method: GET 2023-10-17T18:53:36.337 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.338 DEBUG:boto:Data: 2023-10-17T18:53:36.338 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.339 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.339 DEBUG:boto:Params: {} 2023-10-17T18:53:36.340 DEBUG:boto:Token: None 2023-10-17T18:53:36.340 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU= 2023-10-17T18:53:36.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU=', 'Content-Length': '0'} 2023-10-17T18:53:36.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031805a93fd74a0da-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.344 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.345 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.345 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.345 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.346 DEBUG:boto:Method: GET 2023-10-17T18:53:36.346 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.347 DEBUG:boto:Data: 2023-10-17T18:53:36.347 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.348 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.348 DEBUG:boto:Params: {} 2023-10-17T18:53:36.349 DEBUG:boto:Token: None 2023-10-17T18:53:36.349 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU= 2023-10-17T18:53:36.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU=', 'Content-Length': '0'} 2023-10-17T18:53:36.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091af1acd3757073b-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.353 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.354 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.354 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.355 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.355 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.355 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.356 DEBUG:boto:Data: 2023-10-17T18:53:36.356 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.357 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.357 DEBUG:boto:Params: {} 2023-10-17T18:53:36.358 DEBUG:boto:Token: None 2023-10-17T18:53:36.358 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BfqrjALmO9eTTUe9PBJXL3XYZew= 2023-10-17T18:53:36.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BfqrjALmO9eTTUe9PBJXL3XYZew=', 'Content-Length': '0'} 2023-10-17T18:53:36.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aa0011172ce9d4b-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.362 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.362 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.363 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.363 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.363 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.364 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.364 DEBUG:boto:Data: 2023-10-17T18:53:36.365 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.365 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.366 DEBUG:boto:Params: {} 2023-10-17T18:53:36.366 DEBUG:boto:Token: None 2023-10-17T18:53:36.367 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BfqrjALmO9eTTUe9PBJXL3XYZew= 2023-10-17T18:53:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BfqrjALmO9eTTUe9PBJXL3XYZew=', 'Content-Length': '0'} 2023-10-17T18:53:36.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c041e4754b10e35-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:53:36.371 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.373 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.374 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.374 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.375 DEBUG:boto:Method: GET 2023-10-17T18:53:36.375 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.375 DEBUG:boto:Data: 2023-10-17T18:53:36.376 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.376 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.377 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.377 DEBUG:boto:Params: {} 2023-10-17T18:53:36.378 DEBUG:boto:Token: None 2023-10-17T18:53:36.378 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.379 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU= 2023-10-17T18:53:36.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU=', 'Content-Length': '0'} 2023-10-17T18:53:36.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089757756d80b2ed-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.381 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.382 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.382 DEBUG:boto:path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.383 DEBUG:boto:auth_path=/hbmuog-57/h?versionId=null 2023-10-17T18:53:36.383 DEBUG:boto:Method: GET 2023-10-17T18:53:36.384 DEBUG:boto:Path: /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.384 DEBUG:boto:Data: 2023-10-17T18:53:36.384 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.385 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.385 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.386 DEBUG:boto:Params: {} 2023-10-17T18:53:36.386 DEBUG:boto:Token: None 2023-10-17T18:53:36.387 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h?versionId=null 2023-10-17T18:53:36.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU= 2023-10-17T18:53:36.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+3HKd/J6WWaeMUdQY1mnOqqROEU=', 'Content-Length': '0'} 2023-10-17T18:53:36.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5783f5b1d313c58-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.391 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.391 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.391 DEBUG:boto:path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.392 DEBUG:boto:auth_path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.392 DEBUG:boto:Method: GET 2023-10-17T18:53:36.393 DEBUG:boto:Path: /hbmuog-57/?prefix=h 2023-10-17T18:53:36.393 DEBUG:boto:Data: 2023-10-17T18:53:36.394 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.395 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.395 DEBUG:boto:Params: {} 2023-10-17T18:53:36.395 DEBUG:boto:Token: None 2023-10-17T18:53:36.396 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4118d59a86b01f9-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.400 DEBUG:boto:b'hbmuog-57h1000falseh2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.401 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.401 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.402 DEBUG:boto:path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.402 DEBUG:boto:auth_path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.403 DEBUG:boto:Method: GET 2023-10-17T18:53:36.403 DEBUG:boto:Path: /hbmuog-57/?prefix=h 2023-10-17T18:53:36.404 DEBUG:boto:Data: 2023-10-17T18:53:36.404 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.404 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.405 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.405 DEBUG:boto:Params: {} 2023-10-17T18:53:36.406 DEBUG:boto:Token: None 2023-10-17T18:53:36.406 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.407 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec986c8663cf40e-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.410 DEBUG:boto:b'hbmuog-57h1000falseh2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.411 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.411 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.412 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.412 DEBUG:boto:Path: /hbmuog-57/h 2023-10-17T18:53:36.412 DEBUG:boto:Data: 2023-10-17T18:53:36.413 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.414 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.414 DEBUG:boto:Params: {} 2023-10-17T18:53:36.414 DEBUG:boto:Token: None 2023-10-17T18:53:36.415 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h 2023-10-17T18:53:36.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HOTf5koJMDCvKH2q1bfuiRPWhkM= 2023-10-17T18:53:36.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HOTf5koJMDCvKH2q1bfuiRPWhkM=', 'Content-Length': '0'} 2023-10-17T18:53:36.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90948073c8d73a8-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.418 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.419 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.419 DEBUG:boto:Method: HEAD 2023-10-17T18:53:36.420 DEBUG:boto:Path: /hbmuog-57/h 2023-10-17T18:53:36.420 DEBUG:boto:Data: 2023-10-17T18:53:36.421 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.421 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.421 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.422 DEBUG:boto:Params: {} 2023-10-17T18:53:36.422 DEBUG:boto:Token: None 2023-10-17T18:53:36.423 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h 2023-10-17T18:53:36.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HOTf5koJMDCvKH2q1bfuiRPWhkM= 2023-10-17T18:53:36.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HOTf5koJMDCvKH2q1bfuiRPWhkM=', 'Content-Length': '0'} 2023-10-17T18:53:36.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093738ab5bf639d66-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-17T18:53:36.426 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.427 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.427 DEBUG:boto:Method: GET 2023-10-17T18:53:36.428 DEBUG:boto:Path: /hbmuog-57/h 2023-10-17T18:53:36.428 DEBUG:boto:Data: 2023-10-17T18:53:36.429 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.429 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.430 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.430 DEBUG:boto:Params: {} 2023-10-17T18:53:36.430 DEBUG:boto:Token: None 2023-10-17T18:53:36.431 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h 2023-10-17T18:53:36.431 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1++NqsBhZ72hTxFbgig/LD4Hit0= 2023-10-17T18:53:36.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1++NqsBhZ72hTxFbgig/LD4Hit0=', 'Content-Length': '0'} 2023-10-17T18:53:36.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-57:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028919d5c3d9e3d80-00652ed830-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.478 DEBUG:boto:path=/hbmuog-57/h 2023-10-17T18:53:36.479 DEBUG:boto:auth_path=/hbmuog-57/h 2023-10-17T18:53:36.480 DEBUG:boto:Method: GET 2023-10-17T18:53:36.480 DEBUG:boto:Path: /hbmuog-57/h 2023-10-17T18:53:36.480 DEBUG:boto:Data: 2023-10-17T18:53:36.481 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.481 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.482 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.482 DEBUG:boto:Params: {} 2023-10-17T18:53:36.483 DEBUG:boto:Token: None 2023-10-17T18:53:36.483 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/h 2023-10-17T18:53:36.484 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1++NqsBhZ72hTxFbgig/LD4Hit0= 2023-10-17T18:53:36.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1++NqsBhZ72hTxFbgig/LD4Hit0=', 'Content-Length': '0'} 2023-10-17T18:53:36.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219b116410b9d8ac-00652ed830-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.487 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.488 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.488 DEBUG:boto:path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.488 DEBUG:boto:auth_path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.489 DEBUG:boto:Method: GET 2023-10-17T18:53:36.489 DEBUG:boto:Path: /hbmuog-57/?prefix=h 2023-10-17T18:53:36.490 DEBUG:boto:Data: 2023-10-17T18:53:36.490 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.491 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.491 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.491 DEBUG:boto:Params: {} 2023-10-17T18:53:36.492 DEBUG:boto:Token: None 2023-10-17T18:53:36.492 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.493 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682c3e7b40c4b3a6-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.497 DEBUG:boto:b'hbmuog-57h1000falseh2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.497 DEBUG:boto:path=/hbmuog-57/ 2023-10-17T18:53:36.498 DEBUG:boto:auth_path=/hbmuog-57/ 2023-10-17T18:53:36.498 DEBUG:boto:path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.499 DEBUG:boto:auth_path=/hbmuog-57/?prefix=h 2023-10-17T18:53:36.499 DEBUG:boto:Method: GET 2023-10-17T18:53:36.500 DEBUG:boto:Path: /hbmuog-57/?prefix=h 2023-10-17T18:53:36.500 DEBUG:boto:Data: 2023-10-17T18:53:36.500 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.501 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.502 DEBUG:boto:Params: {} 2023-10-17T18:53:36.502 DEBUG:boto:Token: None 2023-10-17T18:53:36.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-57/ 2023-10-17T18:53:36.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA= 2023-10-17T18:53:36.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lCMZDNW6dg89lVPMG5CCvWH3XzA=', 'Content-Length': '0'} 2023-10-17T18:53:36.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b484358a55a01fe3-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.546 DEBUG:boto:b'hbmuog-57h1000falseh2023-10-17T18:50:43.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:36.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-57 zones={a2, a1} 2023-10-17T18:53:36.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-10-17T18:53:36.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.550 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:36.606 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:36.639 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:53:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:53:36.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.652 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568643.290608_686.1", 2023-10-17T18:53:36.653 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:50:43.290608Z" 2023-10-17T18:53:36.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.653 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:53:36.654 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:53:36.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.654 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.655 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:53:36.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.655 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.656 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:53:36.656 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.656 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:53:36.657 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568643.290608_686.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:36.658 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:53:36.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.659 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:36.711 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:36.711 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:36.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:36.740+0000 7fa16a97b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:36.741+0000 7fa16a97b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:36.755 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:53:36.755 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:53:36.755 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:53:36.756 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:53:36.756 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:53:36.756 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:53:36.756 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:36.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.757 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:53:36.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.757 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:53:36.757 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:36.758 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:36.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568643.290608_686.1", 2023-10-17T18:53:36.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:36.758 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:53:36.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:36.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:50:43.290608Z", 2023-10-17T18:53:36.759 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:36.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:53:36.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:36.760 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:36.761 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:53:36.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:36.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:36.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:36.762 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:53:36.762 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:36.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.763 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:53:36.763 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:36.763 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:36.763 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.763 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:36.764 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:36.764 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:36.764 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:36.764 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:36.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.765 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:53:36.765 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:36.766 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:36.766 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.766 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:36.766 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:36.766 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:36.767 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:36.767 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:36.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.768 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:53:36.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.768 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:53:36.769 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:53:36.769 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:53:36.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.770 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:53:36Z" 2023-10-17T18:53:36.770 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:53:36.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568643.290608_686.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:50:43.290608Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:53:36Z" } 2023-10-17T18:53:36.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568643.290608_686.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:50:43.290608Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:53:36.772 DEBUG:rgw_multi.tests:log_status={0: '1_1697568643.290608_686.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:36.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568643.290608_686.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:36.773 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:53:36.773 DEBUG:boto:path=/ 2023-10-17T18:53:36.774 DEBUG:boto:auth_path=/ 2023-10-17T18:53:36.774 DEBUG:boto:Method: GET 2023-10-17T18:53:36.774 DEBUG:boto:Path: / 2023-10-17T18:53:36.775 DEBUG:boto:Data: 2023-10-17T18:53:36.775 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.776 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.776 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.776 DEBUG:boto:Params: {} 2023-10-17T18:53:36.777 DEBUG:boto:Token: None 2023-10-17T18:53:36.777 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT / 2023-10-17T18:53:36.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ONqBVUi2mtN9AeMGUGYhj91ePYc= 2023-10-17T18:53:36.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ONqBVUi2mtN9AeMGUGYhj91ePYc=', 'Content-Length': '0'} 2023-10-17T18:53:36.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c3a5ec7c7607ee3-00652ed830-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.783 DEBUG:boto:path=/ 2023-10-17T18:53:36.784 DEBUG:boto:auth_path=/ 2023-10-17T18:53:36.784 DEBUG:boto:Method: GET 2023-10-17T18:53:36.785 DEBUG:boto:Path: / 2023-10-17T18:53:36.785 DEBUG:boto:Data: 2023-10-17T18:53:36.786 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.786 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:36.786 DEBUG:boto:Port: 8001 2023-10-17T18:53:36.787 DEBUG:boto:Params: {} 2023-10-17T18:53:36.787 DEBUG:boto:Token: None 2023-10-17T18:53:36.788 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:36 GMT / 2023-10-17T18:53:36.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ONqBVUi2mtN9AeMGUGYhj91ePYc= 2023-10-17T18:53:36.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ONqBVUi2mtN9AeMGUGYhj91ePYc=', 'Content-Length': '0'} 2023-10-17T18:53:36.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b433cb719816a956-00652ed830-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-10-17T18:53:36.793 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:36.794 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:36.794 DEBUG:boto:Method: PUT 2023-10-17T18:53:36.795 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:36.795 DEBUG:boto:Data: 2023-10-17T18:53:36.796 DEBUG:boto:Headers: {} 2023-10-17T18:53:36.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:36.796 DEBUG:boto:Port: 8000 2023-10-17T18:53:36.797 DEBUG:boto:Params: {} 2023-10-17T18:53:36.797 DEBUG:boto:Token: None 2023-10-17T18:53:36.798 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:53:36 GMT /hbmuog-58/ 2023-10-17T18:53:36.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rUlEI+g9XqmLidwjb4M55s+QeKU= 2023-10-17T18:53:36.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rUlEI+g9XqmLidwjb4M55s+QeKU=', 'Content-Length': '0'} 2023-10-17T18:53:36.813 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e60e21973b49b8c2-00652ed830-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:36.814 DEBUG:rgw_multi.tests:created bucket=hbmuog-58 2023-10-17T18:53:36.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.815 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:36.865 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:36.866 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:36.897 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:53:36.910 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:53:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:53:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:53:36.812003Z" 2023-10-17T18:53:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:53:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:53:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:53:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:53:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:36.914 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:53:36.915 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:36.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:53:36.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:36.917 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:36.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:36.968 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:36.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:36.996+0000 7f6cb5869780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:36.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:36.997+0000 7f6cb5869780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:53:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:53:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:53:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:53:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:53:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:53:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:53:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:53:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568643.290608_686.1", 2023-10-17T18:53:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:53:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:50:43.290608Z", 2023-10-17T18:53:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:53:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:53:37.018 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:37.018 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:37.018 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:37.018 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:53:37.019 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:37.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:37.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:37.020 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:53:37.020 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:37.020 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:37.020 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:37.020 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:37.021 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:37.021 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:37.021 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:37.021 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:37.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:37.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:37.022 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:53:37.023 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:37.023 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:37.023 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:37.023 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:37.023 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:37.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:37.024 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:37.024 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:37.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:37.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:37.025 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:53:37.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.025 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:53:37.026 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:53:37.026 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:53:37.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:37.027 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:53:36Z" 2023-10-17T18:53:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:53:37.028 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568643.290608_686.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:50:43.290608Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:53:36Z" } 2023-10-17T18:53:37.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568643.290608_686.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:50:43.290608Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:53:37.029 DEBUG:rgw_multi.tests:log_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:37.029 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568643.290608_686.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:37.030 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568816.812003_700.1 target=1_1697568643.290608_686.1 2023-10-17T18:53:42.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:42.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:42.032 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:42.092 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:42.124+0000 7f4d4a9af780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:42.125+0000 7f4d4a9af780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:53:42.140 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:53:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:53:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:53:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:53:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:53:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:53:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:53:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:53:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:53:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:53:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:53:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:53:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:53:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:53:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:53:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:53:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:53:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:53:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:53:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:53:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:53:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:53:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:53:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:53:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:53:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:53:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:53:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:53:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:53:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:53:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:53:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:53:42Z" 2023-10-17T18:53:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:53:42.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:53:42Z" } 2023-10-17T18:53:42.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:53:42.159 DEBUG:rgw_multi.tests:log_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:42.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:53:42.160 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:53:42.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:42.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:53:42.163 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:53:42.218 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:42.218 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:53:42.260 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:53:42.260 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:53:42.261 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000005", 2023-10-17T18:53:42.261 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:53:42.261 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:53:42.261 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:53:42.262 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:53:42.262 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:53:42.262 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:53:42.262 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000013", 2023-10-17T18:53:42.263 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:53:42.263 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006" 2023-10-17T18:53:42.263 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:53:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:53:42.265 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.265 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.266 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.266 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.266 DEBUG:boto:Data: 2023-10-17T18:53:42.267 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.268 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.268 DEBUG:boto:Params: {} 2023-10-17T18:53:42.269 DEBUG:boto:Token: None 2023-10-17T18:53:42.270 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.270 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dbe3eac5310b31c-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.274 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:42.274 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:42.275 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.275 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:42.276 DEBUG:boto:Data: 2023-10-17T18:53:42.276 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.276 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.277 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.277 DEBUG:boto:Params: {} 2023-10-17T18:53:42.278 DEBUG:boto:Token: None 2023-10-17T18:53:42.278 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:42.279 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gtKfBzgKVzkZHC4GnjAyevT1qx4= 2023-10-17T18:53:42.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gtKfBzgKVzkZHC4GnjAyevT1qx4='} 2023-10-17T18:53:42.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8266866a28a50fa-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.286 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.286 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.286 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.287 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.287 DEBUG:boto:Data: 2023-10-17T18:53:42.287 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.288 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.288 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.289 DEBUG:boto:Params: {} 2023-10-17T18:53:42.289 DEBUG:boto:Token: None 2023-10-17T18:53:42.289 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.290 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6eda1249a299e96-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.293 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:42.293 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:42.294 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.294 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:42.295 DEBUG:boto:Data: 2023-10-17T18:53:42.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.296 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.296 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.297 DEBUG:boto:Params: {} 2023-10-17T18:53:42.297 DEBUG:boto:Token: None 2023-10-17T18:53:42.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:42.298 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Fs5E7pEdmJOiRn1jVF9he55vR+0= 2023-10-17T18:53:42.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Fs5E7pEdmJOiRn1jVF9he55vR+0='} 2023-10-17T18:53:42.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f8b96a46c700aa5-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.304 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.304 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.304 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.305 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.305 DEBUG:boto:Data: 2023-10-17T18:53:42.306 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.306 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.310 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.311 DEBUG:boto:Params: {} 2023-10-17T18:53:42.311 DEBUG:boto:Token: None 2023-10-17T18:53:42.312 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064730bdf68563abc-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.315 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:42.316 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:42.316 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.317 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:42.317 DEBUG:boto:Data: 2023-10-17T18:53:42.318 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.318 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.319 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.319 DEBUG:boto:Params: {} 2023-10-17T18:53:42.320 DEBUG:boto:Token: None 2023-10-17T18:53:42.320 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:42.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eV6Aa4/tPrU9nwl7LN1qH0+ppgE= 2023-10-17T18:53:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eV6Aa4/tPrU9nwl7LN1qH0+ppgE='} 2023-10-17T18:53:42.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f88cb5fed9d1e9cb-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.367 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.368 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.368 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.368 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.369 DEBUG:boto:Data: 2023-10-17T18:53:42.369 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.370 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.371 DEBUG:boto:Params: {} 2023-10-17T18:53:42.371 DEBUG:boto:Token: None 2023-10-17T18:53:42.372 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cbc7ae821629af6-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.376 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:42.376 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:42.377 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.377 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:42.377 DEBUG:boto:Data: 2023-10-17T18:53:42.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.378 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.379 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.379 DEBUG:boto:Params: {} 2023-10-17T18:53:42.380 DEBUG:boto:Token: None 2023-10-17T18:53:42.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:42.381 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ts2GyxwU0J+R8JLq4GuLAtF/zDM= 2023-10-17T18:53:42.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts2GyxwU0J+R8JLq4GuLAtF/zDM='} 2023-10-17T18:53:42.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c231dd01a6abb78-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.387 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.387 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.388 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.388 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.388 DEBUG:boto:Data: 2023-10-17T18:53:42.389 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.389 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.390 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.390 DEBUG:boto:Params: {} 2023-10-17T18:53:42.391 DEBUG:boto:Token: None 2023-10-17T18:53:42.391 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f3c306e342fa1aa-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.395 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:42.395 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:42.395 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.396 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:42.396 DEBUG:boto:Data: 2023-10-17T18:53:42.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.398 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.398 DEBUG:boto:Params: {} 2023-10-17T18:53:42.398 DEBUG:boto:Token: None 2023-10-17T18:53:42.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:42.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3P2nLWKXh3gJ4M6rhPhP/zG1830= 2023-10-17T18:53:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3P2nLWKXh3gJ4M6rhPhP/zG1830='} 2023-10-17T18:53:42.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000502bdb959e0c528b-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.447 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.448 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.448 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.448 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.449 DEBUG:boto:Data: 2023-10-17T18:53:42.449 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.450 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.450 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.451 DEBUG:boto:Params: {} 2023-10-17T18:53:42.451 DEBUG:boto:Token: None 2023-10-17T18:53:42.452 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.452 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d2c1fa462f3a8ce-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.456 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:42.456 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:42.457 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.457 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:42.457 DEBUG:boto:Data: 2023-10-17T18:53:42.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.459 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.459 DEBUG:boto:Params: {} 2023-10-17T18:53:42.459 DEBUG:boto:Token: None 2023-10-17T18:53:42.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:42.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ywfi9fIdNSfv6qdT6Bdi6CksQEY= 2023-10-17T18:53:42.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ywfi9fIdNSfv6qdT6Bdi6CksQEY='} 2023-10-17T18:53:42.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002910003d8ec037b1-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.466 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.467 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.467 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.468 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.468 DEBUG:boto:Data: 2023-10-17T18:53:42.468 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.469 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.470 DEBUG:boto:Params: {} 2023-10-17T18:53:42.470 DEBUG:boto:Token: None 2023-10-17T18:53:42.471 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.472 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005181035d94052bb9-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.475 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:42.475 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:42.475 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.476 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:42.476 DEBUG:boto:Data: 2023-10-17T18:53:42.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.478 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.478 DEBUG:boto:Params: {} 2023-10-17T18:53:42.478 DEBUG:boto:Token: None 2023-10-17T18:53:42.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:42.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m4TmIGgQV8Jmyzs5Znq6FI9QvXQ= 2023-10-17T18:53:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m4TmIGgQV8Jmyzs5Znq6FI9QvXQ='} 2023-10-17T18:53:42.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eea7d78cdfd80ed-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.523 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.524 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.524 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.525 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.525 DEBUG:boto:Data: 2023-10-17T18:53:42.525 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.526 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.526 DEBUG:boto:Params: {} 2023-10-17T18:53:42.527 DEBUG:boto:Token: None 2023-10-17T18:53:42.527 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4bc0d3ef9600df-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.532 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:42.532 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:42.533 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.534 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:42.534 DEBUG:boto:Data: 2023-10-17T18:53:42.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.535 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.535 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.536 DEBUG:boto:Params: {} 2023-10-17T18:53:42.536 DEBUG:boto:Token: None 2023-10-17T18:53:42.537 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:42.537 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+qTGcJbTJSFcOK8JWr1uocmdF8A= 2023-10-17T18:53:42.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+qTGcJbTJSFcOK8JWr1uocmdF8A='} 2023-10-17T18:53:42.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f8e9453151dfca8-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.543 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.544 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.544 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.545 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.545 DEBUG:boto:Data: 2023-10-17T18:53:42.545 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.546 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.547 DEBUG:boto:Params: {} 2023-10-17T18:53:42.547 DEBUG:boto:Token: None 2023-10-17T18:53:42.547 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9600fe73deadac1-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.551 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:42.552 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:42.552 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.553 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:42.553 DEBUG:boto:Data: 2023-10-17T18:53:42.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.554 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.554 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.555 DEBUG:boto:Params: {} 2023-10-17T18:53:42.555 DEBUG:boto:Token: None 2023-10-17T18:53:42.555 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:42.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Acd5PiBPZM6Kl99t9kf4sAKi7O4= 2023-10-17T18:53:42.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Acd5PiBPZM6Kl99t9kf4sAKi7O4='} 2023-10-17T18:53:42.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbc2c6c232066da1-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.599 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:42.600 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:42.600 DEBUG:boto:Method: HEAD 2023-10-17T18:53:42.601 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:42.601 DEBUG:boto:Data: 2023-10-17T18:53:42.601 DEBUG:boto:Headers: {} 2023-10-17T18:53:42.602 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.602 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.603 DEBUG:boto:Params: {} 2023-10-17T18:53:42.603 DEBUG:boto:Token: None 2023-10-17T18:53:42.604 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/ 2023-10-17T18:53:42.604 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg= 2023-10-17T18:53:42.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ngig8uXYd8CH5DXSzVWLob7R0Gg=', 'Content-Length': '0'} 2023-10-17T18:53:42.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000175a49fff3f7bd9a-00652ed836-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.607 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:42.608 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:42.608 DEBUG:boto:Method: PUT 2023-10-17T18:53:42.609 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:42.609 DEBUG:boto:Data: 2023-10-17T18:53:42.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:53:42.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:42.610 DEBUG:boto:Port: 8000 2023-10-17T18:53:42.611 DEBUG:boto:Params: {} 2023-10-17T18:53:42.611 DEBUG:boto:Token: None 2023-10-17T18:53:42.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:53:42 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:42.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4mwoTM4Ybcqsqe9EbYOILy4CN3M= 2023-10-17T18:53:42.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:53:42 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4mwoTM4Ybcqsqe9EbYOILy4CN3M='} 2023-10-17T18:53:42.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000debb34c92e23d3b3-00652ed836-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:42.618 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-58 2023-10-17T18:53:42.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:42.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:42.620 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:53:42.678 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:42.678 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:42.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:42.711+0000 7fe0749b9780 1 waiting to reach incremental sync.. 2023-10-17T18:53:47.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:47.713+0000 7fe0749b9780 1 bucket sync caught up with source: 2023-10-17T18:53:47.715 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.41.6, 00000000001.49.6, , 00000000001.37.6, 00000000001.35.6, 00000000001.59.6, 00000000001.25.6, 00000000001.25.6, 00000000001.67.6, 00000000001.62.6, 00000000001.42.6] 2023-10-17T18:53:47.715 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.41.6, 00000000001.49.6, , 00000000001.37.6, 00000000001.35.6, 00000000001.59.6, 00000000001.25.6, 00000000001.25.6, 00000000001.67.6, 00000000001.62.6, 00000000001.42.6] 2023-10-17T18:53:47.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:47.713+0000 7fe0749b9780 0 bucket checkpoint complete 2023-10-17T18:53:47.732 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-58 2023-10-17T18:53:47.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:47.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:53:47.734 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:53:47.799 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:53:47.799 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:53:47.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:47.836+0000 7f8c44252780 1 waiting to reach incremental sync.. 2023-10-17T18:53:52.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:52.839+0000 7f8c44252780 1 bucket sync caught up with source: 2023-10-17T18:53:52.840 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.182.6, 00000000001.164.6, , 00000000001.341.6, 00000000001.257.6, 00000000001.122.6, 00000000001.117.6, 00000000001.37.6, 00000000001.299.6, 00000000001.239.6, 00000000001.250.6] 2023-10-17T18:53:52.840 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.182.6, 00000000001.164.6, , 00000000001.341.6, 00000000001.257.6, 00000000001.122.6, 00000000001.117.6, 00000000001.37.6, 00000000001.299.6, 00000000001.239.6, 00000000001.250.6] 2023-10-17T18:53:52.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:53:52.839+0000 7f8c44252780 0 bucket checkpoint complete 2023-10-17T18:53:52.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-58 zones={a2, a1} 2023-10-17T18:53:52.858 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.858 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.859 DEBUG:boto:Method: HEAD 2023-10-17T18:53:52.859 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:52.860 DEBUG:boto:Data: 2023-10-17T18:53:52.860 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.860 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:52.861 DEBUG:boto:Port: 8001 2023-10-17T18:53:52.861 DEBUG:boto:Params: {} 2023-10-17T18:53:52.862 DEBUG:boto:Token: None 2023-10-17T18:53:52.862 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/ 2023-10-17T18:53:52.863 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B44uk4gIyr9ildRFN1lraUVuJEM= 2023-10-17T18:53:52.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B44uk4gIyr9ildRFN1lraUVuJEM=', 'Content-Length': '0'} 2023-10-17T18:53:52.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000614c0c460a450a47-00652ed840-1a76-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.867 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.867 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.868 DEBUG:boto:Method: HEAD 2023-10-17T18:53:52.868 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:53:52.869 DEBUG:boto:Data: 2023-10-17T18:53:52.869 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.869 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:52.870 DEBUG:boto:Port: 8000 2023-10-17T18:53:52.870 DEBUG:boto:Params: {} 2023-10-17T18:53:52.871 DEBUG:boto:Token: None 2023-10-17T18:53:52.871 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/ 2023-10-17T18:53:52.872 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B44uk4gIyr9ildRFN1lraUVuJEM= 2023-10-17T18:53:52.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B44uk4gIyr9ildRFN1lraUVuJEM=', 'Content-Length': '0'} 2023-10-17T18:53:52.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fccb5af8a2598c6a-00652ed840-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:53:52.876 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.876 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.876 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.877 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.877 DEBUG:boto:Method: GET 2023-10-17T18:53:52.878 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.878 DEBUG:boto:Data: 2023-10-17T18:53:52.879 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.879 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:52.879 DEBUG:boto:Port: 8001 2023-10-17T18:53:52.880 DEBUG:boto:Params: {} 2023-10-17T18:53:52.880 DEBUG:boto:Token: None 2023-10-17T18:53:52.881 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/?versions 2023-10-17T18:53:52.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g= 2023-10-17T18:53:52.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g=', 'Content-Length': '0'} 2023-10-17T18:53:52.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535b1b0f37301d14-00652ed840-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.885 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:52.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-17T18:53:52.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-17T18:53:52.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-17T18:53:52.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-17T18:53:52.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-17T18:53:52.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-17T18:53:52.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-17T18:53:52.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-17T18:53:52.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-17T18:53:52.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-17T18:53:52.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:53:52.890 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.890 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.891 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.891 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.892 DEBUG:boto:Method: GET 2023-10-17T18:53:52.892 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.892 DEBUG:boto:Data: 2023-10-17T18:53:52.893 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.893 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:52.894 DEBUG:boto:Port: 8000 2023-10-17T18:53:52.894 DEBUG:boto:Params: {} 2023-10-17T18:53:52.894 DEBUG:boto:Token: None 2023-10-17T18:53:52.895 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/?versions 2023-10-17T18:53:52.895 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g= 2023-10-17T18:53:52.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g=', 'Content-Length': '0'} 2023-10-17T18:53:52.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2394bf2b670c5c7-00652ed840-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.899 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:52.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-17T18:53:52.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-17T18:53:52.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-17T18:53:52.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-17T18:53:52.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-17T18:53:52.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-17T18:53:52.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-17T18:53:52.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-17T18:53:52.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-17T18:53:52.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-17T18:53:52.904 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.904 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.905 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.905 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.906 DEBUG:boto:Method: GET 2023-10-17T18:53:52.906 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.907 DEBUG:boto:Data: 2023-10-17T18:53:52.907 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:52.908 DEBUG:boto:Port: 8001 2023-10-17T18:53:52.908 DEBUG:boto:Params: {} 2023-10-17T18:53:52.909 DEBUG:boto:Token: None 2023-10-17T18:53:52.909 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/?versions 2023-10-17T18:53:52.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g= 2023-10-17T18:53:52.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g=', 'Content-Length': '0'} 2023-10-17T18:53:52.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008056ed9414b5b7f7-00652ed840-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.913 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:52.914 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:52.915 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:52.915 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.916 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.916 DEBUG:boto:Method: GET 2023-10-17T18:53:52.916 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:53:52.917 DEBUG:boto:Data: 2023-10-17T18:53:52.917 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:52.918 DEBUG:boto:Port: 8000 2023-10-17T18:53:52.919 DEBUG:boto:Params: {} 2023-10-17T18:53:52.919 DEBUG:boto:Token: None 2023-10-17T18:53:52.919 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/?versions 2023-10-17T18:53:52.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g= 2023-10-17T18:53:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4hjTYGMYMaqvdmZGxRgwVMFr+5g=', 'Content-Length': '0'} 2023-10-17T18:53:52.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029887c4ed910c816-00652ed840-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.923 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:52.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:53:52.925 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.925 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.926 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.926 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.926 DEBUG:boto:Method: GET 2023-10-17T18:53:52.927 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.927 DEBUG:boto:Data: 2023-10-17T18:53:52.928 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.928 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:52.929 DEBUG:boto:Port: 8001 2023-10-17T18:53:52.929 DEBUG:boto:Params: {} 2023-10-17T18:53:52.930 DEBUG:boto:Token: None 2023-10-17T18:53:52.930 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jNZ0zddRi3g0UJkb1qKECZaa4vw= 2023-10-17T18:53:52.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jNZ0zddRi3g0UJkb1qKECZaa4vw=', 'Content-Length': '0'} 2023-10-17T18:53:52.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe3c00e231cfffe3-00652ed840-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.934 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.934 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.935 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.935 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.936 DEBUG:boto:Method: GET 2023-10-17T18:53:52.936 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.936 DEBUG:boto:Data: 2023-10-17T18:53:52.937 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:52.938 DEBUG:boto:Port: 8000 2023-10-17T18:53:52.938 DEBUG:boto:Params: {} 2023-10-17T18:53:52.938 DEBUG:boto:Token: None 2023-10-17T18:53:52.939 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jNZ0zddRi3g0UJkb1qKECZaa4vw= 2023-10-17T18:53:52.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jNZ0zddRi3g0UJkb1qKECZaa4vw=', 'Content-Length': '0'} 2023-10-17T18:53:52.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d16c6a4506edfc-00652ed840-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.982 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.983 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.984 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.984 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.984 DEBUG:boto:Method: HEAD 2023-10-17T18:53:52.985 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.985 DEBUG:boto:Data: 2023-10-17T18:53:52.986 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:52.987 DEBUG:boto:Port: 8001 2023-10-17T18:53:52.987 DEBUG:boto:Params: {} 2023-10-17T18:53:52.988 DEBUG:boto:Token: None 2023-10-17T18:53:52.988 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IEw9hm3kKV4grEjjSHVoCYur4kY= 2023-10-17T18:53:52.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IEw9hm3kKV4grEjjSHVoCYur4kY=', 'Content-Length': '0'} 2023-10-17T18:53:52.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34414850830b21a-00652ed840-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:52.991 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.992 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:52.992 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.993 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.993 DEBUG:boto:Method: HEAD 2023-10-17T18:53:52.993 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.994 DEBUG:boto:Data: 2023-10-17T18:53:52.994 DEBUG:boto:Headers: {} 2023-10-17T18:53:52.994 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:52.995 DEBUG:boto:Port: 8000 2023-10-17T18:53:52.995 DEBUG:boto:Params: {} 2023-10-17T18:53:52.996 DEBUG:boto:Token: None 2023-10-17T18:53:52.996 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:52 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:52.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IEw9hm3kKV4grEjjSHVoCYur4kY= 2023-10-17T18:53:52.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IEw9hm3kKV4grEjjSHVoCYur4kY=', 'Content-Length': '0'} 2023-10-17T18:53:52.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000145a983f3e18d762-00652ed840-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:53:53.000 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.000 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.001 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.001 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.002 DEBUG:boto:Method: GET 2023-10-17T18:53:53.002 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.003 DEBUG:boto:Data: 2023-10-17T18:53:53.003 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.004 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.004 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.005 DEBUG:boto:Params: {} 2023-10-17T18:53:53.005 DEBUG:boto:Token: None 2023-10-17T18:53:53.006 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.007 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cl6kRMG6nJOuE76dvnDXT9RHkOc= 2023-10-17T18:53:53.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cl6kRMG6nJOuE76dvnDXT9RHkOc=', 'Content-Length': '0'} 2023-10-17T18:53:53.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3cf18f593cadc8-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.010 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.010 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.011 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.011 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.012 DEBUG:boto:Method: GET 2023-10-17T18:53:53.012 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.013 DEBUG:boto:Data: 2023-10-17T18:53:53.013 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.014 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.014 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.015 DEBUG:boto:Params: {} 2023-10-17T18:53:53.015 DEBUG:boto:Token: None 2023-10-17T18:53:53.015 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:53:53.016 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cl6kRMG6nJOuE76dvnDXT9RHkOc= 2023-10-17T18:53:53.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cl6kRMG6nJOuE76dvnDXT9RHkOc=', 'Content-Length': '0'} 2023-10-17T18:53:53.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016502fb47b978036-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.019 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.019 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.020 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.020 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.021 DEBUG:boto:Method: GET 2023-10-17T18:53:53.021 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.021 DEBUG:boto:Data: 2023-10-17T18:53:53.022 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.022 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.023 DEBUG:boto:Params: {} 2023-10-17T18:53:53.023 DEBUG:boto:Token: None 2023-10-17T18:53:53.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5fe9a8ba4b17ad0-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.028 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.028 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.029 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.029 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.030 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.030 DEBUG:boto:Method: GET 2023-10-17T18:53:53.031 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.031 DEBUG:boto:Data: 2023-10-17T18:53:53.031 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.032 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.032 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.033 DEBUG:boto:Params: {} 2023-10-17T18:53:53.033 DEBUG:boto:Token: None 2023-10-17T18:53:53.034 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe38cc928d337b3-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.078 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.079 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.080 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.080 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.080 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.081 DEBUG:boto:Data: 2023-10-17T18:53:53.081 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.082 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.083 DEBUG:boto:Params: {} 2023-10-17T18:53:53.083 DEBUG:boto:Token: None 2023-10-17T18:53:53.084 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bmpJtS9lRH+QbnCdRjfExBHFFzs= 2023-10-17T18:53:53.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bmpJtS9lRH+QbnCdRjfExBHFFzs=', 'Content-Length': '0'} 2023-10-17T18:53:53.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c7e02a3b7ce7e1-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.087 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.088 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.088 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.089 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.089 DEBUG:boto:Data: 2023-10-17T18:53:53.090 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.090 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.091 DEBUG:boto:Params: {} 2023-10-17T18:53:53.091 DEBUG:boto:Token: None 2023-10-17T18:53:53.092 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bmpJtS9lRH+QbnCdRjfExBHFFzs= 2023-10-17T18:53:53.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bmpJtS9lRH+QbnCdRjfExBHFFzs=', 'Content-Length': '0'} 2023-10-17T18:53:53.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614eefdf05921f84-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:53:53.096 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.096 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.096 DEBUG:boto:Method: GET 2023-10-17T18:53:53.097 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.097 DEBUG:boto:Data: 2023-10-17T18:53:53.098 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.099 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.099 DEBUG:boto:Params: {} 2023-10-17T18:53:53.100 DEBUG:boto:Token: None 2023-10-17T18:53:53.100 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.101 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oy1ktmDQhA/3rO5qPdtG5/1Inpw= 2023-10-17T18:53:53.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oy1ktmDQhA/3rO5qPdtG5/1Inpw=', 'Content-Length': '0'} 2023-10-17T18:53:53.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e479b39cd21abd3a-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.146 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.147 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.147 DEBUG:boto:Method: GET 2023-10-17T18:53:53.148 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.148 DEBUG:boto:Data: 2023-10-17T18:53:53.149 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.149 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.150 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.150 DEBUG:boto:Params: {} 2023-10-17T18:53:53.151 DEBUG:boto:Token: None 2023-10-17T18:53:53.151 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:53:53.152 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oy1ktmDQhA/3rO5qPdtG5/1Inpw= 2023-10-17T18:53:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oy1ktmDQhA/3rO5qPdtG5/1Inpw=', 'Content-Length': '0'} 2023-10-17T18:53:53.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce274c2bb35e8a5-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.155 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.156 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.156 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.157 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.157 DEBUG:boto:Method: GET 2023-10-17T18:53:53.158 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.158 DEBUG:boto:Data: 2023-10-17T18:53:53.161 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.161 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.162 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.162 DEBUG:boto:Params: {} 2023-10-17T18:53:53.163 DEBUG:boto:Token: None 2023-10-17T18:53:53.163 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000256b00b892509df-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.167 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.168 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.169 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.169 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.170 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.170 DEBUG:boto:Method: GET 2023-10-17T18:53:53.171 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:53:53.171 DEBUG:boto:Data: 2023-10-17T18:53:53.171 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.172 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.172 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.173 DEBUG:boto:Params: {} 2023-10-17T18:53:53.173 DEBUG:boto:Token: None 2023-10-17T18:53:53.174 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.174 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cdab32580e4bb0e-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.178 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:53:42.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:53:53.179 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.180 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.180 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.180 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.181 DEBUG:boto:Method: GET 2023-10-17T18:53:53.181 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.182 DEBUG:boto:Data: 2023-10-17T18:53:53.182 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.182 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.183 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.183 DEBUG:boto:Params: {} 2023-10-17T18:53:53.184 DEBUG:boto:Token: None 2023-10-17T18:53:53.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.185 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q= 2023-10-17T18:53:53.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q=', 'Content-Length': '0'} 2023-10-17T18:53:53.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049289b546d3a7590-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.188 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.188 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.189 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.189 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.190 DEBUG:boto:Method: GET 2023-10-17T18:53:53.190 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.191 DEBUG:boto:Data: 2023-10-17T18:53:53.191 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.191 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.192 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.192 DEBUG:boto:Params: {} 2023-10-17T18:53:53.193 DEBUG:boto:Token: None 2023-10-17T18:53:53.193 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q= 2023-10-17T18:53:53.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q=', 'Content-Length': '0'} 2023-10-17T18:53:53.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd52c650356ba638-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.196 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.197 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.197 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.198 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.198 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.199 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.199 DEBUG:boto:Data: 2023-10-17T18:53:53.199 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.200 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.200 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.201 DEBUG:boto:Params: {} 2023-10-17T18:53:53.201 DEBUG:boto:Token: None 2023-10-17T18:53:53.202 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.202 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jz1RgP7brN2+/FsmMPq3PzjvTNQ= 2023-10-17T18:53:53.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jz1RgP7brN2+/FsmMPq3PzjvTNQ=', 'Content-Length': '0'} 2023-10-17T18:53:53.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094efc3013e1a873f-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.205 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.205 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.206 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.206 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.207 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.207 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.208 DEBUG:boto:Data: 2023-10-17T18:53:53.208 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.209 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.209 DEBUG:boto:Params: {} 2023-10-17T18:53:53.210 DEBUG:boto:Token: None 2023-10-17T18:53:53.210 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jz1RgP7brN2+/FsmMPq3PzjvTNQ= 2023-10-17T18:53:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jz1RgP7brN2+/FsmMPq3PzjvTNQ=', 'Content-Length': '0'} 2023-10-17T18:53:53.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb3f078aa14e1e0-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:53:53.214 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.215 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.215 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.216 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.216 DEBUG:boto:Method: GET 2023-10-17T18:53:53.216 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.217 DEBUG:boto:Data: 2023-10-17T18:53:53.217 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.218 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.219 DEBUG:boto:Params: {} 2023-10-17T18:53:53.219 DEBUG:boto:Token: None 2023-10-17T18:53:53.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q= 2023-10-17T18:53:53.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q=', 'Content-Length': '0'} 2023-10-17T18:53:53.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005693ce68a95ac4c1-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.266 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.267 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.268 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.268 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.268 DEBUG:boto:Method: GET 2023-10-17T18:53:53.269 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.269 DEBUG:boto:Data: 2023-10-17T18:53:53.270 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.271 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.271 DEBUG:boto:Params: {} 2023-10-17T18:53:53.272 DEBUG:boto:Token: None 2023-10-17T18:53:53.272 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:53:53.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q= 2023-10-17T18:53:53.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ESj5db3uI9M8qQLsLIr6gWYo1Q=', 'Content-Length': '0'} 2023-10-17T18:53:53.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf4250fad89f3ea-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.276 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.277 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.277 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.278 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.278 DEBUG:boto:Method: GET 2023-10-17T18:53:53.279 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.279 DEBUG:boto:Data: 2023-10-17T18:53:53.279 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.280 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.280 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.281 DEBUG:boto:Params: {} 2023-10-17T18:53:53.281 DEBUG:boto:Token: None 2023-10-17T18:53:53.282 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.282 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc423340eb6075cb-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.286 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.287 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.287 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.288 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.288 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.289 DEBUG:boto:Method: GET 2023-10-17T18:53:53.289 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.290 DEBUG:boto:Data: 2023-10-17T18:53:53.290 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.291 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.291 DEBUG:boto:Params: {} 2023-10-17T18:53:53.292 DEBUG:boto:Token: None 2023-10-17T18:53:53.292 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.293 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000134bb52651956612-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.297 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.298 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.298 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.299 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.299 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.299 DEBUG:boto:Data: 2023-10-17T18:53:53.300 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.301 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.301 DEBUG:boto:Params: {} 2023-10-17T18:53:53.302 DEBUG:boto:Token: None 2023-10-17T18:53:53.302 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.303 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kuOsaosUzmM3lQreV9Y7XTro8gw= 2023-10-17T18:53:53.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kuOsaosUzmM3lQreV9Y7XTro8gw=', 'Content-Length': '0'} 2023-10-17T18:53:53.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2c8e5645b82b93-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.306 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.306 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.307 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.307 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.307 DEBUG:boto:Data: 2023-10-17T18:53:53.308 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.308 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.309 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.309 DEBUG:boto:Params: {} 2023-10-17T18:53:53.310 DEBUG:boto:Token: None 2023-10-17T18:53:53.310 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kuOsaosUzmM3lQreV9Y7XTro8gw= 2023-10-17T18:53:53.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kuOsaosUzmM3lQreV9Y7XTro8gw=', 'Content-Length': '0'} 2023-10-17T18:53:53.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a981ca646a23b07-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:53:53.314 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.314 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.315 DEBUG:boto:Method: GET 2023-10-17T18:53:53.315 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.316 DEBUG:boto:Data: 2023-10-17T18:53:53.316 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.317 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.317 DEBUG:boto:Params: {} 2023-10-17T18:53:53.318 DEBUG:boto:Token: None 2023-10-17T18:53:53.318 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8GICf2K20V7dmRHmQjPTG3CBQ2M= 2023-10-17T18:53:53.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8GICf2K20V7dmRHmQjPTG3CBQ2M=', 'Content-Length': '0'} 2023-10-17T18:53:53.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2dc2b5a3dfd867-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.322 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.323 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.323 DEBUG:boto:Method: GET 2023-10-17T18:53:53.324 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.324 DEBUG:boto:Data: 2023-10-17T18:53:53.324 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.325 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.326 DEBUG:boto:Params: {} 2023-10-17T18:53:53.326 DEBUG:boto:Token: None 2023-10-17T18:53:53.327 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:53:53.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8GICf2K20V7dmRHmQjPTG3CBQ2M= 2023-10-17T18:53:53.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8GICf2K20V7dmRHmQjPTG3CBQ2M=', 'Content-Length': '0'} 2023-10-17T18:53:53.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118fcab3a84ac60c-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.331 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.331 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.332 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.332 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.333 DEBUG:boto:Method: GET 2023-10-17T18:53:53.333 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.334 DEBUG:boto:Data: 2023-10-17T18:53:53.334 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.335 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.335 DEBUG:boto:Params: {} 2023-10-17T18:53:53.336 DEBUG:boto:Token: None 2023-10-17T18:53:53.336 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004878446ffdca694e-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.341 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.341 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.342 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.342 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.343 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.343 DEBUG:boto:Method: GET 2023-10-17T18:53:53.344 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:53:53.344 DEBUG:boto:Data: 2023-10-17T18:53:53.344 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.345 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.346 DEBUG:boto:Params: {} 2023-10-17T18:53:53.346 DEBUG:boto:Token: None 2023-10-17T18:53:53.347 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cfdb8ec76fa4250-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.398 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:53:42.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:53:53.400 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.400 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.401 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.401 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.401 DEBUG:boto:Method: GET 2023-10-17T18:53:53.402 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.402 DEBUG:boto:Data: 2023-10-17T18:53:53.403 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.403 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.403 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.404 DEBUG:boto:Params: {} 2023-10-17T18:53:53.404 DEBUG:boto:Token: None 2023-10-17T18:53:53.405 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8= 2023-10-17T18:53:53.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8=', 'Content-Length': '0'} 2023-10-17T18:53:53.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fde9a419f49c603-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.409 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.410 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.410 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.410 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.411 DEBUG:boto:Method: GET 2023-10-17T18:53:53.411 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.412 DEBUG:boto:Data: 2023-10-17T18:53:53.412 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.413 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.413 DEBUG:boto:Params: {} 2023-10-17T18:53:53.414 DEBUG:boto:Token: None 2023-10-17T18:53:53.414 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8= 2023-10-17T18:53:53.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8=', 'Content-Length': '0'} 2023-10-17T18:53:53.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb1f9d6b44b450b4-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.418 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.418 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.419 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.419 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.420 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.420 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.420 DEBUG:boto:Data: 2023-10-17T18:53:53.421 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.421 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.422 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.422 DEBUG:boto:Params: {} 2023-10-17T18:53:53.422 DEBUG:boto:Token: None 2023-10-17T18:53:53.423 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.423 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K8UYK4mxudxRKtjYv8fY9CbLcvM= 2023-10-17T18:53:53.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K8UYK4mxudxRKtjYv8fY9CbLcvM=', 'Content-Length': '0'} 2023-10-17T18:53:53.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6d9c7beec4674e-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.426 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.426 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.427 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.427 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.428 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.428 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.428 DEBUG:boto:Data: 2023-10-17T18:53:53.429 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.429 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.430 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.430 DEBUG:boto:Params: {} 2023-10-17T18:53:53.430 DEBUG:boto:Token: None 2023-10-17T18:53:53.431 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.431 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K8UYK4mxudxRKtjYv8fY9CbLcvM= 2023-10-17T18:53:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K8UYK4mxudxRKtjYv8fY9CbLcvM=', 'Content-Length': '0'} 2023-10-17T18:53:53.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005095e2d5fe686d1d-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:53:53.435 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.435 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.436 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.436 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.437 DEBUG:boto:Method: GET 2023-10-17T18:53:53.437 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.438 DEBUG:boto:Data: 2023-10-17T18:53:53.438 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.439 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.439 DEBUG:boto:Params: {} 2023-10-17T18:53:53.440 DEBUG:boto:Token: None 2023-10-17T18:53:53.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8= 2023-10-17T18:53:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8=', 'Content-Length': '0'} 2023-10-17T18:53:53.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79f6f580a12f090-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.444 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.445 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.445 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.445 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.446 DEBUG:boto:Method: GET 2023-10-17T18:53:53.446 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.447 DEBUG:boto:Data: 2023-10-17T18:53:53.447 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.448 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.448 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.448 DEBUG:boto:Params: {} 2023-10-17T18:53:53.449 DEBUG:boto:Token: None 2023-10-17T18:53:53.449 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:53:53.450 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8= 2023-10-17T18:53:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AMGYK1rsUC85O7yYsplH6WTxoO8=', 'Content-Length': '0'} 2023-10-17T18:53:53.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf181b5a93312b4-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.453 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.454 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.454 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.454 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.455 DEBUG:boto:Method: GET 2023-10-17T18:53:53.455 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.456 DEBUG:boto:Data: 2023-10-17T18:53:53.456 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.457 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.457 DEBUG:boto:Params: {} 2023-10-17T18:53:53.458 DEBUG:boto:Token: None 2023-10-17T18:53:53.458 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.459 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5b839990cbd5cef-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.462 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.463 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.464 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.464 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.464 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.465 DEBUG:boto:Method: GET 2023-10-17T18:53:53.465 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.466 DEBUG:boto:Data: 2023-10-17T18:53:53.466 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.467 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.467 DEBUG:boto:Params: {} 2023-10-17T18:53:53.468 DEBUG:boto:Token: None 2023-10-17T18:53:53.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd1bc7e9e194e26-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.472 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.473 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.473 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.474 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.474 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.474 DEBUG:boto:Data: 2023-10-17T18:53:53.475 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.475 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.475 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.476 DEBUG:boto:Params: {} 2023-10-17T18:53:53.476 DEBUG:boto:Token: None 2023-10-17T18:53:53.477 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.477 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GtSMHukf4EnOwLiLhizzNXX5u+4= 2023-10-17T18:53:53.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GtSMHukf4EnOwLiLhizzNXX5u+4=', 'Content-Length': '0'} 2023-10-17T18:53:53.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009211f4b069696de4-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.480 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.481 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.481 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.481 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.482 DEBUG:boto:Data: 2023-10-17T18:53:53.482 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.483 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.483 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.484 DEBUG:boto:Params: {} 2023-10-17T18:53:53.484 DEBUG:boto:Token: None 2023-10-17T18:53:53.485 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.485 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GtSMHukf4EnOwLiLhizzNXX5u+4= 2023-10-17T18:53:53.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GtSMHukf4EnOwLiLhizzNXX5u+4=', 'Content-Length': '0'} 2023-10-17T18:53:53.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000955ae3e0aee6325-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:53:53.488 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.489 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.489 DEBUG:boto:Method: GET 2023-10-17T18:53:53.490 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.490 DEBUG:boto:Data: 2023-10-17T18:53:53.491 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.491 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.492 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.492 DEBUG:boto:Params: {} 2023-10-17T18:53:53.493 DEBUG:boto:Token: None 2023-10-17T18:53:53.493 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:za0uhZ7fZcXynNgFDto+abZnow0= 2023-10-17T18:53:53.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:za0uhZ7fZcXynNgFDto+abZnow0=', 'Content-Length': '0'} 2023-10-17T18:53:53.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304b05d059aa149a-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.538 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.539 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.539 DEBUG:boto:Method: GET 2023-10-17T18:53:53.540 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.540 DEBUG:boto:Data: 2023-10-17T18:53:53.541 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.542 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.542 DEBUG:boto:Params: {} 2023-10-17T18:53:53.543 DEBUG:boto:Token: None 2023-10-17T18:53:53.543 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:53:53.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:za0uhZ7fZcXynNgFDto+abZnow0= 2023-10-17T18:53:53.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:za0uhZ7fZcXynNgFDto+abZnow0=', 'Content-Length': '0'} 2023-10-17T18:53:53.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004346bd28a53fb2cf-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.547 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.547 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.548 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.548 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.549 DEBUG:boto:Method: GET 2023-10-17T18:53:53.549 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.550 DEBUG:boto:Data: 2023-10-17T18:53:53.550 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.551 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.551 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.551 DEBUG:boto:Params: {} 2023-10-17T18:53:53.552 DEBUG:boto:Token: None 2023-10-17T18:53:53.552 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.553 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513e2a9c4bc7c0ab-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.557 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.557 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.558 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.558 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.559 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.559 DEBUG:boto:Method: GET 2023-10-17T18:53:53.560 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:53:53.560 DEBUG:boto:Data: 2023-10-17T18:53:53.560 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.561 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.562 DEBUG:boto:Params: {} 2023-10-17T18:53:53.562 DEBUG:boto:Token: None 2023-10-17T18:53:53.563 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b382a0b1bc27495-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.610 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:53:42.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:53:53.612 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.612 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.613 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.613 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.614 DEBUG:boto:Method: GET 2023-10-17T18:53:53.614 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.615 DEBUG:boto:Data: 2023-10-17T18:53:53.615 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.615 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.616 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.616 DEBUG:boto:Params: {} 2023-10-17T18:53:53.617 DEBUG:boto:Token: None 2023-10-17T18:53:53.617 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.618 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA= 2023-10-17T18:53:53.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA=', 'Content-Length': '0'} 2023-10-17T18:53:53.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804ab45f4fe70f98-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.621 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.622 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.622 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.623 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.623 DEBUG:boto:Method: GET 2023-10-17T18:53:53.623 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.624 DEBUG:boto:Data: 2023-10-17T18:53:53.624 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.625 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.626 DEBUG:boto:Params: {} 2023-10-17T18:53:53.626 DEBUG:boto:Token: None 2023-10-17T18:53:53.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.627 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA= 2023-10-17T18:53:53.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA=', 'Content-Length': '0'} 2023-10-17T18:53:53.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d269113916328dca-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.631 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.631 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.632 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.632 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.632 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.633 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.633 DEBUG:boto:Data: 2023-10-17T18:53:53.633 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.634 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.635 DEBUG:boto:Params: {} 2023-10-17T18:53:53.635 DEBUG:boto:Token: None 2023-10-17T18:53:53.636 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.636 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ynWUUxrGptlQwZ/VseENZ+81xdQ= 2023-10-17T18:53:53.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ynWUUxrGptlQwZ/VseENZ+81xdQ=', 'Content-Length': '0'} 2023-10-17T18:53:53.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afa19352733c26a-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.639 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.640 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.640 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.641 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.641 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.642 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.642 DEBUG:boto:Data: 2023-10-17T18:53:53.642 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.643 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.644 DEBUG:boto:Params: {} 2023-10-17T18:53:53.644 DEBUG:boto:Token: None 2023-10-17T18:53:53.645 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ynWUUxrGptlQwZ/VseENZ+81xdQ= 2023-10-17T18:53:53.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ynWUUxrGptlQwZ/VseENZ+81xdQ=', 'Content-Length': '0'} 2023-10-17T18:53:53.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e5a1e00038c0a9-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:53:53.649 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.649 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.649 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.650 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.650 DEBUG:boto:Method: GET 2023-10-17T18:53:53.651 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.651 DEBUG:boto:Data: 2023-10-17T18:53:53.652 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.652 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.652 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.653 DEBUG:boto:Params: {} 2023-10-17T18:53:53.653 DEBUG:boto:Token: None 2023-10-17T18:53:53.654 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.654 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA= 2023-10-17T18:53:53.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA=', 'Content-Length': '0'} 2023-10-17T18:53:53.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d510d1e0b2463914-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.657 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.658 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.658 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.659 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.659 DEBUG:boto:Method: GET 2023-10-17T18:53:53.660 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.660 DEBUG:boto:Data: 2023-10-17T18:53:53.660 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.661 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.662 DEBUG:boto:Params: {} 2023-10-17T18:53:53.662 DEBUG:boto:Token: None 2023-10-17T18:53:53.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:53:53.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA= 2023-10-17T18:53:53.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tRO58xhde18CKBvQSWuj1EwWUpA=', 'Content-Length': '0'} 2023-10-17T18:53:53.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02f6048cbfa4413-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.710 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.711 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.711 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.712 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.712 DEBUG:boto:Method: GET 2023-10-17T18:53:53.713 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.713 DEBUG:boto:Data: 2023-10-17T18:53:53.713 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.714 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.714 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.715 DEBUG:boto:Params: {} 2023-10-17T18:53:53.715 DEBUG:boto:Token: None 2023-10-17T18:53:53.716 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.716 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bfea4e445fdaa73-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.720 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.721 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.721 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.722 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.722 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.723 DEBUG:boto:Method: GET 2023-10-17T18:53:53.723 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.724 DEBUG:boto:Data: 2023-10-17T18:53:53.724 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.725 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.725 DEBUG:boto:Params: {} 2023-10-17T18:53:53.726 DEBUG:boto:Token: None 2023-10-17T18:53:53.726 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.727 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e79962166fc71c-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.730 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.731 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.731 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.732 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.732 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.733 DEBUG:boto:Data: 2023-10-17T18:53:53.733 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.733 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.734 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.734 DEBUG:boto:Params: {} 2023-10-17T18:53:53.735 DEBUG:boto:Token: None 2023-10-17T18:53:53.735 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D9eYxp6QCsxD/v/Us03qDVwA5wI= 2023-10-17T18:53:53.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D9eYxp6QCsxD/v/Us03qDVwA5wI=', 'Content-Length': '0'} 2023-10-17T18:53:53.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c17b8b242731a2-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.739 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.739 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.740 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.740 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.740 DEBUG:boto:Data: 2023-10-17T18:53:53.741 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.741 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.742 DEBUG:boto:Params: {} 2023-10-17T18:53:53.742 DEBUG:boto:Token: None 2023-10-17T18:53:53.743 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.743 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D9eYxp6QCsxD/v/Us03qDVwA5wI= 2023-10-17T18:53:53.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D9eYxp6QCsxD/v/Us03qDVwA5wI=', 'Content-Length': '0'} 2023-10-17T18:53:53.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090840767d5f6bbd7-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:53:53.747 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.747 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.748 DEBUG:boto:Method: GET 2023-10-17T18:53:53.748 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.749 DEBUG:boto:Data: 2023-10-17T18:53:53.749 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.750 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.751 DEBUG:boto:Params: {} 2023-10-17T18:53:53.751 DEBUG:boto:Token: None 2023-10-17T18:53:53.752 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EpqEvwrHwCU8R8lx/g60UnrhLIs= 2023-10-17T18:53:53.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EpqEvwrHwCU8R8lx/g60UnrhLIs=', 'Content-Length': '0'} 2023-10-17T18:53:53.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070938ef28ccf2550-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.798 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.799 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.800 DEBUG:boto:Method: GET 2023-10-17T18:53:53.800 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.801 DEBUG:boto:Data: 2023-10-17T18:53:53.801 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.802 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.802 DEBUG:boto:Params: {} 2023-10-17T18:53:53.803 DEBUG:boto:Token: None 2023-10-17T18:53:53.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:53:53.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EpqEvwrHwCU8R8lx/g60UnrhLIs= 2023-10-17T18:53:53.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EpqEvwrHwCU8R8lx/g60UnrhLIs=', 'Content-Length': '0'} 2023-10-17T18:53:53.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ba4a725bf4e431-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.807 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.808 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.808 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.809 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.809 DEBUG:boto:Method: GET 2023-10-17T18:53:53.810 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.810 DEBUG:boto:Data: 2023-10-17T18:53:53.811 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.811 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.811 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.812 DEBUG:boto:Params: {} 2023-10-17T18:53:53.812 DEBUG:boto:Token: None 2023-10-17T18:53:53.813 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.813 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae4a150c4a4783e-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.817 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.818 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.818 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.818 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.819 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.819 DEBUG:boto:Method: GET 2023-10-17T18:53:53.820 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:53:53.820 DEBUG:boto:Data: 2023-10-17T18:53:53.821 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.822 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.822 DEBUG:boto:Params: {} 2023-10-17T18:53:53.823 DEBUG:boto:Token: None 2023-10-17T18:53:53.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a51e85c45953f5b-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.827 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:53:42.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:53:53.828 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.829 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.829 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.829 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.830 DEBUG:boto:Method: GET 2023-10-17T18:53:53.830 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.831 DEBUG:boto:Data: 2023-10-17T18:53:53.831 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.832 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.832 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.833 DEBUG:boto:Params: {} 2023-10-17T18:53:53.833 DEBUG:boto:Token: None 2023-10-17T18:53:53.834 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.834 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU= 2023-10-17T18:53:53.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU=', 'Content-Length': '0'} 2023-10-17T18:53:53.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69083c0793503b8-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.838 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.838 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.839 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.839 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.839 DEBUG:boto:Method: GET 2023-10-17T18:53:53.840 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.840 DEBUG:boto:Data: 2023-10-17T18:53:53.841 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.841 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.842 DEBUG:boto:Params: {} 2023-10-17T18:53:53.842 DEBUG:boto:Token: None 2023-10-17T18:53:53.843 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU= 2023-10-17T18:53:53.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU=', 'Content-Length': '0'} 2023-10-17T18:53:53.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f9b67aba90671d-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.886 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.887 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.887 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.888 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.888 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.889 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.889 DEBUG:boto:Data: 2023-10-17T18:53:53.890 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.890 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.891 DEBUG:boto:Params: {} 2023-10-17T18:53:53.891 DEBUG:boto:Token: None 2023-10-17T18:53:53.892 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o82NJB6NSe+4RxjwTwYkEkVfeAU= 2023-10-17T18:53:53.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o82NJB6NSe+4RxjwTwYkEkVfeAU=', 'Content-Length': '0'} 2023-10-17T18:53:53.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac2816c64749e05-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.895 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.896 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.896 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.897 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.897 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.897 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.898 DEBUG:boto:Data: 2023-10-17T18:53:53.898 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.899 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.900 DEBUG:boto:Params: {} 2023-10-17T18:53:53.900 DEBUG:boto:Token: None 2023-10-17T18:53:53.900 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o82NJB6NSe+4RxjwTwYkEkVfeAU= 2023-10-17T18:53:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o82NJB6NSe+4RxjwTwYkEkVfeAU=', 'Content-Length': '0'} 2023-10-17T18:53:53.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002359752b7ed1c8a1-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:53:53.904 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.905 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.905 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.906 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.906 DEBUG:boto:Method: GET 2023-10-17T18:53:53.907 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.907 DEBUG:boto:Data: 2023-10-17T18:53:53.908 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.908 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.909 DEBUG:boto:Params: {} 2023-10-17T18:53:53.909 DEBUG:boto:Token: None 2023-10-17T18:53:53.910 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU= 2023-10-17T18:53:53.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU=', 'Content-Length': '0'} 2023-10-17T18:53:53.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8e3eb84b374c3e-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.914 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.914 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.915 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.915 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.915 DEBUG:boto:Method: GET 2023-10-17T18:53:53.916 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.916 DEBUG:boto:Data: 2023-10-17T18:53:53.917 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.918 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.918 DEBUG:boto:Params: {} 2023-10-17T18:53:53.919 DEBUG:boto:Token: None 2023-10-17T18:53:53.919 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:53:53.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU= 2023-10-17T18:53:53.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ivHORtXR0WYHWrs1RC/TDtqL5vU=', 'Content-Length': '0'} 2023-10-17T18:53:53.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042bdd6fdefe528d4-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.923 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.923 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.924 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.924 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.925 DEBUG:boto:Method: GET 2023-10-17T18:53:53.925 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.925 DEBUG:boto:Data: 2023-10-17T18:53:53.926 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.927 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.927 DEBUG:boto:Params: {} 2023-10-17T18:53:53.928 DEBUG:boto:Token: None 2023-10-17T18:53:53.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f336b56a4ca9bbe5-00652ed841-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.932 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.933 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:53.933 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:53.934 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.934 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.935 DEBUG:boto:Method: GET 2023-10-17T18:53:53.935 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:53.936 DEBUG:boto:Data: 2023-10-17T18:53:53.936 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.937 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.937 DEBUG:boto:Params: {} 2023-10-17T18:53:53.938 DEBUG:boto:Token: None 2023-10-17T18:53:53.938 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/ 2023-10-17T18:53:53.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c= 2023-10-17T18:53:53.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KF687pLxThmVls9NWLpWE+ob65c=', 'Content-Length': '0'} 2023-10-17T18:53:53.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ce2e4a2d4cefe9-00652ed841-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.982 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:53.983 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.984 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.984 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.984 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.985 DEBUG:boto:Data: 2023-10-17T18:53:53.985 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:53.986 DEBUG:boto:Port: 8001 2023-10-17T18:53:53.986 DEBUG:boto:Params: {} 2023-10-17T18:53:53.987 DEBUG:boto:Token: None 2023-10-17T18:53:53.987 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:htCb7JvVwB5n8u9WUYBZCpPSzQk= 2023-10-17T18:53:53.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:htCb7JvVwB5n8u9WUYBZCpPSzQk=', 'Content-Length': '0'} 2023-10-17T18:53:53.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cb5185657cea1c8-00652ed841-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.991 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.991 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.992 DEBUG:boto:Method: HEAD 2023-10-17T18:53:53.992 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.992 DEBUG:boto:Data: 2023-10-17T18:53:53.993 DEBUG:boto:Headers: {} 2023-10-17T18:53:53.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:53.994 DEBUG:boto:Port: 8000 2023-10-17T18:53:53.994 DEBUG:boto:Params: {} 2023-10-17T18:53:53.994 DEBUG:boto:Token: None 2023-10-17T18:53:53.995 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:53 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:53.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:htCb7JvVwB5n8u9WUYBZCpPSzQk= 2023-10-17T18:53:53.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:htCb7JvVwB5n8u9WUYBZCpPSzQk=', 'Content-Length': '0'} 2023-10-17T18:53:53.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f443832b288b34-00652ed841-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:53.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:53:53.999 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.000 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.000 DEBUG:boto:Method: GET 2023-10-17T18:53:54.000 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.001 DEBUG:boto:Data: 2023-10-17T18:53:54.001 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.002 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.003 DEBUG:boto:Params: {} 2023-10-17T18:53:54.003 DEBUG:boto:Token: None 2023-10-17T18:53:54.004 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zlQ1bq6JHrZxHhaSWu4GS2Mxz4o= 2023-10-17T18:53:54.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zlQ1bq6JHrZxHhaSWu4GS2Mxz4o=', 'Content-Length': '0'} 2023-10-17T18:53:54.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006865efa3d719a316-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.054 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.055 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.056 DEBUG:boto:Method: GET 2023-10-17T18:53:54.056 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.057 DEBUG:boto:Data: 2023-10-17T18:53:54.057 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.058 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.058 DEBUG:boto:Params: {} 2023-10-17T18:53:54.059 DEBUG:boto:Token: None 2023-10-17T18:53:54.060 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:53:54.060 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zlQ1bq6JHrZxHhaSWu4GS2Mxz4o= 2023-10-17T18:53:54.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zlQ1bq6JHrZxHhaSWu4GS2Mxz4o=', 'Content-Length': '0'} 2023-10-17T18:53:54.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2a13bbd6443e17-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.063 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.064 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.064 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.065 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.065 DEBUG:boto:Method: GET 2023-10-17T18:53:54.066 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.066 DEBUG:boto:Data: 2023-10-17T18:53:54.066 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.067 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.067 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.068 DEBUG:boto:Params: {} 2023-10-17T18:53:54.068 DEBUG:boto:Token: None 2023-10-17T18:53:54.069 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.069 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7a69151329d8a2-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.073 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.073 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.074 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.074 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.075 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.075 DEBUG:boto:Method: GET 2023-10-17T18:53:54.075 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:53:54.076 DEBUG:boto:Data: 2023-10-17T18:53:54.076 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.077 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.077 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.078 DEBUG:boto:Params: {} 2023-10-17T18:53:54.078 DEBUG:boto:Token: None 2023-10-17T18:53:54.079 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007286d2c418d4f65f-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.083 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:53:42.462Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:53:54.084 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.085 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.085 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.085 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.086 DEBUG:boto:Method: GET 2023-10-17T18:53:54.086 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.087 DEBUG:boto:Data: 2023-10-17T18:53:54.087 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.087 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.088 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.088 DEBUG:boto:Params: {} 2023-10-17T18:53:54.089 DEBUG:boto:Token: None 2023-10-17T18:53:54.089 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI= 2023-10-17T18:53:54.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI=', 'Content-Length': '0'} 2023-10-17T18:53:54.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f9a3d146fef920-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.093 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.093 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.094 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.094 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.095 DEBUG:boto:Method: GET 2023-10-17T18:53:54.095 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.096 DEBUG:boto:Data: 2023-10-17T18:53:54.096 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.097 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.097 DEBUG:boto:Params: {} 2023-10-17T18:53:54.098 DEBUG:boto:Token: None 2023-10-17T18:53:54.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI= 2023-10-17T18:53:54.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI=', 'Content-Length': '0'} 2023-10-17T18:53:54.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e7f0bdff3cec855-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.142 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.143 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.143 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.144 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.144 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.145 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.145 DEBUG:boto:Data: 2023-10-17T18:53:54.145 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.146 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.147 DEBUG:boto:Params: {} 2023-10-17T18:53:54.147 DEBUG:boto:Token: None 2023-10-17T18:53:54.147 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rBwr/S0ZLGJS4UKTHLL6XKq94Kk= 2023-10-17T18:53:54.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rBwr/S0ZLGJS4UKTHLL6XKq94Kk=', 'Content-Length': '0'} 2023-10-17T18:53:54.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63e17f654398d69-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.151 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.152 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.152 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.152 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.153 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.153 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.154 DEBUG:boto:Data: 2023-10-17T18:53:54.154 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.155 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.155 DEBUG:boto:Params: {} 2023-10-17T18:53:54.156 DEBUG:boto:Token: None 2023-10-17T18:53:54.156 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rBwr/S0ZLGJS4UKTHLL6XKq94Kk= 2023-10-17T18:53:54.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rBwr/S0ZLGJS4UKTHLL6XKq94Kk=', 'Content-Length': '0'} 2023-10-17T18:53:54.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd7667d39b5e6fa-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:53:54.160 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.161 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.161 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.162 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.162 DEBUG:boto:Method: GET 2023-10-17T18:53:54.163 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.163 DEBUG:boto:Data: 2023-10-17T18:53:54.164 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.165 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.165 DEBUG:boto:Params: {} 2023-10-17T18:53:54.166 DEBUG:boto:Token: None 2023-10-17T18:53:54.166 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI= 2023-10-17T18:53:54.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI=', 'Content-Length': '0'} 2023-10-17T18:53:54.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d5e8c259f129d2-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.170 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.171 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.171 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.171 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.172 DEBUG:boto:Method: GET 2023-10-17T18:53:54.172 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.173 DEBUG:boto:Data: 2023-10-17T18:53:54.173 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.173 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.174 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.174 DEBUG:boto:Params: {} 2023-10-17T18:53:54.175 DEBUG:boto:Token: None 2023-10-17T18:53:54.175 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:53:54.176 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI= 2023-10-17T18:53:54.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oqLrA/2Nct86szQM50pGz/PrYAI=', 'Content-Length': '0'} 2023-10-17T18:53:54.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97d74c9710a3de9-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.179 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.179 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.180 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.180 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.181 DEBUG:boto:Method: GET 2023-10-17T18:53:54.181 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.181 DEBUG:boto:Data: 2023-10-17T18:53:54.182 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.185 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.186 DEBUG:boto:Params: {} 2023-10-17T18:53:54.186 DEBUG:boto:Token: None 2023-10-17T18:53:54.187 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387baec61e30d61d-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.191 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.192 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.192 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.193 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.193 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.194 DEBUG:boto:Method: GET 2023-10-17T18:53:54.194 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.195 DEBUG:boto:Data: 2023-10-17T18:53:54.195 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.196 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.196 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.197 DEBUG:boto:Params: {} 2023-10-17T18:53:54.197 DEBUG:boto:Token: None 2023-10-17T18:53:54.197 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.198 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005482f948f36dc762-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.201 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.202 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.203 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.203 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.204 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.204 DEBUG:boto:Data: 2023-10-17T18:53:54.204 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.205 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.205 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.206 DEBUG:boto:Params: {} 2023-10-17T18:53:54.206 DEBUG:boto:Token: None 2023-10-17T18:53:54.207 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.207 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wAvQ7ptI4LWd2QRm4cmSfY6VREs= 2023-10-17T18:53:54.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wAvQ7ptI4LWd2QRm4cmSfY6VREs=', 'Content-Length': '0'} 2023-10-17T18:53:54.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72f73897bda2f75-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.210 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.211 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.211 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.211 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.212 DEBUG:boto:Data: 2023-10-17T18:53:54.212 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.213 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.214 DEBUG:boto:Params: {} 2023-10-17T18:53:54.214 DEBUG:boto:Token: None 2023-10-17T18:53:54.215 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wAvQ7ptI4LWd2QRm4cmSfY6VREs= 2023-10-17T18:53:54.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wAvQ7ptI4LWd2QRm4cmSfY6VREs=', 'Content-Length': '0'} 2023-10-17T18:53:54.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7f92e1af0d3cf5-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:53:54.219 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.219 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.219 DEBUG:boto:Method: GET 2023-10-17T18:53:54.220 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.220 DEBUG:boto:Data: 2023-10-17T18:53:54.221 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.221 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.222 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.222 DEBUG:boto:Params: {} 2023-10-17T18:53:54.223 DEBUG:boto:Token: None 2023-10-17T18:53:54.223 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mxtnuHrCoLso6gndnS01/Wk7Ih8= 2023-10-17T18:53:54.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mxtnuHrCoLso6gndnS01/Wk7Ih8=', 'Content-Length': '0'} 2023-10-17T18:53:54.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e704a41827eb87-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.270 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.271 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.271 DEBUG:boto:Method: GET 2023-10-17T18:53:54.272 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.272 DEBUG:boto:Data: 2023-10-17T18:53:54.273 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.273 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.274 DEBUG:boto:Params: {} 2023-10-17T18:53:54.274 DEBUG:boto:Token: None 2023-10-17T18:53:54.275 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:53:54.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mxtnuHrCoLso6gndnS01/Wk7Ih8= 2023-10-17T18:53:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mxtnuHrCoLso6gndnS01/Wk7Ih8=', 'Content-Length': '0'} 2023-10-17T18:53:54.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4be4cde408ef97-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.279 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.279 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.280 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.280 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.281 DEBUG:boto:Method: GET 2023-10-17T18:53:54.281 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.282 DEBUG:boto:Data: 2023-10-17T18:53:54.282 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.283 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.283 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.284 DEBUG:boto:Params: {} 2023-10-17T18:53:54.284 DEBUG:boto:Token: None 2023-10-17T18:53:54.285 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.285 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba5539aaf927f8bf-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.289 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.290 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.290 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.290 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.291 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.291 DEBUG:boto:Method: GET 2023-10-17T18:53:54.292 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:53:54.292 DEBUG:boto:Data: 2023-10-17T18:53:54.293 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.293 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.293 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.294 DEBUG:boto:Params: {} 2023-10-17T18:53:54.294 DEBUG:boto:Token: None 2023-10-17T18:53:54.295 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539df6e601026cd9-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.299 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:53:42.402Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:53:54.300 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.300 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.300 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.301 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.301 DEBUG:boto:Method: GET 2023-10-17T18:53:54.302 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.302 DEBUG:boto:Data: 2023-10-17T18:53:54.303 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.304 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.304 DEBUG:boto:Params: {} 2023-10-17T18:53:54.304 DEBUG:boto:Token: None 2023-10-17T18:53:54.305 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY= 2023-10-17T18:53:54.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY=', 'Content-Length': '0'} 2023-10-17T18:53:54.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d7e2003f105c55-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.309 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.309 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.310 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.310 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.311 DEBUG:boto:Method: GET 2023-10-17T18:53:54.311 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.312 DEBUG:boto:Data: 2023-10-17T18:53:54.312 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.313 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.313 DEBUG:boto:Params: {} 2023-10-17T18:53:54.314 DEBUG:boto:Token: None 2023-10-17T18:53:54.314 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY= 2023-10-17T18:53:54.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY=', 'Content-Length': '0'} 2023-10-17T18:53:54.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b4874d956bf6dc8-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.362 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.363 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.363 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.364 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.364 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.365 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.365 DEBUG:boto:Data: 2023-10-17T18:53:54.365 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.366 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.367 DEBUG:boto:Params: {} 2023-10-17T18:53:54.367 DEBUG:boto:Token: None 2023-10-17T18:53:54.368 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vCL34P+innA3iRabxFgO6xtsQaY= 2023-10-17T18:53:54.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vCL34P+innA3iRabxFgO6xtsQaY=', 'Content-Length': '0'} 2023-10-17T18:53:54.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b56acd3a1bd926-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.371 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.372 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.372 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.373 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.373 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.373 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.374 DEBUG:boto:Data: 2023-10-17T18:53:54.374 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.375 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.375 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.376 DEBUG:boto:Params: {} 2023-10-17T18:53:54.376 DEBUG:boto:Token: None 2023-10-17T18:53:54.377 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vCL34P+innA3iRabxFgO6xtsQaY= 2023-10-17T18:53:54.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vCL34P+innA3iRabxFgO6xtsQaY=', 'Content-Length': '0'} 2023-10-17T18:53:54.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba5dfb43f6e20d0-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:53:54.380 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.381 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.381 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.382 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.382 DEBUG:boto:Method: GET 2023-10-17T18:53:54.383 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.383 DEBUG:boto:Data: 2023-10-17T18:53:54.383 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.384 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.385 DEBUG:boto:Params: {} 2023-10-17T18:53:54.385 DEBUG:boto:Token: None 2023-10-17T18:53:54.386 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.386 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY= 2023-10-17T18:53:54.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY=', 'Content-Length': '0'} 2023-10-17T18:53:54.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269975c52fa084d5-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.390 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.390 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.391 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.391 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.392 DEBUG:boto:Method: GET 2023-10-17T18:53:54.392 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.393 DEBUG:boto:Data: 2023-10-17T18:53:54.393 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.394 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.394 DEBUG:boto:Params: {} 2023-10-17T18:53:54.395 DEBUG:boto:Token: None 2023-10-17T18:53:54.395 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:53:54.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY= 2023-10-17T18:53:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tH1ajz+AD2IA14SouTnkwJqBXdY=', 'Content-Length': '0'} 2023-10-17T18:53:54.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd60d979ed88b6b-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.399 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.400 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.400 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.400 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.401 DEBUG:boto:Method: GET 2023-10-17T18:53:54.401 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.402 DEBUG:boto:Data: 2023-10-17T18:53:54.402 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.403 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.403 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.404 DEBUG:boto:Params: {} 2023-10-17T18:53:54.404 DEBUG:boto:Token: None 2023-10-17T18:53:54.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07e3bf19d2db120-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.408 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.409 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.410 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.410 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.411 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.411 DEBUG:boto:Method: GET 2023-10-17T18:53:54.411 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.412 DEBUG:boto:Data: 2023-10-17T18:53:54.412 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.413 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.414 DEBUG:boto:Params: {} 2023-10-17T18:53:54.415 DEBUG:boto:Token: None 2023-10-17T18:53:54.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da90c1cd4122085-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.419 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.420 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.420 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.421 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.421 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.421 DEBUG:boto:Data: 2023-10-17T18:53:54.422 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.423 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.423 DEBUG:boto:Params: {} 2023-10-17T18:53:54.424 DEBUG:boto:Token: None 2023-10-17T18:53:54.424 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gjwp76ulO+fGSSnCyxcM0RYqy+s= 2023-10-17T18:53:54.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gjwp76ulO+fGSSnCyxcM0RYqy+s=', 'Content-Length': '0'} 2023-10-17T18:53:54.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a0c7ad92ab3c46-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.428 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.428 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.429 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.429 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.429 DEBUG:boto:Data: 2023-10-17T18:53:54.430 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.431 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.431 DEBUG:boto:Params: {} 2023-10-17T18:53:54.432 DEBUG:boto:Token: None 2023-10-17T18:53:54.432 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.433 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gjwp76ulO+fGSSnCyxcM0RYqy+s= 2023-10-17T18:53:54.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gjwp76ulO+fGSSnCyxcM0RYqy+s=', 'Content-Length': '0'} 2023-10-17T18:53:54.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019282e155532ae2b-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:53:54.436 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.436 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.437 DEBUG:boto:Method: GET 2023-10-17T18:53:54.437 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.437 DEBUG:boto:Data: 2023-10-17T18:53:54.438 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.439 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.439 DEBUG:boto:Params: {} 2023-10-17T18:53:54.440 DEBUG:boto:Token: None 2023-10-17T18:53:54.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vinl7U24eYM2hFofzDk2WKFQlD0= 2023-10-17T18:53:54.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vinl7U24eYM2hFofzDk2WKFQlD0=', 'Content-Length': '0'} 2023-10-17T18:53:54.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8e4fd4a9fd16386-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.486 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.487 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.487 DEBUG:boto:Method: GET 2023-10-17T18:53:54.488 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.488 DEBUG:boto:Data: 2023-10-17T18:53:54.489 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.490 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.490 DEBUG:boto:Params: {} 2023-10-17T18:53:54.491 DEBUG:boto:Token: None 2023-10-17T18:53:54.491 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:53:54.492 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vinl7U24eYM2hFofzDk2WKFQlD0= 2023-10-17T18:53:54.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vinl7U24eYM2hFofzDk2WKFQlD0=', 'Content-Length': '0'} 2023-10-17T18:53:54.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002284394e238c213f-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.495 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.496 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.496 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.497 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.497 DEBUG:boto:Method: GET 2023-10-17T18:53:54.498 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.498 DEBUG:boto:Data: 2023-10-17T18:53:54.498 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.499 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.500 DEBUG:boto:Params: {} 2023-10-17T18:53:54.500 DEBUG:boto:Token: None 2023-10-17T18:53:54.501 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e00785329b64588-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.505 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.505 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.506 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.506 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.507 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.507 DEBUG:boto:Method: GET 2023-10-17T18:53:54.507 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:53:54.508 DEBUG:boto:Data: 2023-10-17T18:53:54.508 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.509 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.509 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.510 DEBUG:boto:Params: {} 2023-10-17T18:53:54.510 DEBUG:boto:Token: None 2023-10-17T18:53:54.511 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.512 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf76ca89c236768-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.515 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:53:42.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:53:54.516 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.517 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.517 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.518 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.518 DEBUG:boto:Method: GET 2023-10-17T18:53:54.519 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.519 DEBUG:boto:Data: 2023-10-17T18:53:54.520 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.521 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.521 DEBUG:boto:Params: {} 2023-10-17T18:53:54.522 DEBUG:boto:Token: None 2023-10-17T18:53:54.522 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI= 2023-10-17T18:53:54.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI=', 'Content-Length': '0'} 2023-10-17T18:53:54.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9ad033dd4c2034-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.526 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.527 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.527 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.528 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.528 DEBUG:boto:Method: GET 2023-10-17T18:53:54.529 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.529 DEBUG:boto:Data: 2023-10-17T18:53:54.530 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.531 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.531 DEBUG:boto:Params: {} 2023-10-17T18:53:54.532 DEBUG:boto:Token: None 2023-10-17T18:53:54.532 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI= 2023-10-17T18:53:54.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI=', 'Content-Length': '0'} 2023-10-17T18:53:54.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3400b1cd65121f-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.536 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.537 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.537 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.537 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.538 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.538 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.539 DEBUG:boto:Data: 2023-10-17T18:53:54.539 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.540 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.540 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.541 DEBUG:boto:Params: {} 2023-10-17T18:53:54.541 DEBUG:boto:Token: None 2023-10-17T18:53:54.542 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.542 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b2OA9zhcCgcssXs0XsNLNAQAs5o= 2023-10-17T18:53:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b2OA9zhcCgcssXs0XsNLNAQAs5o=', 'Content-Length': '0'} 2023-10-17T18:53:54.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026054487af9ba8b-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.545 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.546 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.546 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.547 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.547 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.548 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.548 DEBUG:boto:Data: 2023-10-17T18:53:54.549 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.550 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.550 DEBUG:boto:Params: {} 2023-10-17T18:53:54.551 DEBUG:boto:Token: None 2023-10-17T18:53:54.551 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b2OA9zhcCgcssXs0XsNLNAQAs5o= 2023-10-17T18:53:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b2OA9zhcCgcssXs0XsNLNAQAs5o=', 'Content-Length': '0'} 2023-10-17T18:53:54.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25e7b61e875947a-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:53:54.555 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.556 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.556 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.557 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.557 DEBUG:boto:Method: GET 2023-10-17T18:53:54.558 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.558 DEBUG:boto:Data: 2023-10-17T18:53:54.559 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.559 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.560 DEBUG:boto:Params: {} 2023-10-17T18:53:54.560 DEBUG:boto:Token: None 2023-10-17T18:53:54.561 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI= 2023-10-17T18:53:54.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI=', 'Content-Length': '0'} 2023-10-17T18:53:54.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afce0d58c267283f-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.564 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.565 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.565 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.566 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.566 DEBUG:boto:Method: GET 2023-10-17T18:53:54.567 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.567 DEBUG:boto:Data: 2023-10-17T18:53:54.568 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.568 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.569 DEBUG:boto:Params: {} 2023-10-17T18:53:54.569 DEBUG:boto:Token: None 2023-10-17T18:53:54.570 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:53:54.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI= 2023-10-17T18:53:54.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:glUMd/JdRCItyBxBiEI+6ZRbvbI=', 'Content-Length': '0'} 2023-10-17T18:53:54.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a75f23e910e35538-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.574 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.574 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.575 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.575 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.575 DEBUG:boto:Method: GET 2023-10-17T18:53:54.576 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.576 DEBUG:boto:Data: 2023-10-17T18:53:54.577 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.578 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.578 DEBUG:boto:Params: {} 2023-10-17T18:53:54.578 DEBUG:boto:Token: None 2023-10-17T18:53:54.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6fa0424b4669883-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.583 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.584 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.584 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.585 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.585 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.586 DEBUG:boto:Method: GET 2023-10-17T18:53:54.586 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.586 DEBUG:boto:Data: 2023-10-17T18:53:54.587 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.588 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.588 DEBUG:boto:Params: {} 2023-10-17T18:53:54.589 DEBUG:boto:Token: None 2023-10-17T18:53:54.589 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.590 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecba4335be3f92eb-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.593 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.594 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.594 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.595 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.595 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.596 DEBUG:boto:Data: 2023-10-17T18:53:54.596 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.597 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.597 DEBUG:boto:Params: {} 2023-10-17T18:53:54.598 DEBUG:boto:Token: None 2023-10-17T18:53:54.598 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.599 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WQx2wP+ikMronT9cGtf0A2VmcCQ= 2023-10-17T18:53:54.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WQx2wP+ikMronT9cGtf0A2VmcCQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eba47b5b48d4af7-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.602 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.602 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.603 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.603 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.604 DEBUG:boto:Data: 2023-10-17T18:53:54.604 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.604 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.605 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.605 DEBUG:boto:Params: {} 2023-10-17T18:53:54.605 DEBUG:boto:Token: None 2023-10-17T18:53:54.606 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.606 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WQx2wP+ikMronT9cGtf0A2VmcCQ= 2023-10-17T18:53:54.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WQx2wP+ikMronT9cGtf0A2VmcCQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ef193e25084fb5-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:53:54.610 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.610 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.611 DEBUG:boto:Method: GET 2023-10-17T18:53:54.611 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.612 DEBUG:boto:Data: 2023-10-17T18:53:54.612 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.613 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.613 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.614 DEBUG:boto:Params: {} 2023-10-17T18:53:54.614 DEBUG:boto:Token: None 2023-10-17T18:53:54.615 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.615 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xfIyWNK25jDIZcpvrO/h2rrkgcA= 2023-10-17T18:53:54.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xfIyWNK25jDIZcpvrO/h2rrkgcA=', 'Content-Length': '0'} 2023-10-17T18:53:54.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec176c3a3b36453-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.618 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.619 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.619 DEBUG:boto:Method: GET 2023-10-17T18:53:54.620 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.620 DEBUG:boto:Data: 2023-10-17T18:53:54.621 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.621 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.622 DEBUG:boto:Params: {} 2023-10-17T18:53:54.622 DEBUG:boto:Token: None 2023-10-17T18:53:54.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:53:54.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xfIyWNK25jDIZcpvrO/h2rrkgcA= 2023-10-17T18:53:54.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xfIyWNK25jDIZcpvrO/h2rrkgcA=', 'Content-Length': '0'} 2023-10-17T18:53:54.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a52f067acff2320-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.626 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.627 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.627 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.628 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.628 DEBUG:boto:Method: GET 2023-10-17T18:53:54.629 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.629 DEBUG:boto:Data: 2023-10-17T18:53:54.630 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.630 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.630 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.631 DEBUG:boto:Params: {} 2023-10-17T18:53:54.631 DEBUG:boto:Token: None 2023-10-17T18:53:54.632 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.632 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073dc3d0130e32605-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.636 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.636 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.637 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.637 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.638 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.638 DEBUG:boto:Method: GET 2023-10-17T18:53:54.639 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:53:54.639 DEBUG:boto:Data: 2023-10-17T18:53:54.639 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.640 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.641 DEBUG:boto:Params: {} 2023-10-17T18:53:54.641 DEBUG:boto:Token: None 2023-10-17T18:53:54.641 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e40fe35a0df28215-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.646 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:53:42.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:53:54.647 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.647 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.648 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.648 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.648 DEBUG:boto:Method: GET 2023-10-17T18:53:54.649 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.649 DEBUG:boto:Data: 2023-10-17T18:53:54.650 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.650 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.651 DEBUG:boto:Params: {} 2023-10-17T18:53:54.651 DEBUG:boto:Token: None 2023-10-17T18:53:54.652 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.652 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0= 2023-10-17T18:53:54.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0=', 'Content-Length': '0'} 2023-10-17T18:53:54.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff13f3f2f1c3675-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.698 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.699 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.699 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.700 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.700 DEBUG:boto:Method: GET 2023-10-17T18:53:54.701 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.701 DEBUG:boto:Data: 2023-10-17T18:53:54.701 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.702 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.703 DEBUG:boto:Params: {} 2023-10-17T18:53:54.703 DEBUG:boto:Token: None 2023-10-17T18:53:54.704 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0= 2023-10-17T18:53:54.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0=', 'Content-Length': '0'} 2023-10-17T18:53:54.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eea72ffc8a09c23-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.707 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.708 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.708 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.709 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.709 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.709 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.710 DEBUG:boto:Data: 2023-10-17T18:53:54.710 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.711 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.712 DEBUG:boto:Params: {} 2023-10-17T18:53:54.712 DEBUG:boto:Token: None 2023-10-17T18:53:54.713 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPjU+oQK6CNbIf8TlVFkWagEFnw= 2023-10-17T18:53:54.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPjU+oQK6CNbIf8TlVFkWagEFnw=', 'Content-Length': '0'} 2023-10-17T18:53:54.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98a08c6c4672a07-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.716 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.717 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.717 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.718 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.718 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.719 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.719 DEBUG:boto:Data: 2023-10-17T18:53:54.719 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.720 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.720 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.721 DEBUG:boto:Params: {} 2023-10-17T18:53:54.721 DEBUG:boto:Token: None 2023-10-17T18:53:54.722 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.722 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPjU+oQK6CNbIf8TlVFkWagEFnw= 2023-10-17T18:53:54.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPjU+oQK6CNbIf8TlVFkWagEFnw=', 'Content-Length': '0'} 2023-10-17T18:53:54.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164c590603a02641-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:53:54.726 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.726 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.727 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.727 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.728 DEBUG:boto:Method: GET 2023-10-17T18:53:54.728 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.729 DEBUG:boto:Data: 2023-10-17T18:53:54.729 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.730 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.730 DEBUG:boto:Params: {} 2023-10-17T18:53:54.731 DEBUG:boto:Token: None 2023-10-17T18:53:54.731 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0= 2023-10-17T18:53:54.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0=', 'Content-Length': '0'} 2023-10-17T18:53:54.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa628f0bafb2f7c8-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.735 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.735 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.735 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.736 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.736 DEBUG:boto:Method: GET 2023-10-17T18:53:54.737 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.737 DEBUG:boto:Data: 2023-10-17T18:53:54.738 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.738 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.739 DEBUG:boto:Params: {} 2023-10-17T18:53:54.739 DEBUG:boto:Token: None 2023-10-17T18:53:54.740 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:53:54.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0= 2023-10-17T18:53:54.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tYoG+cqE4lUSAmo2kORCnUJ/Mx0=', 'Content-Length': '0'} 2023-10-17T18:53:54.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000145fbd2be17a8944-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.786 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.787 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.787 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.788 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.788 DEBUG:boto:Method: GET 2023-10-17T18:53:54.789 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.789 DEBUG:boto:Data: 2023-10-17T18:53:54.790 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.790 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.791 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.791 DEBUG:boto:Params: {} 2023-10-17T18:53:54.792 DEBUG:boto:Token: None 2023-10-17T18:53:54.792 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602ecdc340d3ee44-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.796 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.797 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.797 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.797 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.798 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.798 DEBUG:boto:Method: GET 2023-10-17T18:53:54.799 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.799 DEBUG:boto:Data: 2023-10-17T18:53:54.800 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.800 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.801 DEBUG:boto:Params: {} 2023-10-17T18:53:54.801 DEBUG:boto:Token: None 2023-10-17T18:53:54.802 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002178db38851ded76-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.806 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.806 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.807 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.807 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.808 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.808 DEBUG:boto:Data: 2023-10-17T18:53:54.809 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.810 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.810 DEBUG:boto:Params: {} 2023-10-17T18:53:54.810 DEBUG:boto:Token: None 2023-10-17T18:53:54.811 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PkVH94ROmoUAtkPHLWFzHOX7D7I= 2023-10-17T18:53:54.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PkVH94ROmoUAtkPHLWFzHOX7D7I=', 'Content-Length': '0'} 2023-10-17T18:53:54.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e262f906960e09ba-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.815 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.815 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.816 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.816 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.816 DEBUG:boto:Data: 2023-10-17T18:53:54.817 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.818 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.818 DEBUG:boto:Params: {} 2023-10-17T18:53:54.819 DEBUG:boto:Token: None 2023-10-17T18:53:54.819 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PkVH94ROmoUAtkPHLWFzHOX7D7I= 2023-10-17T18:53:54.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PkVH94ROmoUAtkPHLWFzHOX7D7I=', 'Content-Length': '0'} 2023-10-17T18:53:54.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075db2317b44f2e12-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:53:54.823 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.823 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.824 DEBUG:boto:Method: GET 2023-10-17T18:53:54.824 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.825 DEBUG:boto:Data: 2023-10-17T18:53:54.825 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.825 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.826 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.826 DEBUG:boto:Params: {} 2023-10-17T18:53:54.827 DEBUG:boto:Token: None 2023-10-17T18:53:54.827 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NfyTE9QV2B1n4aYkz8AZIFR5AE= 2023-10-17T18:53:54.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NfyTE9QV2B1n4aYkz8AZIFR5AE=', 'Content-Length': '0'} 2023-10-17T18:53:54.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef70fc4487c3c10a-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.831 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.832 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.832 DEBUG:boto:Method: GET 2023-10-17T18:53:54.832 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.833 DEBUG:boto:Data: 2023-10-17T18:53:54.833 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.834 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.834 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.834 DEBUG:boto:Params: {} 2023-10-17T18:53:54.835 DEBUG:boto:Token: None 2023-10-17T18:53:54.835 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:53:54.836 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NfyTE9QV2B1n4aYkz8AZIFR5AE= 2023-10-17T18:53:54.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NfyTE9QV2B1n4aYkz8AZIFR5AE=', 'Content-Length': '0'} 2023-10-17T18:53:54.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030fbba2af90886f8-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.839 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.839 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.840 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.840 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.841 DEBUG:boto:Method: GET 2023-10-17T18:53:54.841 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.841 DEBUG:boto:Data: 2023-10-17T18:53:54.842 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.842 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.843 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.843 DEBUG:boto:Params: {} 2023-10-17T18:53:54.844 DEBUG:boto:Token: None 2023-10-17T18:53:54.844 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021d979c9922ff6c3-00652ed842-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.890 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.891 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:54.892 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:54.892 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.893 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.893 DEBUG:boto:Method: GET 2023-10-17T18:53:54.893 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:53:54.894 DEBUG:boto:Data: 2023-10-17T18:53:54.894 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.895 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.895 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.895 DEBUG:boto:Params: {} 2023-10-17T18:53:54.896 DEBUG:boto:Token: None 2023-10-17T18:53:54.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/ 2023-10-17T18:53:54.897 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ= 2023-10-17T18:53:54.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gkVGxSt5ORIVtQhO76+qxrSGYdQ=', 'Content-Length': '0'} 2023-10-17T18:53:54.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506ff99b0d19d0e8-00652ed842-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.900 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:53:42.558Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:54.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:53:54.901 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.902 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.902 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.903 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.903 DEBUG:boto:Method: GET 2023-10-17T18:53:54.904 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.904 DEBUG:boto:Data: 2023-10-17T18:53:54.905 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.905 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.906 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.906 DEBUG:boto:Params: {} 2023-10-17T18:53:54.907 DEBUG:boto:Token: None 2023-10-17T18:53:54.907 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo= 2023-10-17T18:53:54.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo=', 'Content-Length': '0'} 2023-10-17T18:53:54.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afcd20db491e9377-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.911 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.911 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.912 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.912 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.913 DEBUG:boto:Method: GET 2023-10-17T18:53:54.913 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.914 DEBUG:boto:Data: 2023-10-17T18:53:54.914 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.914 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.915 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.915 DEBUG:boto:Params: {} 2023-10-17T18:53:54.916 DEBUG:boto:Token: None 2023-10-17T18:53:54.916 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo= 2023-10-17T18:53:54.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo=', 'Content-Length': '0'} 2023-10-17T18:53:54.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b68303bdccd552a-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.920 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.921 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.921 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.921 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.922 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.922 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.923 DEBUG:boto:Data: 2023-10-17T18:53:54.923 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.924 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.924 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.925 DEBUG:boto:Params: {} 2023-10-17T18:53:54.925 DEBUG:boto:Token: None 2023-10-17T18:53:54.926 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.926 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UKNVww1uHlJ46BnIE5nV02dZ9b8= 2023-10-17T18:53:54.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UKNVww1uHlJ46BnIE5nV02dZ9b8=', 'Content-Length': '0'} 2023-10-17T18:53:54.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc44eac2504243e-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.929 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.930 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.930 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.931 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.931 DEBUG:boto:Method: HEAD 2023-10-17T18:53:54.931 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.932 DEBUG:boto:Data: 2023-10-17T18:53:54.932 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.933 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.933 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.933 DEBUG:boto:Params: {} 2023-10-17T18:53:54.934 DEBUG:boto:Token: None 2023-10-17T18:53:54.934 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.935 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UKNVww1uHlJ46BnIE5nV02dZ9b8= 2023-10-17T18:53:54.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UKNVww1uHlJ46BnIE5nV02dZ9b8=', 'Content-Length': '0'} 2023-10-17T18:53:54.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203dd9a3783ff67b-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:53:54.938 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.939 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.939 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.939 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.940 DEBUG:boto:Method: GET 2023-10-17T18:53:54.940 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.941 DEBUG:boto:Data: 2023-10-17T18:53:54.941 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:54.942 DEBUG:boto:Port: 8001 2023-10-17T18:53:54.943 DEBUG:boto:Params: {} 2023-10-17T18:53:54.943 DEBUG:boto:Token: None 2023-10-17T18:53:54.943 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.944 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo= 2023-10-17T18:53:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo=', 'Content-Length': '0'} 2023-10-17T18:53:54.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f787b1247ce2b216-00652ed842-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:54.947 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.948 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:54.948 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.949 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.949 DEBUG:boto:Method: GET 2023-10-17T18:53:54.950 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.950 DEBUG:boto:Data: 2023-10-17T18:53:54.950 DEBUG:boto:Headers: {} 2023-10-17T18:53:54.951 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:54.951 DEBUG:boto:Port: 8000 2023-10-17T18:53:54.952 DEBUG:boto:Params: {} 2023-10-17T18:53:54.952 DEBUG:boto:Token: None 2023-10-17T18:53:54.953 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:54 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:53:54.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo= 2023-10-17T18:53:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kaj1X4OjoAjiZtPO0lQ8B358ljo=', 'Content-Length': '0'} 2023-10-17T18:53:54.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887f06e275403c86-00652ed842-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.002 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:55.003 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:55.003 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.004 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.004 DEBUG:boto:Method: GET 2023-10-17T18:53:55.005 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.005 DEBUG:boto:Data: 2023-10-17T18:53:55.006 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.006 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:55.007 DEBUG:boto:Port: 8001 2023-10-17T18:53:55.007 DEBUG:boto:Params: {} 2023-10-17T18:53:55.007 DEBUG:boto:Token: None 2023-10-17T18:53:55.008 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/ 2023-10-17T18:53:55.008 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU= 2023-10-17T18:53:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU=', 'Content-Length': '0'} 2023-10-17T18:53:55.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c6e8dd0ea88a8e1-00652ed843-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.012 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:55.013 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:55.014 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:55.014 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.015 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.015 DEBUG:boto:Method: GET 2023-10-17T18:53:55.016 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.016 DEBUG:boto:Data: 2023-10-17T18:53:55.016 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:55.017 DEBUG:boto:Port: 8000 2023-10-17T18:53:55.018 DEBUG:boto:Params: {} 2023-10-17T18:53:55.018 DEBUG:boto:Token: None 2023-10-17T18:53:55.019 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/ 2023-10-17T18:53:55.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU= 2023-10-17T18:53:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU=', 'Content-Length': '0'} 2023-10-17T18:53:55.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3134bfe0379a5e2-00652ed843-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.023 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:55.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.024 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.025 DEBUG:boto:Method: HEAD 2023-10-17T18:53:55.025 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.026 DEBUG:boto:Data: 2023-10-17T18:53:55.026 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:55.027 DEBUG:boto:Port: 8001 2023-10-17T18:53:55.027 DEBUG:boto:Params: {} 2023-10-17T18:53:55.028 DEBUG:boto:Token: None 2023-10-17T18:53:55.028 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P1SPp7EEan6Al0EFAkPE19ac1XY= 2023-10-17T18:53:55.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P1SPp7EEan6Al0EFAkPE19ac1XY=', 'Content-Length': '0'} 2023-10-17T18:53:55.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ffa6d5c6467271-00652ed843-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.032 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.032 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.033 DEBUG:boto:Method: HEAD 2023-10-17T18:53:55.033 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.034 DEBUG:boto:Data: 2023-10-17T18:53:55.034 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:55.035 DEBUG:boto:Port: 8000 2023-10-17T18:53:55.035 DEBUG:boto:Params: {} 2023-10-17T18:53:55.036 DEBUG:boto:Token: None 2023-10-17T18:53:55.036 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P1SPp7EEan6Al0EFAkPE19ac1XY= 2023-10-17T18:53:55.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P1SPp7EEan6Al0EFAkPE19ac1XY=', 'Content-Length': '0'} 2023-10-17T18:53:55.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002353b58b0c1942b6-00652ed843-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:53:55.040 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.041 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.041 DEBUG:boto:Method: GET 2023-10-17T18:53:55.042 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.042 DEBUG:boto:Data: 2023-10-17T18:53:55.043 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.043 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:55.044 DEBUG:boto:Port: 8001 2023-10-17T18:53:55.044 DEBUG:boto:Params: {} 2023-10-17T18:53:55.045 DEBUG:boto:Token: None 2023-10-17T18:53:55.045 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.046 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jNpbzgjrli3dpYjp2Wdn7XJQbuY= 2023-10-17T18:53:55.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jNpbzgjrli3dpYjp2Wdn7XJQbuY=', 'Content-Length': '0'} 2023-10-17T18:53:55.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b47c6bafd717046-00652ed843-1a76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.090 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.091 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.091 DEBUG:boto:Method: GET 2023-10-17T18:53:55.092 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.092 DEBUG:boto:Data: 2023-10-17T18:53:55.093 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.093 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:55.094 DEBUG:boto:Port: 8000 2023-10-17T18:53:55.094 DEBUG:boto:Params: {} 2023-10-17T18:53:55.094 DEBUG:boto:Token: None 2023-10-17T18:53:55.095 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:53:55.095 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jNpbzgjrli3dpYjp2Wdn7XJQbuY= 2023-10-17T18:53:55.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jNpbzgjrli3dpYjp2Wdn7XJQbuY=', 'Content-Length': '0'} 2023-10-17T18:53:55.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:53:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b849590acdc7abb-00652ed843-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.099 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:55.100 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:55.100 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.100 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.101 DEBUG:boto:Method: GET 2023-10-17T18:53:55.101 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.102 DEBUG:boto:Data: 2023-10-17T18:53:55.102 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:53:55.103 DEBUG:boto:Port: 8001 2023-10-17T18:53:55.103 DEBUG:boto:Params: {} 2023-10-17T18:53:55.104 DEBUG:boto:Token: None 2023-10-17T18:53:55.104 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/ 2023-10-17T18:53:55.105 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU= 2023-10-17T18:53:55.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU=', 'Content-Length': '0'} 2023-10-17T18:53:55.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002efaf808017c14e-00652ed843-1a76-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.108 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:55.109 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:53:55.110 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:53:55.110 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.111 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.111 DEBUG:boto:Method: GET 2023-10-17T18:53:55.112 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:53:55.112 DEBUG:boto:Data: 2023-10-17T18:53:55.112 DEBUG:boto:Headers: {} 2023-10-17T18:53:55.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:53:55.113 DEBUG:boto:Port: 8000 2023-10-17T18:53:55.114 DEBUG:boto:Params: {} 2023-10-17T18:53:55.114 DEBUG:boto:Token: None 2023-10-17T18:53:55.115 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:53:55 GMT /hbmuog-58/ 2023-10-17T18:53:55.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU= 2023-10-17T18:53:55.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:53:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zVtomtLti6Czi1CKUJKRQtPKRU=', 'Content-Length': '0'} 2023-10-17T18:53:55.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98860e9eb0793bb-00652ed843-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:53:55.158 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:53:42.482Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:53:55.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-58 zones={a2, a1} 2023-10-17T18:53:55.160 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T18:53:55.160 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T18:53:55.161 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:53:55.160+0000 7f20124f6700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 156935) UID: 0 2023-10-17T18:53:55.162 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T18:53:55.160+0000 7f201675f7c0 -1 shutting down 2023-10-17T18:53:57.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T18:54:01.260 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T18:54:01.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:01.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:01.261 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:01.372 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:01.372 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:01.376 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 10 2023-10-17T18:54:01.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:01.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:01.415 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:01.468 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:01.469 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:01.509 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000007", 2023-10-17T18:54:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000016", 2023-10-17T18:54:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000024", 2023-10-17T18:54:01.510 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000020", 2023-10-17T18:54:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000028", 2023-10-17T18:54:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000032", 2023-10-17T18:54:01.511 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:54:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000019", 2023-10-17T18:54:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000013", 2023-10-17T18:54:01.512 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021", 2023-10-17T18:54:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000030", 2023-10-17T18:54:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000026", 2023-10-17T18:54:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000033", 2023-10-17T18:54:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:54:01.513 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000076" 2023-10-17T18:54:01.514 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:01.514 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:01.515 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.516 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.516 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.516 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.517 DEBUG:boto:Data: 2023-10-17T18:54:01.517 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.518 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.519 DEBUG:boto:Params: {} 2023-10-17T18:54:01.519 DEBUG:boto:Token: None 2023-10-17T18:54:01.520 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc3ea052362801c0-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.524 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:01.525 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:01.525 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.525 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:01.526 DEBUG:boto:Data: 2023-10-17T18:54:01.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.527 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.527 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.527 DEBUG:boto:Params: {} 2023-10-17T18:54:01.528 DEBUG:boto:Token: None 2023-10-17T18:54:01.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:01.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cmTQtoWB4PRGms/wi3jV2PrsC2w= 2023-10-17T18:54:01.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cmTQtoWB4PRGms/wi3jV2PrsC2w='} 2023-10-17T18:54:01.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7e30e83d8138e8f-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.535 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.535 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.536 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.536 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.536 DEBUG:boto:Data: 2023-10-17T18:54:01.537 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.537 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.538 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.538 DEBUG:boto:Params: {} 2023-10-17T18:54:01.538 DEBUG:boto:Token: None 2023-10-17T18:54:01.539 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000613008f0ac7752f3-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.543 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:01.544 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:01.544 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.544 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:01.545 DEBUG:boto:Data: 2023-10-17T18:54:01.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.546 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.546 DEBUG:boto:Params: {} 2023-10-17T18:54:01.547 DEBUG:boto:Token: None 2023-10-17T18:54:01.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:01.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:32napvWUF9u2KQLtwc0CW5HL8oo= 2023-10-17T18:54:01.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:32napvWUF9u2KQLtwc0CW5HL8oo='} 2023-10-17T18:54:01.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9da10cb6fa45cdc-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.554 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.555 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.555 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.556 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.556 DEBUG:boto:Data: 2023-10-17T18:54:01.557 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.557 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.558 DEBUG:boto:Params: {} 2023-10-17T18:54:01.558 DEBUG:boto:Token: None 2023-10-17T18:54:01.559 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.559 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000163cb7807f74251a-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.563 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:01.563 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:01.564 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.564 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:01.565 DEBUG:boto:Data: 2023-10-17T18:54:01.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.566 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.566 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.567 DEBUG:boto:Params: {} 2023-10-17T18:54:01.567 DEBUG:boto:Token: None 2023-10-17T18:54:01.567 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:01.568 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PavuvWouL1SWSboEWzwsEJK6W6c= 2023-10-17T18:54:01.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PavuvWouL1SWSboEWzwsEJK6W6c='} 2023-10-17T18:54:01.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb1ccf48b469c7c1-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.611 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.611 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.612 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.612 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.613 DEBUG:boto:Data: 2023-10-17T18:54:01.613 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.613 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.614 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.614 DEBUG:boto:Params: {} 2023-10-17T18:54:01.615 DEBUG:boto:Token: None 2023-10-17T18:54:01.615 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e730e1b8d7b7d83-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.619 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:01.620 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:01.620 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.621 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:01.621 DEBUG:boto:Data: 2023-10-17T18:54:01.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.623 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.623 DEBUG:boto:Params: {} 2023-10-17T18:54:01.623 DEBUG:boto:Token: None 2023-10-17T18:54:01.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:01.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ofD/471AhM/2x3gCtUyg1x2WGO4= 2023-10-17T18:54:01.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ofD/471AhM/2x3gCtUyg1x2WGO4='} 2023-10-17T18:54:01.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eaf99c94274ac1f0-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.631 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.631 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.631 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.632 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.632 DEBUG:boto:Data: 2023-10-17T18:54:01.633 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.633 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.634 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.634 DEBUG:boto:Params: {} 2023-10-17T18:54:01.635 DEBUG:boto:Token: None 2023-10-17T18:54:01.635 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.635 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf8bd1c68ad0193-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.639 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:01.639 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:01.640 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.640 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:01.640 DEBUG:boto:Data: 2023-10-17T18:54:01.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.641 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.642 DEBUG:boto:Params: {} 2023-10-17T18:54:01.642 DEBUG:boto:Token: None 2023-10-17T18:54:01.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:01.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:scIA8KxxIgHQGnkqyjY0d10EKHQ= 2023-10-17T18:54:01.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:scIA8KxxIgHQGnkqyjY0d10EKHQ='} 2023-10-17T18:54:01.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015ef7ccd6417940d-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.687 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.688 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.688 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.688 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.689 DEBUG:boto:Data: 2023-10-17T18:54:01.689 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.691 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.691 DEBUG:boto:Params: {} 2023-10-17T18:54:01.692 DEBUG:boto:Token: None 2023-10-17T18:54:01.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6404a0094766dc6-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.697 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:01.697 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:01.697 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.698 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:01.698 DEBUG:boto:Data: 2023-10-17T18:54:01.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.699 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.700 DEBUG:boto:Params: {} 2023-10-17T18:54:01.700 DEBUG:boto:Token: None 2023-10-17T18:54:01.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:01.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ou3bTgxscuR1Ld+5/ga/dfZDhOw= 2023-10-17T18:54:01.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ou3bTgxscuR1Ld+5/ga/dfZDhOw='} 2023-10-17T18:54:01.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005eadac59bd6747ba-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.707 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.707 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.708 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.708 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.709 DEBUG:boto:Data: 2023-10-17T18:54:01.709 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.710 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.710 DEBUG:boto:Params: {} 2023-10-17T18:54:01.711 DEBUG:boto:Token: None 2023-10-17T18:54:01.711 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8e989e9d5171471-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.716 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:01.716 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:01.717 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.717 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:01.718 DEBUG:boto:Data: 2023-10-17T18:54:01.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.719 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.719 DEBUG:boto:Params: {} 2023-10-17T18:54:01.720 DEBUG:boto:Token: None 2023-10-17T18:54:01.720 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:01.721 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wYht1vHmkRh3jBpCkoztsuDfWyI= 2023-10-17T18:54:01.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wYht1vHmkRh3jBpCkoztsuDfWyI='} 2023-10-17T18:54:01.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000918137db2ef19718-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.767 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.768 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.768 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.769 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.769 DEBUG:boto:Data: 2023-10-17T18:54:01.769 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.770 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.771 DEBUG:boto:Params: {} 2023-10-17T18:54:01.771 DEBUG:boto:Token: None 2023-10-17T18:54:01.772 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a76a1bba30deeaad-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.776 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:01.776 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:01.777 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.777 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:01.777 DEBUG:boto:Data: 2023-10-17T18:54:01.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.778 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.779 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.779 DEBUG:boto:Params: {} 2023-10-17T18:54:01.779 DEBUG:boto:Token: None 2023-10-17T18:54:01.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:01.780 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zz/hJXG6j1Daidxe4DBmTKPO10g= 2023-10-17T18:54:01.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zz/hJXG6j1Daidxe4DBmTKPO10g='} 2023-10-17T18:54:01.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca58054f082c55e0-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.786 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.787 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.787 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.787 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.788 DEBUG:boto:Data: 2023-10-17T18:54:01.788 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.789 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.789 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.789 DEBUG:boto:Params: {} 2023-10-17T18:54:01.790 DEBUG:boto:Token: None 2023-10-17T18:54:01.790 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.791 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e071f3c130cc681-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.794 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:01.795 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:01.795 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.796 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:01.796 DEBUG:boto:Data: 2023-10-17T18:54:01.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.797 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.798 DEBUG:boto:Params: {} 2023-10-17T18:54:01.798 DEBUG:boto:Token: None 2023-10-17T18:54:01.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:01.799 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ur1M4OecjKmA+TGAvY7YQZqHn5Y= 2023-10-17T18:54:01.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ur1M4OecjKmA+TGAvY7YQZqHn5Y='} 2023-10-17T18:54:01.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a7696e8958157f8-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.843 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.843 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.844 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.844 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.844 DEBUG:boto:Data: 2023-10-17T18:54:01.845 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.846 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.846 DEBUG:boto:Params: {} 2023-10-17T18:54:01.846 DEBUG:boto:Token: None 2023-10-17T18:54:01.847 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e9c8b8ad4a8c7d1-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.851 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:01.851 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:01.852 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.852 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:01.853 DEBUG:boto:Data: 2023-10-17T18:54:01.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.853 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.854 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.854 DEBUG:boto:Params: {} 2023-10-17T18:54:01.855 DEBUG:boto:Token: None 2023-10-17T18:54:01.855 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:01.855 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BCUL2rXF7iD9rB+7W2dup0VieDw= 2023-10-17T18:54:01.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BCUL2rXF7iD9rB+7W2dup0VieDw='} 2023-10-17T18:54:01.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087960368751dcc53-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.861 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.862 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.862 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.862 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.863 DEBUG:boto:Data: 2023-10-17T18:54:01.863 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.864 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.864 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.864 DEBUG:boto:Params: {} 2023-10-17T18:54:01.865 DEBUG:boto:Token: None 2023-10-17T18:54:01.865 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.866 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0b71610453a9b51-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.870 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:01.870 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:01.870 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.871 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:01.871 DEBUG:boto:Data: 2023-10-17T18:54:01.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.872 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.873 DEBUG:boto:Params: {} 2023-10-17T18:54:01.873 DEBUG:boto:Token: None 2023-10-17T18:54:01.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:01.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QFgUAsOco5C43iux92KLMqkUbRg= 2023-10-17T18:54:01.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QFgUAsOco5C43iux92KLMqkUbRg='} 2023-10-17T18:54:01.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031e5f05c967786b7-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.919 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.919 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.920 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.920 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.920 DEBUG:boto:Data: 2023-10-17T18:54:01.921 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.921 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.922 DEBUG:boto:Params: {} 2023-10-17T18:54:01.922 DEBUG:boto:Token: None 2023-10-17T18:54:01.923 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85d06d365660487-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.927 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:01.927 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:01.927 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.928 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:01.928 DEBUG:boto:Data: 2023-10-17T18:54:01.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.929 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.929 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.930 DEBUG:boto:Params: {} 2023-10-17T18:54:01.930 DEBUG:boto:Token: None 2023-10-17T18:54:01.931 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:01.931 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I+Y+eGFLyPM9lPGMGhTJkCVMDJw= 2023-10-17T18:54:01.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I+Y+eGFLyPM9lPGMGhTJkCVMDJw='} 2023-10-17T18:54:01.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4826e978c4ed859-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.937 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.937 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.938 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.938 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.939 DEBUG:boto:Data: 2023-10-17T18:54:01.939 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.940 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.940 DEBUG:boto:Params: {} 2023-10-17T18:54:01.941 DEBUG:boto:Token: None 2023-10-17T18:54:01.941 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c92ac1840765dafd-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.945 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:01.945 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:01.946 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.946 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:01.947 DEBUG:boto:Data: 2023-10-17T18:54:01.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.947 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.948 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.948 DEBUG:boto:Params: {} 2023-10-17T18:54:01.949 DEBUG:boto:Token: None 2023-10-17T18:54:01.949 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:01.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQhVKtmiOmsrQfSRi6UCAzDctyI= 2023-10-17T18:54:01.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQhVKtmiOmsrQfSRi6UCAzDctyI='} 2023-10-17T18:54:01.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061289cddde045c6d-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.955 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:01.955 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:01.956 DEBUG:boto:Method: HEAD 2023-10-17T18:54:01.956 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:01.956 DEBUG:boto:Data: 2023-10-17T18:54:01.957 DEBUG:boto:Headers: {} 2023-10-17T18:54:01.957 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.957 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.958 DEBUG:boto:Params: {} 2023-10-17T18:54:01.958 DEBUG:boto:Token: None 2023-10-17T18:54:01.959 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/ 2023-10-17T18:54:01.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk= 2023-10-17T18:54:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbNRNRB7EUhK5YAU4U9kbTjJzuk=', 'Content-Length': '0'} 2023-10-17T18:54:01.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004130c78ace36d6ff-00652ed849-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:01.963 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:01.963 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:01.963 DEBUG:boto:Method: PUT 2023-10-17T18:54:01.964 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:01.964 DEBUG:boto:Data: 2023-10-17T18:54:01.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:01.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:01.965 DEBUG:boto:Port: 8000 2023-10-17T18:54:01.966 DEBUG:boto:Params: {} 2023-10-17T18:54:01.966 DEBUG:boto:Token: None 2023-10-17T18:54:01.967 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:01 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:01.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L2CwZLexfx1LjsS06Mof4LHTz1c= 2023-10-17T18:54:01.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:01 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L2CwZLexfx1LjsS06Mof4LHTz1c='} 2023-10-17T18:54:02.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c42552a14c159f38-00652ed849-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.015 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.015 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.016 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.016 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.017 DEBUG:boto:Data: 2023-10-17T18:54:02.017 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.018 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.018 DEBUG:boto:Params: {} 2023-10-17T18:54:02.019 DEBUG:boto:Token: None 2023-10-17T18:54:02.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdb65e4848f97c4e-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.023 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.024 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.024 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.024 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.025 DEBUG:boto:Data: 2023-10-17T18:54:02.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.026 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.027 DEBUG:boto:Params: {} 2023-10-17T18:54:02.027 DEBUG:boto:Token: None 2023-10-17T18:54:02.027 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.028 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6BXsy6LUy8e85jvj9gYNgkNTGWc= 2023-10-17T18:54:02.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6BXsy6LUy8e85jvj9gYNgkNTGWc='} 2023-10-17T18:54:02.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb62b2be0c02a880-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.035 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.035 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.036 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.036 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.037 DEBUG:boto:Data: 2023-10-17T18:54:02.037 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.038 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.038 DEBUG:boto:Params: {} 2023-10-17T18:54:02.039 DEBUG:boto:Token: None 2023-10-17T18:54:02.039 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.040 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ccdff02bd0ba5a8-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.043 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.044 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.044 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.044 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.045 DEBUG:boto:Data: 2023-10-17T18:54:02.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.046 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.046 DEBUG:boto:Params: {} 2023-10-17T18:54:02.047 DEBUG:boto:Token: None 2023-10-17T18:54:02.047 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jcqoQ/8ceLHE/OStXXi8iBdjwf0= 2023-10-17T18:54:02.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jcqoQ/8ceLHE/OStXXi8iBdjwf0='} 2023-10-17T18:54:02.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5eb0ae6bc87d181-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:02.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:02.092 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:02.149 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:02.191 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:02.192 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:02.198 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 21 2023-10-17T18:54:02.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:02.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:02.242 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:02.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:02.295 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:02.338 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:02.338 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:02.338 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000005", 2023-10-17T18:54:02.338 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:54:02.339 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000016", 2023-10-17T18:54:02.339 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021", 2023-10-17T18:54:02.339 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:54:02.339 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000025", 2023-10-17T18:54:02.340 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000030", 2023-10-17T18:54:02.340 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000015", 2023-10-17T18:54:02.340 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000019", 2023-10-17T18:54:02.340 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:54:02.340 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000009", 2023-10-17T18:54:02.341 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000014", 2023-10-17T18:54:02.341 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:54:02.341 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:54:02.341 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:02.342 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006", 2023-10-17T18:54:02.342 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:54:02.342 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:02.342 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000007" 2023-10-17T18:54:02.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:02.343 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:02.344 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.344 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.345 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.345 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.346 DEBUG:boto:Data: 2023-10-17T18:54:02.346 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.347 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.347 DEBUG:boto:Params: {} 2023-10-17T18:54:02.348 DEBUG:boto:Token: None 2023-10-17T18:54:02.348 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.349 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad39362759429cca-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.354 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:02.354 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:02.355 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.355 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:02.356 DEBUG:boto:Data: 2023-10-17T18:54:02.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.357 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.357 DEBUG:boto:Params: {} 2023-10-17T18:54:02.358 DEBUG:boto:Token: None 2023-10-17T18:54:02.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:02.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5R9hb9n/yCvmoZiDac0kv2fGmro= 2023-10-17T18:54:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5R9hb9n/yCvmoZiDac0kv2fGmro='} 2023-10-17T18:54:02.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090f95fb50411a991-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.367 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.367 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.367 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.368 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.368 DEBUG:boto:Data: 2023-10-17T18:54:02.369 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.369 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.370 DEBUG:boto:Params: {} 2023-10-17T18:54:02.370 DEBUG:boto:Token: None 2023-10-17T18:54:02.371 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.371 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b82ad1c725bc8444-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.375 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:02.376 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:02.376 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.376 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:02.377 DEBUG:boto:Data: 2023-10-17T18:54:02.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.377 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.378 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.378 DEBUG:boto:Params: {} 2023-10-17T18:54:02.379 DEBUG:boto:Token: None 2023-10-17T18:54:02.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:02.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GpFoY68zValrhmACGtq2g1uRB4Y= 2023-10-17T18:54:02.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GpFoY68zValrhmACGtq2g1uRB4Y='} 2023-10-17T18:54:02.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000672130aacb8102cf-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.423 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.423 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.424 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.424 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.425 DEBUG:boto:Data: 2023-10-17T18:54:02.425 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.426 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.426 DEBUG:boto:Params: {} 2023-10-17T18:54:02.427 DEBUG:boto:Token: None 2023-10-17T18:54:02.427 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.428 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002efe8529d3b7afa4-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.431 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.432 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.432 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.433 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.433 DEBUG:boto:Data: 2023-10-17T18:54:02.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.434 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.435 DEBUG:boto:Params: {} 2023-10-17T18:54:02.435 DEBUG:boto:Token: None 2023-10-17T18:54:02.436 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:02.436 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jcqoQ/8ceLHE/OStXXi8iBdjwf0= 2023-10-17T18:54:02.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jcqoQ/8ceLHE/OStXXi8iBdjwf0='} 2023-10-17T18:54:02.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065243b7346e6b83a-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.443 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.443 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.444 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.444 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.445 DEBUG:boto:Data: 2023-10-17T18:54:02.445 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.446 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.446 DEBUG:boto:Params: {} 2023-10-17T18:54:02.447 DEBUG:boto:Token: None 2023-10-17T18:54:02.447 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc368a4cd08d3619-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.452 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:02.452 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:02.453 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.453 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:02.453 DEBUG:boto:Data: 2023-10-17T18:54:02.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.455 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.455 DEBUG:boto:Params: {} 2023-10-17T18:54:02.455 DEBUG:boto:Token: None 2023-10-17T18:54:02.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:02.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:70aHD4y+dSE1YwppV2Nqd/WCc4k= 2023-10-17T18:54:02.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:70aHD4y+dSE1YwppV2Nqd/WCc4k='} 2023-10-17T18:54:02.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c99e005221e017d-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.507 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.507 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.508 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.508 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.509 DEBUG:boto:Data: 2023-10-17T18:54:02.509 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.510 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.510 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.510 DEBUG:boto:Params: {} 2023-10-17T18:54:02.511 DEBUG:boto:Token: None 2023-10-17T18:54:02.511 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.512 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a2460510bce9d0f-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.516 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:02.516 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:02.517 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.517 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:02.517 DEBUG:boto:Data: 2023-10-17T18:54:02.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.519 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.519 DEBUG:boto:Params: {} 2023-10-17T18:54:02.519 DEBUG:boto:Token: None 2023-10-17T18:54:02.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:02.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wgJtTvcpsMgyghD1l4w5PUFsn+w= 2023-10-17T18:54:02.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wgJtTvcpsMgyghD1l4w5PUFsn+w='} 2023-10-17T18:54:02.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073148df24895dde2-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.528 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.528 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.529 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.529 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.529 DEBUG:boto:Data: 2023-10-17T18:54:02.530 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.531 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.531 DEBUG:boto:Params: {} 2023-10-17T18:54:02.532 DEBUG:boto:Token: None 2023-10-17T18:54:02.532 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2a01d16d12e962f-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.536 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:02.536 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:02.537 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.537 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:02.538 DEBUG:boto:Data: 2023-10-17T18:54:02.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.539 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.539 DEBUG:boto:Params: {} 2023-10-17T18:54:02.540 DEBUG:boto:Token: None 2023-10-17T18:54:02.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:02.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:clzz0V4YJg2CK+6yY2GJYH6t1wE= 2023-10-17T18:54:02.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:clzz0V4YJg2CK+6yY2GJYH6t1wE='} 2023-10-17T18:54:02.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d5825f9f6ea379a-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.587 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.588 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.588 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.589 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.589 DEBUG:boto:Data: 2023-10-17T18:54:02.590 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.591 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.591 DEBUG:boto:Params: {} 2023-10-17T18:54:02.591 DEBUG:boto:Token: None 2023-10-17T18:54:02.592 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005df9afefef36125d-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.596 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:02.597 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:02.597 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.597 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:02.598 DEBUG:boto:Data: 2023-10-17T18:54:02.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.599 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.599 DEBUG:boto:Params: {} 2023-10-17T18:54:02.600 DEBUG:boto:Token: None 2023-10-17T18:54:02.600 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:02.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3ROrcOGFP5p2Z3G5e5o5/PxE0AQ= 2023-10-17T18:54:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3ROrcOGFP5p2Z3G5e5o5/PxE0AQ='} 2023-10-17T18:54:02.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006128d58b7b158714-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.608 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.609 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.609 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.610 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.610 DEBUG:boto:Data: 2023-10-17T18:54:02.611 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.612 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.612 DEBUG:boto:Params: {} 2023-10-17T18:54:02.613 DEBUG:boto:Token: None 2023-10-17T18:54:02.614 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.615 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c36e36f5e4183f9-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.619 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:02.620 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:02.620 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.621 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:02.621 DEBUG:boto:Data: 2023-10-17T18:54:02.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.622 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.623 DEBUG:boto:Params: {} 2023-10-17T18:54:02.623 DEBUG:boto:Token: None 2023-10-17T18:54:02.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:02.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Uyq8ufBm+CnL8D3rshxryL9M5rk= 2023-10-17T18:54:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uyq8ufBm+CnL8D3rshxryL9M5rk='} 2023-10-17T18:54:02.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e0cecb3ccddef19-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.667 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.667 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.668 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.668 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.669 DEBUG:boto:Data: 2023-10-17T18:54:02.669 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.669 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.670 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.670 DEBUG:boto:Params: {} 2023-10-17T18:54:02.671 DEBUG:boto:Token: None 2023-10-17T18:54:02.671 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058919f241e890f7f-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.676 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:02.676 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:02.677 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.677 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:02.677 DEBUG:boto:Data: 2023-10-17T18:54:02.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.679 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.680 DEBUG:boto:Params: {} 2023-10-17T18:54:02.680 DEBUG:boto:Token: None 2023-10-17T18:54:02.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:02.681 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DEdcs/1xDOkqmQShFZrryny6k/8= 2023-10-17T18:54:02.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DEdcs/1xDOkqmQShFZrryny6k/8='} 2023-10-17T18:54:02.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003cb0beb7fcb3a1d-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.688 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.689 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.689 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.689 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.690 DEBUG:boto:Data: 2023-10-17T18:54:02.690 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.691 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.691 DEBUG:boto:Params: {} 2023-10-17T18:54:02.692 DEBUG:boto:Token: None 2023-10-17T18:54:02.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbba457cd56a7f01-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.696 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:02.697 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:02.697 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.697 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:02.698 DEBUG:boto:Data: 2023-10-17T18:54:02.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.699 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.699 DEBUG:boto:Params: {} 2023-10-17T18:54:02.700 DEBUG:boto:Token: None 2023-10-17T18:54:02.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:02.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g35GbGk3aXQErsNwQnPpazM79sk= 2023-10-17T18:54:02.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g35GbGk3aXQErsNwQnPpazM79sk='} 2023-10-17T18:54:02.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000365d57517aad7f4-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.747 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.748 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.748 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.749 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.749 DEBUG:boto:Data: 2023-10-17T18:54:02.749 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.750 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.751 DEBUG:boto:Params: {} 2023-10-17T18:54:02.751 DEBUG:boto:Token: None 2023-10-17T18:54:02.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2c2745d4dc49224-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.756 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:02.756 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:02.757 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.757 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:02.757 DEBUG:boto:Data: 2023-10-17T18:54:02.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.759 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.759 DEBUG:boto:Params: {} 2023-10-17T18:54:02.760 DEBUG:boto:Token: None 2023-10-17T18:54:02.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:02.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wUrGvR+NOFsjFZYDg/pCilk8LuQ= 2023-10-17T18:54:02.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wUrGvR+NOFsjFZYDg/pCilk8LuQ='} 2023-10-17T18:54:02.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000453e530ce884b4bb-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.767 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.767 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.768 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.768 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.769 DEBUG:boto:Data: 2023-10-17T18:54:02.769 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.769 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.770 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.770 DEBUG:boto:Params: {} 2023-10-17T18:54:02.771 DEBUG:boto:Token: None 2023-10-17T18:54:02.772 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051b4ec64b3451cf4-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.776 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:02.776 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:02.776 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.777 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:02.777 DEBUG:boto:Data: 2023-10-17T18:54:02.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.778 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.779 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.779 DEBUG:boto:Params: {} 2023-10-17T18:54:02.780 DEBUG:boto:Token: None 2023-10-17T18:54:02.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:02.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hxHNd/sfBG0XrAqHHuGBZhaYt4g= 2023-10-17T18:54:02.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hxHNd/sfBG0XrAqHHuGBZhaYt4g='} 2023-10-17T18:54:02.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000504a6b62ca780c92-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.827 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.828 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.828 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.829 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.829 DEBUG:boto:Data: 2023-10-17T18:54:02.829 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.830 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.830 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.831 DEBUG:boto:Params: {} 2023-10-17T18:54:02.831 DEBUG:boto:Token: None 2023-10-17T18:54:02.832 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.832 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000516999c71238a2a0-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.836 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:02.837 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:02.837 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.837 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:02.838 DEBUG:boto:Data: 2023-10-17T18:54:02.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.839 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.840 DEBUG:boto:Params: {} 2023-10-17T18:54:02.840 DEBUG:boto:Token: None 2023-10-17T18:54:02.841 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:02.841 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:it4OTCIH+n2nmT/EcHwpHRw44y4= 2023-10-17T18:54:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:it4OTCIH+n2nmT/EcHwpHRw44y4='} 2023-10-17T18:54:02.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e9f2b22cd15c22d-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.847 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.848 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.848 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.848 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.849 DEBUG:boto:Data: 2023-10-17T18:54:02.849 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.850 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.850 DEBUG:boto:Params: {} 2023-10-17T18:54:02.851 DEBUG:boto:Token: None 2023-10-17T18:54:02.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ec107188a6eedf5-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.855 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.856 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.856 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.856 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.857 DEBUG:boto:Data: 2023-10-17T18:54:02.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.858 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.859 DEBUG:boto:Params: {} 2023-10-17T18:54:02.859 DEBUG:boto:Token: None 2023-10-17T18:54:02.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:02.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6BXsy6LUy8e85jvj9gYNgkNTGWc= 2023-10-17T18:54:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6BXsy6LUy8e85jvj9gYNgkNTGWc='} 2023-10-17T18:54:02.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030d980ab6bb7a97a-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.907 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.907 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.908 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.908 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.909 DEBUG:boto:Data: 2023-10-17T18:54:02.909 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.910 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.910 DEBUG:boto:Params: {} 2023-10-17T18:54:02.911 DEBUG:boto:Token: None 2023-10-17T18:54:02.911 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044e929c6bbc7f428-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.916 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:02.916 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:02.916 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.917 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:02.917 DEBUG:boto:Data: 2023-10-17T18:54:02.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.919 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.919 DEBUG:boto:Params: {} 2023-10-17T18:54:02.920 DEBUG:boto:Token: None 2023-10-17T18:54:02.920 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:02.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k6bFaepkCSJnw38fpOS5NSqJrO4= 2023-10-17T18:54:02.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k6bFaepkCSJnw38fpOS5NSqJrO4='} 2023-10-17T18:54:02.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005187a0cc787e7ad6-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.928 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.928 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.929 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.929 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.929 DEBUG:boto:Data: 2023-10-17T18:54:02.930 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.931 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.931 DEBUG:boto:Params: {} 2023-10-17T18:54:02.932 DEBUG:boto:Token: None 2023-10-17T18:54:02.932 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T18:54:02.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d91a3a8b2da40d6-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.936 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:02.937 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:02.937 DEBUG:boto:Method: PUT 2023-10-17T18:54:02.937 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:02.938 DEBUG:boto:Data: 2023-10-17T18:54:02.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:02.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.939 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.940 DEBUG:boto:Params: {} 2023-10-17T18:54:02.940 DEBUG:boto:Token: None 2023-10-17T18:54:02.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:02.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CucVPCkNAyR7JPgELeKDBcUZmCI= 2023-10-17T18:54:02.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CucVPCkNAyR7JPgELeKDBcUZmCI='} 2023-10-17T18:54:02.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e1bd0600fc0509-00652ed84a-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:02.991 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:02.992 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:02.992 DEBUG:boto:Method: HEAD 2023-10-17T18:54:02.993 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:02.993 DEBUG:boto:Data: 2023-10-17T18:54:02.993 DEBUG:boto:Headers: {} 2023-10-17T18:54:02.994 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:02.994 DEBUG:boto:Port: 8000 2023-10-17T18:54:02.995 DEBUG:boto:Params: {} 2023-10-17T18:54:02.995 DEBUG:boto:Token: None 2023-10-17T18:54:02.996 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:02 GMT /hbmuog-58/ 2023-10-17T18:54:02.996 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc= 2023-10-17T18:54:02.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:02 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0v8qMm9O5ZV69jg9SShRrxlrhc=', 'Content-Length': '0'} 2023-10-17T18:54:02.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1862ae258bcbe32-00652ed84a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.000 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:03.000 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:03.001 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.001 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:03.002 DEBUG:boto:Data: 2023-10-17T18:54:03.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.003 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.003 DEBUG:boto:Params: {} 2023-10-17T18:54:03.004 DEBUG:boto:Token: None 2023-10-17T18:54:03.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:03.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K41ieQ+q44EQa8fO5jP9VAesx+4= 2023-10-17T18:54:03.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K41ieQ+q44EQa8fO5jP9VAesx+4='} 2023-10-17T18:54:03.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c3754cf58670dd9-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.012 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.013 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.013 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.014 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.014 DEBUG:boto:Data: 2023-10-17T18:54:03.015 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.015 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.015 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.016 DEBUG:boto:Params: {} 2023-10-17T18:54:03.016 DEBUG:boto:Token: None 2023-10-17T18:54:03.017 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4e47d5556321404-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.021 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.021 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.022 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.022 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.023 DEBUG:boto:Data: 2023-10-17T18:54:03.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.024 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.024 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.025 DEBUG:boto:Params: {} 2023-10-17T18:54:03.025 DEBUG:boto:Token: None 2023-10-17T18:54:03.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.026 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ddxsqdQZsQHBsMM7GvMpG/+K8s= 2023-10-17T18:54:03.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ddxsqdQZsQHBsMM7GvMpG/+K8s='} 2023-10-17T18:54:03.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b77c28ef12d8eb7b-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:03.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:03.072 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:03.130 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:03.130 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:03.176 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:03.177 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:03.183 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 25 2023-10-17T18:54:03.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:03.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:03.234 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:03.330 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000058", 2023-10-17T18:54:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000026", 2023-10-17T18:54:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:54:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000046", 2023-10-17T18:54:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000014", 2023-10-17T18:54:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000050", 2023-10-17T18:54:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000032", 2023-10-17T18:54:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:54:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000039", 2023-10-17T18:54:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000007", 2023-10-17T18:54:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006", 2023-10-17T18:54:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000040", 2023-10-17T18:54:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000009", 2023-10-17T18:54:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000045", 2023-10-17T18:54:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000013", 2023-10-17T18:54:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:54:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000033", 2023-10-17T18:54:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021", 2023-10-17T18:54:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:54:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000055", 2023-10-17T18:54:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000008" 2023-10-17T18:54:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:03.336 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:03.338 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.338 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.339 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.339 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.339 DEBUG:boto:Data: 2023-10-17T18:54:03.340 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.340 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.341 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.341 DEBUG:boto:Params: {} 2023-10-17T18:54:03.342 DEBUG:boto:Token: None 2023-10-17T18:54:03.342 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.343 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d8f1193615b47c2-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.347 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:03.347 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:03.348 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.348 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:03.349 DEBUG:boto:Data: 2023-10-17T18:54:03.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.349 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.350 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.350 DEBUG:boto:Params: {} 2023-10-17T18:54:03.351 DEBUG:boto:Token: None 2023-10-17T18:54:03.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:03.352 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NsSmeP5OTuCb4HhAFpIPT8pEpdU= 2023-10-17T18:54:03.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NsSmeP5OTuCb4HhAFpIPT8pEpdU='} 2023-10-17T18:54:03.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047b897edb11bc350-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.358 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.359 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.359 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.360 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.360 DEBUG:boto:Data: 2023-10-17T18:54:03.360 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.361 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.361 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.362 DEBUG:boto:Params: {} 2023-10-17T18:54:03.362 DEBUG:boto:Token: None 2023-10-17T18:54:03.363 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.363 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e270bca2a52168e0-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.367 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:03.368 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:03.368 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.368 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:03.369 DEBUG:boto:Data: 2023-10-17T18:54:03.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.370 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.371 DEBUG:boto:Params: {} 2023-10-17T18:54:03.371 DEBUG:boto:Token: None 2023-10-17T18:54:03.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:03.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dzIMO14UNOdKu94bRtVgB1S1F0o= 2023-10-17T18:54:03.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dzIMO14UNOdKu94bRtVgB1S1F0o='} 2023-10-17T18:54:03.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058ab36ddbe92d7b8-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.419 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.420 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.420 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.421 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.421 DEBUG:boto:Data: 2023-10-17T18:54:03.421 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.422 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.423 DEBUG:boto:Params: {} 2023-10-17T18:54:03.423 DEBUG:boto:Token: None 2023-10-17T18:54:03.424 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.424 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6555f8d24963218-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.428 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:03.429 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:03.429 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.430 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:03.430 DEBUG:boto:Data: 2023-10-17T18:54:03.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.431 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.431 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.432 DEBUG:boto:Params: {} 2023-10-17T18:54:03.432 DEBUG:boto:Token: None 2023-10-17T18:54:03.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:03.433 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vIj3EyX686rngjUhLO9QCvo8jYc= 2023-10-17T18:54:03.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vIj3EyX686rngjUhLO9QCvo8jYc='} 2023-10-17T18:54:03.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c023ea63eb5600d5-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.440 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.440 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.441 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.441 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.441 DEBUG:boto:Data: 2023-10-17T18:54:03.442 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.442 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.443 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.443 DEBUG:boto:Params: {} 2023-10-17T18:54:03.444 DEBUG:boto:Token: None 2023-10-17T18:54:03.444 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c16f236e4944deb2-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.449 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:03.449 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:03.449 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.450 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:03.450 DEBUG:boto:Data: 2023-10-17T18:54:03.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.452 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.452 DEBUG:boto:Params: {} 2023-10-17T18:54:03.453 DEBUG:boto:Token: None 2023-10-17T18:54:03.453 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:03.454 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WumJqS4fwdIEikloyUKyW3X1RqY= 2023-10-17T18:54:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WumJqS4fwdIEikloyUKyW3X1RqY='} 2023-10-17T18:54:03.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8bce9f37255f528-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.503 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.504 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.504 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.504 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.505 DEBUG:boto:Data: 2023-10-17T18:54:03.505 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.506 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.507 DEBUG:boto:Params: {} 2023-10-17T18:54:03.507 DEBUG:boto:Token: None 2023-10-17T18:54:03.508 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036da208ef9b751e5-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.512 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:03.512 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:03.513 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.513 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:03.514 DEBUG:boto:Data: 2023-10-17T18:54:03.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.515 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.516 DEBUG:boto:Params: {} 2023-10-17T18:54:03.516 DEBUG:boto:Token: None 2023-10-17T18:54:03.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:03.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ixXvgHY814yqCDS95Zr3Gkq7pEk= 2023-10-17T18:54:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ixXvgHY814yqCDS95Zr3Gkq7pEk='} 2023-10-17T18:54:03.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000912701556af783a5-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.525 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.525 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.526 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.526 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.527 DEBUG:boto:Data: 2023-10-17T18:54:03.527 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.528 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.528 DEBUG:boto:Params: {} 2023-10-17T18:54:03.529 DEBUG:boto:Token: None 2023-10-17T18:54:03.529 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.530 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c00ece981802d56d-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.534 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.534 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.535 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.535 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.536 DEBUG:boto:Data: 2023-10-17T18:54:03.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.537 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.537 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.538 DEBUG:boto:Params: {} 2023-10-17T18:54:03.538 DEBUG:boto:Token: None 2023-10-17T18:54:03.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:03.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ddxsqdQZsQHBsMM7GvMpG/+K8s= 2023-10-17T18:54:03.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ddxsqdQZsQHBsMM7GvMpG/+K8s='} 2023-10-17T18:54:03.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6182f61c340446c-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.584 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.584 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.585 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.585 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.586 DEBUG:boto:Data: 2023-10-17T18:54:03.586 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.587 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.588 DEBUG:boto:Params: {} 2023-10-17T18:54:03.588 DEBUG:boto:Token: None 2023-10-17T18:54:03.589 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85d0f95e9e41440-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.594 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:03.594 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:03.595 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.595 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:03.596 DEBUG:boto:Data: 2023-10-17T18:54:03.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.598 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.598 DEBUG:boto:Params: {} 2023-10-17T18:54:03.599 DEBUG:boto:Token: None 2023-10-17T18:54:03.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:03.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jVJJ+RCewkz/0iq1vnYhpOQFaIw= 2023-10-17T18:54:03.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jVJJ+RCewkz/0iq1vnYhpOQFaIw='} 2023-10-17T18:54:03.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b52ba08331c27789-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.607 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.608 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.608 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.609 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.609 DEBUG:boto:Data: 2023-10-17T18:54:03.610 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.610 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.611 DEBUG:boto:Params: {} 2023-10-17T18:54:03.611 DEBUG:boto:Token: None 2023-10-17T18:54:03.612 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009706968686f0bcd5-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.616 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:03.617 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:03.617 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.618 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:03.618 DEBUG:boto:Data: 2023-10-17T18:54:03.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.620 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.620 DEBUG:boto:Params: {} 2023-10-17T18:54:03.620 DEBUG:boto:Token: None 2023-10-17T18:54:03.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:03.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5S66T0nK8ZAqSCUEYS3thGIfc6E= 2023-10-17T18:54:03.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5S66T0nK8ZAqSCUEYS3thGIfc6E='} 2023-10-17T18:54:03.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6ee2f7e032aa15f-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.667 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.668 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.668 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.669 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.669 DEBUG:boto:Data: 2023-10-17T18:54:03.670 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.671 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.671 DEBUG:boto:Params: {} 2023-10-17T18:54:03.671 DEBUG:boto:Token: None 2023-10-17T18:54:03.672 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d6f0c7ab57da0ee-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.677 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:03.677 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:03.678 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.679 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:03.679 DEBUG:boto:Data: 2023-10-17T18:54:03.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.680 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.681 DEBUG:boto:Params: {} 2023-10-17T18:54:03.681 DEBUG:boto:Token: None 2023-10-17T18:54:03.682 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:03.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YM9JaWkihHb+MrEI6TzC1uRUfqU= 2023-10-17T18:54:03.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YM9JaWkihHb+MrEI6TzC1uRUfqU='} 2023-10-17T18:54:03.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef8006c5be7ccd63-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.690 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.691 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.691 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.692 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.692 DEBUG:boto:Data: 2023-10-17T18:54:03.693 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.694 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.694 DEBUG:boto:Params: {} 2023-10-17T18:54:03.695 DEBUG:boto:Token: None 2023-10-17T18:54:03.696 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4bdd1efe14f45f5-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.701 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:03.701 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:03.702 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.702 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:03.703 DEBUG:boto:Data: 2023-10-17T18:54:03.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.704 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.704 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.705 DEBUG:boto:Params: {} 2023-10-17T18:54:03.706 DEBUG:boto:Token: None 2023-10-17T18:54:03.706 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:03.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mm4szmZVvh+MF7Zu9pjdOle11tM= 2023-10-17T18:54:03.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mm4szmZVvh+MF7Zu9pjdOle11tM='} 2023-10-17T18:54:03.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a9b02de634802f0-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.751 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.752 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.752 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.753 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.753 DEBUG:boto:Data: 2023-10-17T18:54:03.754 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.754 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.755 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.756 DEBUG:boto:Params: {} 2023-10-17T18:54:03.756 DEBUG:boto:Token: None 2023-10-17T18:54:03.757 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3bed6d8dd1e8f69-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.762 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:03.762 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:03.763 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.763 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:03.764 DEBUG:boto:Data: 2023-10-17T18:54:03.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.766 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.766 DEBUG:boto:Params: {} 2023-10-17T18:54:03.767 DEBUG:boto:Token: None 2023-10-17T18:54:03.767 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:03.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:saa71tw4/78jwnWTojzVf61Ghw0= 2023-10-17T18:54:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:saa71tw4/78jwnWTojzVf61Ghw0='} 2023-10-17T18:54:03.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005863915a802ab4eb-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.775 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.776 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.777 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.777 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.778 DEBUG:boto:Data: 2023-10-17T18:54:03.778 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.779 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.779 DEBUG:boto:Params: {} 2023-10-17T18:54:03.780 DEBUG:boto:Token: None 2023-10-17T18:54:03.780 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008313407d3d92ed1f-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.785 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:03.785 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:03.786 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.786 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:03.787 DEBUG:boto:Data: 2023-10-17T18:54:03.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.789 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.789 DEBUG:boto:Params: {} 2023-10-17T18:54:03.790 DEBUG:boto:Token: None 2023-10-17T18:54:03.790 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:03.791 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ftBRFnLMbaId6dwajSv6ysEAX3c= 2023-10-17T18:54:03.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ftBRFnLMbaId6dwajSv6ysEAX3c='} 2023-10-17T18:54:03.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091288eef36dffe85-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.835 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.836 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.836 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.836 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.837 DEBUG:boto:Data: 2023-10-17T18:54:03.837 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.838 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.839 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.839 DEBUG:boto:Params: {} 2023-10-17T18:54:03.840 DEBUG:boto:Token: None 2023-10-17T18:54:03.840 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.841 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006289ad556f8046e2-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.845 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:03.845 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:03.846 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.846 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:03.846 DEBUG:boto:Data: 2023-10-17T18:54:03.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.848 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.848 DEBUG:boto:Params: {} 2023-10-17T18:54:03.849 DEBUG:boto:Token: None 2023-10-17T18:54:03.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:03.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r03ZQONNQAy+pGWwH3J+3XLMlwM= 2023-10-17T18:54:03.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r03ZQONNQAy+pGWwH3J+3XLMlwM='} 2023-10-17T18:54:03.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b76dc76932c96a56-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.856 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.857 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.857 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.858 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.858 DEBUG:boto:Data: 2023-10-17T18:54:03.859 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.859 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.860 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.860 DEBUG:boto:Params: {} 2023-10-17T18:54:03.861 DEBUG:boto:Token: None 2023-10-17T18:54:03.861 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042492d831b727b74-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.865 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:03.866 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:03.866 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.867 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:03.868 DEBUG:boto:Data: 2023-10-17T18:54:03.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.869 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.869 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.870 DEBUG:boto:Params: {} 2023-10-17T18:54:03.870 DEBUG:boto:Token: None 2023-10-17T18:54:03.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:03.871 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UkYc1di0yrpXLwa6ZizAinzO5+I= 2023-10-17T18:54:03.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UkYc1di0yrpXLwa6ZizAinzO5+I='} 2023-10-17T18:54:03.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000786603ce817188ee-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.919 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.920 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.920 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.920 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.921 DEBUG:boto:Data: 2023-10-17T18:54:03.921 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.922 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.922 DEBUG:boto:Params: {} 2023-10-17T18:54:03.923 DEBUG:boto:Token: None 2023-10-17T18:54:03.923 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.924 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f1a5c75baf0e6ba-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.928 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:03.928 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:03.928 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.929 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:03.929 DEBUG:boto:Data: 2023-10-17T18:54:03.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.931 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.931 DEBUG:boto:Params: {} 2023-10-17T18:54:03.932 DEBUG:boto:Token: None 2023-10-17T18:54:03.933 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:03.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5Q/I09scqFTV3NvjPJWPMHeFNmI= 2023-10-17T18:54:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5Q/I09scqFTV3NvjPJWPMHeFNmI='} 2023-10-17T18:54:03.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db9497340256c2c0-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.941 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:03.941 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:03.941 DEBUG:boto:Method: HEAD 2023-10-17T18:54:03.942 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:03.942 DEBUG:boto:Data: 2023-10-17T18:54:03.943 DEBUG:boto:Headers: {} 2023-10-17T18:54:03.943 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.944 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.944 DEBUG:boto:Params: {} 2023-10-17T18:54:03.944 DEBUG:boto:Token: None 2023-10-17T18:54:03.945 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/ 2023-10-17T18:54:03.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s= 2023-10-17T18:54:03.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y10jdMTKDXm1TM9nklhqpriRG+s=', 'Content-Length': '0'} 2023-10-17T18:54:03.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac65c3606867996b-00652ed84b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.949 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:03.949 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:03.950 DEBUG:boto:Method: PUT 2023-10-17T18:54:03.950 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:03.951 DEBUG:boto:Data: 2023-10-17T18:54:03.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:03.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:03.952 DEBUG:boto:Port: 8000 2023-10-17T18:54:03.952 DEBUG:boto:Params: {} 2023-10-17T18:54:03.953 DEBUG:boto:Token: None 2023-10-17T18:54:03.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:03 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:03.954 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QYGXgZNxsvOMLgurYJVL92yVMTw= 2023-10-17T18:54:03.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QYGXgZNxsvOMLgurYJVL92yVMTw='} 2023-10-17T18:54:03.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031e80b2a2aca6ea0-00652ed84b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:03.999 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.000 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.000 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.001 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.001 DEBUG:boto:Data: 2023-10-17T18:54:04.001 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.002 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.003 DEBUG:boto:Params: {} 2023-10-17T18:54:04.003 DEBUG:boto:Token: None 2023-10-17T18:54:04.004 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a986f09e0e2522c0-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.008 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:04.009 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:04.009 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.010 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:04.010 DEBUG:boto:Data: 2023-10-17T18:54:04.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.011 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.012 DEBUG:boto:Params: {} 2023-10-17T18:54:04.012 DEBUG:boto:Token: None 2023-10-17T18:54:04.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:04.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lHmARAuGq9BG3J6euaWUPqlFY7Q= 2023-10-17T18:54:04.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lHmARAuGq9BG3J6euaWUPqlFY7Q='} 2023-10-17T18:54:04.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000408263755e976ae-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.021 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.021 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.022 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.022 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.022 DEBUG:boto:Data: 2023-10-17T18:54:04.023 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.024 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.024 DEBUG:boto:Params: {} 2023-10-17T18:54:04.024 DEBUG:boto:Token: None 2023-10-17T18:54:04.025 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf253ff370817bda-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.029 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:04.029 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:04.030 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.030 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:04.031 DEBUG:boto:Data: 2023-10-17T18:54:04.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.031 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.032 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.032 DEBUG:boto:Params: {} 2023-10-17T18:54:04.033 DEBUG:boto:Token: None 2023-10-17T18:54:04.033 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:04.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JMNSlNyfNVk/WQH0QngoWFvO2Xc= 2023-10-17T18:54:04.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JMNSlNyfNVk/WQH0QngoWFvO2Xc='} 2023-10-17T18:54:04.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052718f5a71301ad2-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.079 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.080 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.080 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.081 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.081 DEBUG:boto:Data: 2023-10-17T18:54:04.081 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.082 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.083 DEBUG:boto:Params: {} 2023-10-17T18:54:04.083 DEBUG:boto:Token: None 2023-10-17T18:54:04.084 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072cd6fb0cf8b9a2a-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.088 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:04.089 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:04.089 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.089 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:04.090 DEBUG:boto:Data: 2023-10-17T18:54:04.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.091 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.092 DEBUG:boto:Params: {} 2023-10-17T18:54:04.092 DEBUG:boto:Token: None 2023-10-17T18:54:04.092 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:04.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fPA6sF8L4n/YIyAN2XnD/Q0Ov8Y= 2023-10-17T18:54:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fPA6sF8L4n/YIyAN2XnD/Q0Ov8Y='} 2023-10-17T18:54:04.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c417eb67ec635c0b-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.100 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.101 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.101 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.102 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.102 DEBUG:boto:Data: 2023-10-17T18:54:04.102 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.103 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.103 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.104 DEBUG:boto:Params: {} 2023-10-17T18:54:04.104 DEBUG:boto:Token: None 2023-10-17T18:54:04.105 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.105 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed356561e45439c7-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.110 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.110 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.111 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.111 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.111 DEBUG:boto:Data: 2023-10-17T18:54:04.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.112 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.113 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.113 DEBUG:boto:Params: {} 2023-10-17T18:54:04.114 DEBUG:boto:Token: None 2023-10-17T18:54:04.114 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U19fiIvY/z2/qcbvcgZHhiVOXIA= 2023-10-17T18:54:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U19fiIvY/z2/qcbvcgZHhiVOXIA='} 2023-10-17T18:54:04.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db4302be2bc9b7f6-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.159 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.159 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.160 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.160 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.161 DEBUG:boto:Data: 2023-10-17T18:54:04.161 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.162 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.162 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.163 DEBUG:boto:Params: {} 2023-10-17T18:54:04.163 DEBUG:boto:Token: None 2023-10-17T18:54:04.164 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d2d65e2ea446542-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.168 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:04.169 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:04.169 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.170 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:04.170 DEBUG:boto:Data: 2023-10-17T18:54:04.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.171 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.172 DEBUG:boto:Params: {} 2023-10-17T18:54:04.172 DEBUG:boto:Token: None 2023-10-17T18:54:04.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:04.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2qHv9ZjBMJivRaFPvxWp+EMPDys= 2023-10-17T18:54:04.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2qHv9ZjBMJivRaFPvxWp+EMPDys='} 2023-10-17T18:54:04.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abf7bc43faf5f53d-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:04.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:04.182 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:04.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:04.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:04.286 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:04.286 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:04.295 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 33 2023-10-17T18:54:04.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:04.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:04.349 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:04.402 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:04.403 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:04.446 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:04.446 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:04.446 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006", 2023-10-17T18:54:04.447 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000059", 2023-10-17T18:54:04.447 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000027", 2023-10-17T18:54:04.447 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:54:04.447 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:54:04.448 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:54:04.448 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000019", 2023-10-17T18:54:04.448 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000049", 2023-10-17T18:54:04.448 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000017", 2023-10-17T18:54:04.449 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000008", 2023-10-17T18:54:04.449 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000015", 2023-10-17T18:54:04.449 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000031", 2023-10-17T18:54:04.449 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000022", 2023-10-17T18:54:04.450 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000013", 2023-10-17T18:54:04.450 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000020", 2023-10-17T18:54:04.450 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000037", 2023-10-17T18:54:04.450 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000005", 2023-10-17T18:54:04.450 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:54:04.451 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:54:04.451 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000042", 2023-10-17T18:54:04.451 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:54:04.451 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:04.452 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000009", 2023-10-17T18:54:04.452 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000025", 2023-10-17T18:54:04.452 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000016", 2023-10-17T18:54:04.452 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:04.452 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000014", 2023-10-17T18:54:04.453 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000030", 2023-10-17T18:54:04.453 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021" 2023-10-17T18:54:04.453 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:04.453 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:04.454 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.455 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.455 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.456 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.456 DEBUG:boto:Data: 2023-10-17T18:54:04.457 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.458 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.458 DEBUG:boto:Params: {} 2023-10-17T18:54:04.459 DEBUG:boto:Token: None 2023-10-17T18:54:04.459 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024f400d50af5e5c5-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.464 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:04.465 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:04.465 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.466 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:04.466 DEBUG:boto:Data: 2023-10-17T18:54:04.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.467 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.468 DEBUG:boto:Params: {} 2023-10-17T18:54:04.468 DEBUG:boto:Token: None 2023-10-17T18:54:04.469 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:04.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eK5A8AOjVVmcf4czwWkmcHlUT7I= 2023-10-17T18:54:04.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eK5A8AOjVVmcf4czwWkmcHlUT7I='} 2023-10-17T18:54:04.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009447411cd2ed0d05-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.515 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.515 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.516 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.516 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.517 DEBUG:boto:Data: 2023-10-17T18:54:04.517 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.518 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.518 DEBUG:boto:Params: {} 2023-10-17T18:54:04.519 DEBUG:boto:Token: None 2023-10-17T18:54:04.519 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020f549a40b96da82-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.524 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:04.524 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:04.525 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.525 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:04.526 DEBUG:boto:Data: 2023-10-17T18:54:04.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.527 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.527 DEBUG:boto:Params: {} 2023-10-17T18:54:04.528 DEBUG:boto:Token: None 2023-10-17T18:54:04.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:04.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AyVA2Nl6d6wMlGPJl2X07GzW/AE= 2023-10-17T18:54:04.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AyVA2Nl6d6wMlGPJl2X07GzW/AE='} 2023-10-17T18:54:04.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac3feb5a7fec1588-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.535 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.535 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.536 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.536 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.537 DEBUG:boto:Data: 2023-10-17T18:54:04.537 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.538 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.539 DEBUG:boto:Params: {} 2023-10-17T18:54:04.539 DEBUG:boto:Token: None 2023-10-17T18:54:04.540 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029fca0ee69613ece-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.545 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:04.546 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:04.547 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.547 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:04.547 DEBUG:boto:Data: 2023-10-17T18:54:04.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.548 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.549 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.549 DEBUG:boto:Params: {} 2023-10-17T18:54:04.550 DEBUG:boto:Token: None 2023-10-17T18:54:04.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:04.551 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+kpyjcy65HnPWFU63V1DHlDLrJA= 2023-10-17T18:54:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+kpyjcy65HnPWFU63V1DHlDLrJA='} 2023-10-17T18:54:04.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a73a2f051cf8e65-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.595 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.595 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.596 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.596 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.596 DEBUG:boto:Data: 2023-10-17T18:54:04.597 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.598 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.598 DEBUG:boto:Params: {} 2023-10-17T18:54:04.602 DEBUG:boto:Token: None 2023-10-17T18:54:04.602 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.603 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f00fb1e66c962881-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.607 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:04.607 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:04.608 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.608 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:04.608 DEBUG:boto:Data: 2023-10-17T18:54:04.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.610 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.610 DEBUG:boto:Params: {} 2023-10-17T18:54:04.611 DEBUG:boto:Token: None 2023-10-17T18:54:04.611 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:04.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BzE9/MVC5jTGK/fXOIFyVIUtMkM= 2023-10-17T18:54:04.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BzE9/MVC5jTGK/fXOIFyVIUtMkM='} 2023-10-17T18:54:04.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004151a0b626d6e034-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.620 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.620 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.620 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.621 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.621 DEBUG:boto:Data: 2023-10-17T18:54:04.622 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.622 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.623 DEBUG:boto:Params: {} 2023-10-17T18:54:04.623 DEBUG:boto:Token: None 2023-10-17T18:54:04.624 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0ec13a0d24b8025-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.628 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:04.628 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:04.629 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.629 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:04.629 DEBUG:boto:Data: 2023-10-17T18:54:04.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.630 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.631 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.631 DEBUG:boto:Params: {} 2023-10-17T18:54:04.631 DEBUG:boto:Token: None 2023-10-17T18:54:04.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:04.632 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NFNFVDS+Y9YJKfXt8XvyCz36Dlo= 2023-10-17T18:54:04.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NFNFVDS+Y9YJKfXt8XvyCz36Dlo='} 2023-10-17T18:54:04.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c0ae4fd2e544831-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.679 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.679 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.680 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.680 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.681 DEBUG:boto:Data: 2023-10-17T18:54:04.681 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.682 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.682 DEBUG:boto:Params: {} 2023-10-17T18:54:04.683 DEBUG:boto:Token: None 2023-10-17T18:54:04.683 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000954538a41200b00f-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.688 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:04.689 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:04.689 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.690 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:04.690 DEBUG:boto:Data: 2023-10-17T18:54:04.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.691 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.692 DEBUG:boto:Params: {} 2023-10-17T18:54:04.692 DEBUG:boto:Token: None 2023-10-17T18:54:04.693 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:04.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLE6hxqI0AESIEULzuFnYkBpvEA= 2023-10-17T18:54:04.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLE6hxqI0AESIEULzuFnYkBpvEA='} 2023-10-17T18:54:04.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052040daa482b8d82-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.700 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.701 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.701 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.701 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.702 DEBUG:boto:Data: 2023-10-17T18:54:04.702 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.703 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.703 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.703 DEBUG:boto:Params: {} 2023-10-17T18:54:04.704 DEBUG:boto:Token: None 2023-10-17T18:54:04.704 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.705 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1d0a091487bf434-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.708 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:04.709 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:04.709 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.710 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:04.710 DEBUG:boto:Data: 2023-10-17T18:54:04.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.711 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.712 DEBUG:boto:Params: {} 2023-10-17T18:54:04.712 DEBUG:boto:Token: None 2023-10-17T18:54:04.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:04.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p7eDIFpnDOu39eUnDH19IdzQXi8= 2023-10-17T18:54:04.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p7eDIFpnDOu39eUnDH19IdzQXi8='} 2023-10-17T18:54:04.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8c954c210952485-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.759 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.759 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.760 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.760 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.761 DEBUG:boto:Data: 2023-10-17T18:54:04.761 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.762 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.762 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.763 DEBUG:boto:Params: {} 2023-10-17T18:54:04.763 DEBUG:boto:Token: None 2023-10-17T18:54:04.763 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.764 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff28bbc019dd9f02-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.768 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:04.769 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:04.769 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.770 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:04.770 DEBUG:boto:Data: 2023-10-17T18:54:04.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.771 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.772 DEBUG:boto:Params: {} 2023-10-17T18:54:04.772 DEBUG:boto:Token: None 2023-10-17T18:54:04.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:04.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u2GNXnQ0SK/nJLgEeYuTwh4S5gQ= 2023-10-17T18:54:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u2GNXnQ0SK/nJLgEeYuTwh4S5gQ='} 2023-10-17T18:54:04.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005016d3e5d5094b3b-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.780 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.781 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.781 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.782 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.782 DEBUG:boto:Data: 2023-10-17T18:54:04.782 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.783 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.784 DEBUG:boto:Params: {} 2023-10-17T18:54:04.784 DEBUG:boto:Token: None 2023-10-17T18:54:04.784 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b17e621cfabff6a2-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.789 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:04.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:04.789 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.790 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:04.790 DEBUG:boto:Data: 2023-10-17T18:54:04.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.791 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.792 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.792 DEBUG:boto:Params: {} 2023-10-17T18:54:04.792 DEBUG:boto:Token: None 2023-10-17T18:54:04.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:04.793 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nTBIW9WAeNsqk+npmQl5TBJLcfM= 2023-10-17T18:54:04.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nTBIW9WAeNsqk+npmQl5TBJLcfM='} 2023-10-17T18:54:04.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000698841773ae7db5-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.839 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.839 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.840 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.840 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.841 DEBUG:boto:Data: 2023-10-17T18:54:04.841 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.842 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.842 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.843 DEBUG:boto:Params: {} 2023-10-17T18:54:04.843 DEBUG:boto:Token: None 2023-10-17T18:54:04.844 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41c62ac56b6356c-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.848 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:04.849 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:04.849 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.850 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:04.850 DEBUG:boto:Data: 2023-10-17T18:54:04.851 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.851 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.852 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.852 DEBUG:boto:Params: {} 2023-10-17T18:54:04.852 DEBUG:boto:Token: None 2023-10-17T18:54:04.853 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:04.853 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UJxMBANDJ3Kwg/yLS/JjvKrfNO8= 2023-10-17T18:54:04.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UJxMBANDJ3Kwg/yLS/JjvKrfNO8='} 2023-10-17T18:54:04.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc480bfc4af98166-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.859 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.860 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.860 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.861 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.861 DEBUG:boto:Data: 2023-10-17T18:54:04.862 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.863 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.863 DEBUG:boto:Params: {} 2023-10-17T18:54:04.863 DEBUG:boto:Token: None 2023-10-17T18:54:04.864 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c70ff523b511636f-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.869 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:04.869 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:04.869 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.870 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:04.870 DEBUG:boto:Data: 2023-10-17T18:54:04.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.871 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.872 DEBUG:boto:Params: {} 2023-10-17T18:54:04.872 DEBUG:boto:Token: None 2023-10-17T18:54:04.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:04.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GEtY5klpUL1Up2N2D8wKQ6AIS40= 2023-10-17T18:54:04.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GEtY5klpUL1Up2N2D8wKQ6AIS40='} 2023-10-17T18:54:04.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c144a5a9fea137df-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.919 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.919 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.920 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.920 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.921 DEBUG:boto:Data: 2023-10-17T18:54:04.921 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.921 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.922 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.922 DEBUG:boto:Params: {} 2023-10-17T18:54:04.922 DEBUG:boto:Token: None 2023-10-17T18:54:04.923 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089fc3022ef8a2c32-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.927 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.928 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.928 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.929 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.929 DEBUG:boto:Data: 2023-10-17T18:54:04.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.931 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.931 DEBUG:boto:Params: {} 2023-10-17T18:54:04.931 DEBUG:boto:Token: None 2023-10-17T18:54:04.932 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:04.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:U19fiIvY/z2/qcbvcgZHhiVOXIA= 2023-10-17T18:54:04.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U19fiIvY/z2/qcbvcgZHhiVOXIA='} 2023-10-17T18:54:04.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ba365905a350dbd-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.940 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.941 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:04.941 DEBUG:boto:Method: HEAD 2023-10-17T18:54:04.941 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:04.942 DEBUG:boto:Data: 2023-10-17T18:54:04.942 DEBUG:boto:Headers: {} 2023-10-17T18:54:04.943 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.943 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.943 DEBUG:boto:Params: {} 2023-10-17T18:54:04.944 DEBUG:boto:Token: None 2023-10-17T18:54:04.944 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/ 2023-10-17T18:54:04.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro= 2023-10-17T18:54:04.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1etive8iv2M8Md2gZEagY5N0kro=', 'Content-Length': '0'} 2023-10-17T18:54:04.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af602600653ffeb9-00652ed84c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.949 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:04.949 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:04.950 DEBUG:boto:Method: PUT 2023-10-17T18:54:04.950 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:04.951 DEBUG:boto:Data: 2023-10-17T18:54:04.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:04.952 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:04.952 DEBUG:boto:Port: 8000 2023-10-17T18:54:04.952 DEBUG:boto:Params: {} 2023-10-17T18:54:04.953 DEBUG:boto:Token: None 2023-10-17T18:54:04.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:04 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:04.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8KBnyl/53qP7TfVz73xE3frZjEY= 2023-10-17T18:54:04.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:04 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8KBnyl/53qP7TfVz73xE3frZjEY='} 2023-10-17T18:54:04.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000174535da2490ba2e-00652ed84c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:04.999 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:04.999 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.000 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.000 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.001 DEBUG:boto:Data: 2023-10-17T18:54:05.001 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.002 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.002 DEBUG:boto:Params: {} 2023-10-17T18:54:05.003 DEBUG:boto:Token: None 2023-10-17T18:54:05.003 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006e6fca3320cb8d8-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.008 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:05.009 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:05.009 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.010 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:05.010 DEBUG:boto:Data: 2023-10-17T18:54:05.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.011 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.012 DEBUG:boto:Params: {} 2023-10-17T18:54:05.012 DEBUG:boto:Token: None 2023-10-17T18:54:05.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:05.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RSsFeDCDi0tSGPrC3J4DbFh/OUs= 2023-10-17T18:54:05.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RSsFeDCDi0tSGPrC3J4DbFh/OUs='} 2023-10-17T18:54:05.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013a6960cecf05c90-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.020 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.021 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.021 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.022 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.022 DEBUG:boto:Data: 2023-10-17T18:54:05.023 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.024 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.024 DEBUG:boto:Params: {} 2023-10-17T18:54:05.024 DEBUG:boto:Token: None 2023-10-17T18:54:05.025 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.026 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cd3fad12e9fc238-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.030 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:05.030 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:05.031 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.031 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:05.032 DEBUG:boto:Data: 2023-10-17T18:54:05.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.032 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.033 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.033 DEBUG:boto:Params: {} 2023-10-17T18:54:05.033 DEBUG:boto:Token: None 2023-10-17T18:54:05.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:05.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jGU/u4XSrZa5/EnVlWtYBxl6dFA= 2023-10-17T18:54:05.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jGU/u4XSrZa5/EnVlWtYBxl6dFA='} 2023-10-17T18:54:05.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003221a22433b2011f-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.080 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.080 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.081 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.081 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.081 DEBUG:boto:Data: 2023-10-17T18:54:05.082 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.083 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.083 DEBUG:boto:Params: {} 2023-10-17T18:54:05.084 DEBUG:boto:Token: None 2023-10-17T18:54:05.084 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c210d63ab69806a-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.089 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.089 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.090 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.090 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.090 DEBUG:boto:Data: 2023-10-17T18:54:05.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.092 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.092 DEBUG:boto:Params: {} 2023-10-17T18:54:05.093 DEBUG:boto:Token: None 2023-10-17T18:54:05.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4WVkSe2oLWOUhVr4fZV7ANnrAe4= 2023-10-17T18:54:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4WVkSe2oLWOUhVr4fZV7ANnrAe4='} 2023-10-17T18:54:05.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cc1731e2dd3ef67-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.101 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.102 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.102 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.102 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.103 DEBUG:boto:Data: 2023-10-17T18:54:05.103 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.104 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.104 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.105 DEBUG:boto:Params: {} 2023-10-17T18:54:05.105 DEBUG:boto:Token: None 2023-10-17T18:54:05.106 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.106 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003723162c647dea37-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.110 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:05.111 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:05.111 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.111 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:05.112 DEBUG:boto:Data: 2023-10-17T18:54:05.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.113 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.113 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.113 DEBUG:boto:Params: {} 2023-10-17T18:54:05.114 DEBUG:boto:Token: None 2023-10-17T18:54:05.114 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:05.115 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GgUYcYIuaKnaqecbigdB0FZcFW4= 2023-10-17T18:54:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GgUYcYIuaKnaqecbigdB0FZcFW4='} 2023-10-17T18:54:05.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000881291dcd604565a-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.163 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.164 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.165 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.165 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.166 DEBUG:boto:Data: 2023-10-17T18:54:05.166 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.167 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.167 DEBUG:boto:Params: {} 2023-10-17T18:54:05.168 DEBUG:boto:Token: None 2023-10-17T18:54:05.168 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.169 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044a73253e71693a6-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.173 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:05.173 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:05.174 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.174 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:05.175 DEBUG:boto:Data: 2023-10-17T18:54:05.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.176 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.176 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.177 DEBUG:boto:Params: {} 2023-10-17T18:54:05.177 DEBUG:boto:Token: None 2023-10-17T18:54:05.177 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:05.178 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+Ij0M7Hrky8B/9En9i82OaqR++4= 2023-10-17T18:54:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+Ij0M7Hrky8B/9En9i82OaqR++4='} 2023-10-17T18:54:05.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077c50b1c73a68ed5-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.185 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.186 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.186 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.187 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.187 DEBUG:boto:Data: 2023-10-17T18:54:05.187 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.188 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.189 DEBUG:boto:Params: {} 2023-10-17T18:54:05.189 DEBUG:boto:Token: None 2023-10-17T18:54:05.189 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.190 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002108096f29272c43-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.194 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:05.195 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:05.195 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.196 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:05.196 DEBUG:boto:Data: 2023-10-17T18:54:05.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.197 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.198 DEBUG:boto:Params: {} 2023-10-17T18:54:05.198 DEBUG:boto:Token: None 2023-10-17T18:54:05.199 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:05.199 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tS+DTjl78SZZKOROpw1r8hRosps= 2023-10-17T18:54:05.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tS+DTjl78SZZKOROpw1r8hRosps='} 2023-10-17T18:54:05.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f84a84bee77bed07-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.243 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.244 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.244 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.244 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.245 DEBUG:boto:Data: 2023-10-17T18:54:05.245 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.246 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.246 DEBUG:boto:Params: {} 2023-10-17T18:54:05.247 DEBUG:boto:Token: None 2023-10-17T18:54:05.247 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb72757b4137fc41-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.252 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:05.253 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:05.253 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.253 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:05.254 DEBUG:boto:Data: 2023-10-17T18:54:05.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.255 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.255 DEBUG:boto:Params: {} 2023-10-17T18:54:05.256 DEBUG:boto:Token: None 2023-10-17T18:54:05.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:05.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZHCMZ7lr3bevJvvnbCg+ZzWUjv4= 2023-10-17T18:54:05.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZHCMZ7lr3bevJvvnbCg+ZzWUjv4='} 2023-10-17T18:54:05.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e746760be8d9a76-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.264 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.264 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.265 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.265 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.266 DEBUG:boto:Data: 2023-10-17T18:54:05.266 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.266 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.267 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.267 DEBUG:boto:Params: {} 2023-10-17T18:54:05.268 DEBUG:boto:Token: None 2023-10-17T18:54:05.268 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.269 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6c5f1fda00f7471-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.273 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:05.273 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:05.274 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.274 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:05.275 DEBUG:boto:Data: 2023-10-17T18:54:05.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.275 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.276 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.276 DEBUG:boto:Params: {} 2023-10-17T18:54:05.277 DEBUG:boto:Token: None 2023-10-17T18:54:05.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:05.278 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y5fnGrtMaaUBSHmPH0FxeuQMt+A= 2023-10-17T18:54:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y5fnGrtMaaUBSHmPH0FxeuQMt+A='} 2023-10-17T18:54:05.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c56b052ed49bf4c-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.323 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.324 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.324 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.325 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.325 DEBUG:boto:Data: 2023-10-17T18:54:05.325 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.326 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.327 DEBUG:boto:Params: {} 2023-10-17T18:54:05.327 DEBUG:boto:Token: None 2023-10-17T18:54:05.328 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3ffa87a3b20dd44-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.332 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:05.333 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:05.333 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.334 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:05.334 DEBUG:boto:Data: 2023-10-17T18:54:05.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.335 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.336 DEBUG:boto:Params: {} 2023-10-17T18:54:05.336 DEBUG:boto:Token: None 2023-10-17T18:54:05.336 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:05.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JfN/1v/FLuKbWmL2+wzO0wVeeno= 2023-10-17T18:54:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JfN/1v/FLuKbWmL2+wzO0wVeeno='} 2023-10-17T18:54:05.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bdf45ea2f52480f-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.344 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.344 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.345 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.345 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.345 DEBUG:boto:Data: 2023-10-17T18:54:05.346 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.347 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.347 DEBUG:boto:Params: {} 2023-10-17T18:54:05.348 DEBUG:boto:Token: None 2023-10-17T18:54:05.348 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010110f98dfea1f5b-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.352 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:05.353 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:05.353 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.354 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:05.354 DEBUG:boto:Data: 2023-10-17T18:54:05.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.355 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.355 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.356 DEBUG:boto:Params: {} 2023-10-17T18:54:05.356 DEBUG:boto:Token: None 2023-10-17T18:54:05.357 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:05.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ju2o4lYPdYhcagwqsAT1C8+Bp/Q= 2023-10-17T18:54:05.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ju2o4lYPdYhcagwqsAT1C8+Bp/Q='} 2023-10-17T18:54:05.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dd1fa9f31f28141-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.403 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.403 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.404 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.404 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.405 DEBUG:boto:Data: 2023-10-17T18:54:05.405 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.406 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.406 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.407 DEBUG:boto:Params: {} 2023-10-17T18:54:05.407 DEBUG:boto:Token: None 2023-10-17T18:54:05.408 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae823f8e982f0ae1-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.412 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:05.413 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:05.413 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.414 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:05.414 DEBUG:boto:Data: 2023-10-17T18:54:05.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.415 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.416 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.416 DEBUG:boto:Params: {} 2023-10-17T18:54:05.416 DEBUG:boto:Token: None 2023-10-17T18:54:05.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:05.417 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EYQUL8szUt3r8FMaowB4L4p4r6Q= 2023-10-17T18:54:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EYQUL8szUt3r8FMaowB4L4p4r6Q='} 2023-10-17T18:54:05.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e2bc566beab35c8-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.425 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.425 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.426 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.426 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.427 DEBUG:boto:Data: 2023-10-17T18:54:05.427 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.428 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.428 DEBUG:boto:Params: {} 2023-10-17T18:54:05.429 DEBUG:boto:Token: None 2023-10-17T18:54:05.429 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000280e307056e3fcc1-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.434 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:05.434 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:05.435 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.435 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:05.436 DEBUG:boto:Data: 2023-10-17T18:54:05.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.437 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.437 DEBUG:boto:Params: {} 2023-10-17T18:54:05.437 DEBUG:boto:Token: None 2023-10-17T18:54:05.438 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:05.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eCPcPZzQkSVtjF5J4FKtasnUrA4= 2023-10-17T18:54:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eCPcPZzQkSVtjF5J4FKtasnUrA4='} 2023-10-17T18:54:05.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000369a54ae41f2813f-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.483 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.483 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.484 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.484 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.484 DEBUG:boto:Data: 2023-10-17T18:54:05.485 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.486 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.486 DEBUG:boto:Params: {} 2023-10-17T18:54:05.487 DEBUG:boto:Token: None 2023-10-17T18:54:05.487 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ec3f5fa2b86b84e-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.492 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:05.492 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:05.493 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.493 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:05.494 DEBUG:boto:Data: 2023-10-17T18:54:05.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.495 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.495 DEBUG:boto:Params: {} 2023-10-17T18:54:05.496 DEBUG:boto:Token: None 2023-10-17T18:54:05.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:05.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R5JZE9LaD/1ONL6poZNNzzeKk1U= 2023-10-17T18:54:05.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R5JZE9LaD/1ONL6poZNNzzeKk1U='} 2023-10-17T18:54:05.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e905522514966ed9-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.503 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.504 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.504 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.504 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.505 DEBUG:boto:Data: 2023-10-17T18:54:05.505 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.506 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.507 DEBUG:boto:Params: {} 2023-10-17T18:54:05.507 DEBUG:boto:Token: None 2023-10-17T18:54:05.508 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000060f9815bb553e94-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.512 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:05.513 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:05.513 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.514 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:05.514 DEBUG:boto:Data: 2023-10-17T18:54:05.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.516 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.516 DEBUG:boto:Params: {} 2023-10-17T18:54:05.517 DEBUG:boto:Token: None 2023-10-17T18:54:05.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:05.518 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4dJ/9hLMKtLqHpGaN3HgYrFAkr8= 2023-10-17T18:54:05.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4dJ/9hLMKtLqHpGaN3HgYrFAkr8='} 2023-10-17T18:54:05.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036473ed4a5bfa1e5-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:05.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:05.564 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:05.619 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:05.619 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:05.668 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:05.668 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:05.679 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 41 2023-10-17T18:54:05.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:05.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:05.752 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:05.804 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:05.804 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:05.846 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000019", 2023-10-17T18:54:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006", 2023-10-17T18:54:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000467", 2023-10-17T18:54:05.847 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000005", 2023-10-17T18:54:05.848 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000056", 2023-10-17T18:54:05.848 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:54:05.848 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000023", 2023-10-17T18:54:05.848 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000317", 2023-10-17T18:54:05.849 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000022", 2023-10-17T18:54:05.849 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:05.849 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021", 2023-10-17T18:54:05.849 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000009", 2023-10-17T18:54:05.850 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000271", 2023-10-17T18:54:05.850 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000008", 2023-10-17T18:54:05.850 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000018", 2023-10-17T18:54:05.850 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000007", 2023-10-17T18:54:05.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000017", 2023-10-17T18:54:05.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000162", 2023-10-17T18:54:05.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000016", 2023-10-17T18:54:05.851 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:54:05.852 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000423", 2023-10-17T18:54:05.852 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:54:05.852 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:54:05.852 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:54:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000283", 2023-10-17T18:54:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:54:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000061", 2023-10-17T18:54:05.853 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000050", 2023-10-17T18:54:05.854 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000029", 2023-10-17T18:54:05.854 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000133" 2023-10-17T18:54:05.854 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:05.854 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:05.856 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.856 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.856 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.857 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.857 DEBUG:boto:Data: 2023-10-17T18:54:05.858 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.859 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.859 DEBUG:boto:Params: {} 2023-10-17T18:54:05.860 DEBUG:boto:Token: None 2023-10-17T18:54:05.860 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000329d6d723083b02a-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.865 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:05.866 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:05.866 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.866 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:05.867 DEBUG:boto:Data: 2023-10-17T18:54:05.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.868 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.868 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.869 DEBUG:boto:Params: {} 2023-10-17T18:54:05.869 DEBUG:boto:Token: None 2023-10-17T18:54:05.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:05.870 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iiJ5DKWYq62oxtWcHimpDaQDkJ0= 2023-10-17T18:54:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iiJ5DKWYq62oxtWcHimpDaQDkJ0='} 2023-10-17T18:54:05.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b213fca3b42220c-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.876 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.877 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.877 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.878 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.878 DEBUG:boto:Data: 2023-10-17T18:54:05.878 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.879 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.879 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.880 DEBUG:boto:Params: {} 2023-10-17T18:54:05.880 DEBUG:boto:Token: None 2023-10-17T18:54:05.880 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008341ca891baaf8c7-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.885 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:05.885 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:05.886 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.886 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:05.887 DEBUG:boto:Data: 2023-10-17T18:54:05.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.887 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.888 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.888 DEBUG:boto:Params: {} 2023-10-17T18:54:05.889 DEBUG:boto:Token: None 2023-10-17T18:54:05.889 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:05.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KH2Mf3pSoCVCeAXcym5aapvdmRQ= 2023-10-17T18:54:05.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KH2Mf3pSoCVCeAXcym5aapvdmRQ='} 2023-10-17T18:54:05.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065b08221b79e72bb-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.897 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.897 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.898 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.898 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.899 DEBUG:boto:Data: 2023-10-17T18:54:05.899 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.900 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.900 DEBUG:boto:Params: {} 2023-10-17T18:54:05.900 DEBUG:boto:Token: None 2023-10-17T18:54:05.901 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080c2491a2393bf21-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.905 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:05.906 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:05.906 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.907 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:05.907 DEBUG:boto:Data: 2023-10-17T18:54:05.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.908 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.908 DEBUG:boto:Params: {} 2023-10-17T18:54:05.909 DEBUG:boto:Token: None 2023-10-17T18:54:05.909 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:05.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ncv6LIPlKBHHPciuXc2AsczjO6Q= 2023-10-17T18:54:05.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ncv6LIPlKBHHPciuXc2AsczjO6Q='} 2023-10-17T18:54:05.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd145050ccfc1dac-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.959 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.960 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.960 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.960 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.961 DEBUG:boto:Data: 2023-10-17T18:54:05.961 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.962 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.962 DEBUG:boto:Params: {} 2023-10-17T18:54:05.963 DEBUG:boto:Token: None 2023-10-17T18:54:05.963 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083efa0a04278a11f-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.968 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:05.969 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:05.969 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.969 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:05.970 DEBUG:boto:Data: 2023-10-17T18:54:05.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.971 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.972 DEBUG:boto:Params: {} 2023-10-17T18:54:05.972 DEBUG:boto:Token: None 2023-10-17T18:54:05.972 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:05.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RgKdBinGOSq5jURmziAQKSeRsvs= 2023-10-17T18:54:05.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RgKdBinGOSq5jURmziAQKSeRsvs='} 2023-10-17T18:54:05.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069fd21b02b6a81d5-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.980 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:05.981 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:05.981 DEBUG:boto:Method: HEAD 2023-10-17T18:54:05.982 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:05.982 DEBUG:boto:Data: 2023-10-17T18:54:05.982 DEBUG:boto:Headers: {} 2023-10-17T18:54:05.983 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.983 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.984 DEBUG:boto:Params: {} 2023-10-17T18:54:05.984 DEBUG:boto:Token: None 2023-10-17T18:54:05.985 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/ 2023-10-17T18:54:05.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4= 2023-10-17T18:54:05.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uQXynO3h7/cPLRze/ADqXzlllc4=', 'Content-Length': '0'} 2023-10-17T18:54:05.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ede1eb715ef360b-00652ed84d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:05.989 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.990 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.990 DEBUG:boto:Method: PUT 2023-10-17T18:54:05.991 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.991 DEBUG:boto:Data: 2023-10-17T18:54:05.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:05.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:05.992 DEBUG:boto:Port: 8000 2023-10-17T18:54:05.993 DEBUG:boto:Params: {} 2023-10-17T18:54:05.993 DEBUG:boto:Token: None 2023-10-17T18:54:05.994 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:05 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:05.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4WVkSe2oLWOUhVr4fZV7ANnrAe4= 2023-10-17T18:54:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:05 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4WVkSe2oLWOUhVr4fZV7ANnrAe4='} 2023-10-17T18:54:06.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0f3a20552596f61-00652ed84d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.043 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.043 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.044 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.044 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.045 DEBUG:boto:Data: 2023-10-17T18:54:06.045 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.046 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.046 DEBUG:boto:Params: {} 2023-10-17T18:54:06.047 DEBUG:boto:Token: None 2023-10-17T18:54:06.047 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003507a85ebf600227-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.052 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:06.052 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:06.053 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.053 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:06.053 DEBUG:boto:Data: 2023-10-17T18:54:06.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.055 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.055 DEBUG:boto:Params: {} 2023-10-17T18:54:06.055 DEBUG:boto:Token: None 2023-10-17T18:54:06.056 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:06.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iww6uQwGFAdv+HDjOuxfxvGFRr4= 2023-10-17T18:54:06.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iww6uQwGFAdv+HDjOuxfxvGFRr4='} 2023-10-17T18:54:06.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9b6f6aa4f1f5878-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.063 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.064 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.064 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.065 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.065 DEBUG:boto:Data: 2023-10-17T18:54:06.065 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.066 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.067 DEBUG:boto:Params: {} 2023-10-17T18:54:06.067 DEBUG:boto:Token: None 2023-10-17T18:54:06.068 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e8b7664b1e6492d-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.072 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:06.072 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:06.073 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.073 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:06.074 DEBUG:boto:Data: 2023-10-17T18:54:06.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.074 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.075 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.075 DEBUG:boto:Params: {} 2023-10-17T18:54:06.076 DEBUG:boto:Token: None 2023-10-17T18:54:06.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:06.077 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6zkpouBsuJuUvGyO1pq1X5kyss= 2023-10-17T18:54:06.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6zkpouBsuJuUvGyO1pq1X5kyss='} 2023-10-17T18:54:06.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099c8375c6836bf85-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.123 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.123 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.124 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.124 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.124 DEBUG:boto:Data: 2023-10-17T18:54:06.125 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.125 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.126 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.126 DEBUG:boto:Params: {} 2023-10-17T18:54:06.127 DEBUG:boto:Token: None 2023-10-17T18:54:06.127 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.127 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000382d2c45dab75b27-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.131 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:06.132 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:06.132 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.133 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:06.133 DEBUG:boto:Data: 2023-10-17T18:54:06.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.134 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.135 DEBUG:boto:Params: {} 2023-10-17T18:54:06.135 DEBUG:boto:Token: None 2023-10-17T18:54:06.135 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:06.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8L8C+s+Ebb1lX/lx48cPzzB2ai0= 2023-10-17T18:54:06.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8L8C+s+Ebb1lX/lx48cPzzB2ai0='} 2023-10-17T18:54:06.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca59a219dbdf8990-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.143 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.143 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.144 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.144 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.144 DEBUG:boto:Data: 2023-10-17T18:54:06.145 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.145 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.146 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.146 DEBUG:boto:Params: {} 2023-10-17T18:54:06.146 DEBUG:boto:Token: None 2023-10-17T18:54:06.147 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.147 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057bf5e8ccab4dc5c-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.151 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:06.152 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:06.152 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.153 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:06.153 DEBUG:boto:Data: 2023-10-17T18:54:06.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.154 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.154 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.155 DEBUG:boto:Params: {} 2023-10-17T18:54:06.155 DEBUG:boto:Token: None 2023-10-17T18:54:06.156 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:06.156 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e6YAjj32GfD9Idrz6DLaKAX4e30= 2023-10-17T18:54:06.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e6YAjj32GfD9Idrz6DLaKAX4e30='} 2023-10-17T18:54:06.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006da8c9d09211c07e-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.203 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.203 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.204 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.204 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.205 DEBUG:boto:Data: 2023-10-17T18:54:06.205 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.206 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.207 DEBUG:boto:Params: {} 2023-10-17T18:54:06.207 DEBUG:boto:Token: None 2023-10-17T18:54:06.208 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b1f307becd52df7-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.212 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:06.213 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:06.213 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.214 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:06.214 DEBUG:boto:Data: 2023-10-17T18:54:06.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.215 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.216 DEBUG:boto:Params: {} 2023-10-17T18:54:06.216 DEBUG:boto:Token: None 2023-10-17T18:54:06.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:06.217 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tRHrnh1702H+9gaOs/Mt/IKkPPI= 2023-10-17T18:54:06.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tRHrnh1702H+9gaOs/Mt/IKkPPI='} 2023-10-17T18:54:06.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b089ef351d4a0697-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.224 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.224 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.225 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.225 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.226 DEBUG:boto:Data: 2023-10-17T18:54:06.226 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.227 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.227 DEBUG:boto:Params: {} 2023-10-17T18:54:06.228 DEBUG:boto:Token: None 2023-10-17T18:54:06.228 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c6118ee23d401ac-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.232 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:06.233 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:06.233 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.234 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:06.234 DEBUG:boto:Data: 2023-10-17T18:54:06.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.235 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.236 DEBUG:boto:Params: {} 2023-10-17T18:54:06.236 DEBUG:boto:Token: None 2023-10-17T18:54:06.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:06.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xfjywfcmd1PDi9oQOCTQq+UROZY= 2023-10-17T18:54:06.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xfjywfcmd1PDi9oQOCTQq+UROZY='} 2023-10-17T18:54:06.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc579d1f86bdfc1b-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.283 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.283 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.284 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.284 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.285 DEBUG:boto:Data: 2023-10-17T18:54:06.285 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.286 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.286 DEBUG:boto:Params: {} 2023-10-17T18:54:06.287 DEBUG:boto:Token: None 2023-10-17T18:54:06.287 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ee6065583bbe625-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.292 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:06.292 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:06.293 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.293 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:06.294 DEBUG:boto:Data: 2023-10-17T18:54:06.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.295 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.295 DEBUG:boto:Params: {} 2023-10-17T18:54:06.296 DEBUG:boto:Token: None 2023-10-17T18:54:06.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:06.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MUFjAmtt/Eugm0RRSLNoif5TNig= 2023-10-17T18:54:06.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MUFjAmtt/Eugm0RRSLNoif5TNig='} 2023-10-17T18:54:06.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006c36698f1fc3bd0-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.304 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.304 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.305 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.305 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.306 DEBUG:boto:Data: 2023-10-17T18:54:06.306 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.307 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.307 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.307 DEBUG:boto:Params: {} 2023-10-17T18:54:06.308 DEBUG:boto:Token: None 2023-10-17T18:54:06.308 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.309 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bcd3a234e14bffe-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.313 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:06.313 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:06.314 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.314 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:06.315 DEBUG:boto:Data: 2023-10-17T18:54:06.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.315 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.316 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.316 DEBUG:boto:Params: {} 2023-10-17T18:54:06.317 DEBUG:boto:Token: None 2023-10-17T18:54:06.317 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:06.318 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XfQ9J87SiWQzTiVj4ljlC8fIU0E= 2023-10-17T18:54:06.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XfQ9J87SiWQzTiVj4ljlC8fIU0E='} 2023-10-17T18:54:06.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006033506610fb1a67-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.363 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.363 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.364 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.364 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.364 DEBUG:boto:Data: 2023-10-17T18:54:06.365 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.366 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.366 DEBUG:boto:Params: {} 2023-10-17T18:54:06.367 DEBUG:boto:Token: None 2023-10-17T18:54:06.367 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec2346a44f91b44-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.372 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:06.372 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:06.373 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.373 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:06.373 DEBUG:boto:Data: 2023-10-17T18:54:06.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.375 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.375 DEBUG:boto:Params: {} 2023-10-17T18:54:06.375 DEBUG:boto:Token: None 2023-10-17T18:54:06.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:06.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QfzstYmVUKCIxw43keW0S3FCwos= 2023-10-17T18:54:06.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QfzstYmVUKCIxw43keW0S3FCwos='} 2023-10-17T18:54:06.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bf4a0bec2f89289-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.384 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.384 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.385 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.385 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.385 DEBUG:boto:Data: 2023-10-17T18:54:06.386 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.387 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.387 DEBUG:boto:Params: {} 2023-10-17T18:54:06.388 DEBUG:boto:Token: None 2023-10-17T18:54:06.388 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0893ed342d62c80-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.392 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:06.393 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:06.393 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.394 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:06.394 DEBUG:boto:Data: 2023-10-17T18:54:06.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.395 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.395 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.395 DEBUG:boto:Params: {} 2023-10-17T18:54:06.396 DEBUG:boto:Token: None 2023-10-17T18:54:06.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:06.397 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yqOEbetsLpAkkgbmPySWfPAjF90= 2023-10-17T18:54:06.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yqOEbetsLpAkkgbmPySWfPAjF90='} 2023-10-17T18:54:06.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edd24414d7a23326-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.443 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.443 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.444 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.444 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.445 DEBUG:boto:Data: 2023-10-17T18:54:06.445 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.446 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.447 DEBUG:boto:Params: {} 2023-10-17T18:54:06.447 DEBUG:boto:Token: None 2023-10-17T18:54:06.448 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d49af65cc0e03873-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.452 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:06.453 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:06.453 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.454 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:06.454 DEBUG:boto:Data: 2023-10-17T18:54:06.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.455 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.456 DEBUG:boto:Params: {} 2023-10-17T18:54:06.456 DEBUG:boto:Token: None 2023-10-17T18:54:06.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:06.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I1fW5oSixn3W0GafIbYiVu0NUE4= 2023-10-17T18:54:06.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I1fW5oSixn3W0GafIbYiVu0NUE4='} 2023-10-17T18:54:06.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ede4bc0ad05bc255-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.465 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.465 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.465 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.466 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.466 DEBUG:boto:Data: 2023-10-17T18:54:06.467 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.467 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.468 DEBUG:boto:Params: {} 2023-10-17T18:54:06.468 DEBUG:boto:Token: None 2023-10-17T18:54:06.469 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000588c7f307db3f6f1-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.473 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:06.474 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:06.474 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.475 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:06.475 DEBUG:boto:Data: 2023-10-17T18:54:06.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.476 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.477 DEBUG:boto:Params: {} 2023-10-17T18:54:06.477 DEBUG:boto:Token: None 2023-10-17T18:54:06.478 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:06.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LXLGbgq3zhM4f10bHQRyjkWs6Ek= 2023-10-17T18:54:06.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LXLGbgq3zhM4f10bHQRyjkWs6Ek='} 2023-10-17T18:54:06.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002481d6affe64c79e-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.523 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.524 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.524 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.525 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.525 DEBUG:boto:Data: 2023-10-17T18:54:06.526 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.527 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.527 DEBUG:boto:Params: {} 2023-10-17T18:54:06.527 DEBUG:boto:Token: None 2023-10-17T18:54:06.528 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5f270578793f733-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.533 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:06.534 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:06.534 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.535 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:06.535 DEBUG:boto:Data: 2023-10-17T18:54:06.536 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.536 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.537 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.537 DEBUG:boto:Params: {} 2023-10-17T18:54:06.537 DEBUG:boto:Token: None 2023-10-17T18:54:06.538 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:06.539 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I/IzY451PydLdDSB8QOcyRNSMRk= 2023-10-17T18:54:06.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I/IzY451PydLdDSB8QOcyRNSMRk='} 2023-10-17T18:54:06.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa8efa8097883f6f-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.545 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.545 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.546 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.546 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.546 DEBUG:boto:Data: 2023-10-17T18:54:06.547 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.547 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.548 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.548 DEBUG:boto:Params: {} 2023-10-17T18:54:06.549 DEBUG:boto:Token: None 2023-10-17T18:54:06.549 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.550 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5906090fd29780d-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.554 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:06.555 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:06.555 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.556 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:06.556 DEBUG:boto:Data: 2023-10-17T18:54:06.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.557 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.558 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.558 DEBUG:boto:Params: {} 2023-10-17T18:54:06.559 DEBUG:boto:Token: None 2023-10-17T18:54:06.559 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:06.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o8LwdO46zm3P9a/AwBy/kA9WFsM= 2023-10-17T18:54:06.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o8LwdO46zm3P9a/AwBy/kA9WFsM='} 2023-10-17T18:54:06.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b505ed3a5c2535c-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.603 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.604 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.604 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.605 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.605 DEBUG:boto:Data: 2023-10-17T18:54:06.606 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.606 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.607 DEBUG:boto:Params: {} 2023-10-17T18:54:06.607 DEBUG:boto:Token: None 2023-10-17T18:54:06.608 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.608 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f4ff3ce56e183bc-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.613 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:06.614 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:06.614 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.615 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:06.615 DEBUG:boto:Data: 2023-10-17T18:54:06.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.616 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.616 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.617 DEBUG:boto:Params: {} 2023-10-17T18:54:06.617 DEBUG:boto:Token: None 2023-10-17T18:54:06.618 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:06.618 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xUA47j2YhJ+fpgCc3Q9y/Bghhtw= 2023-10-17T18:54:06.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xUA47j2YhJ+fpgCc3Q9y/Bghhtw='} 2023-10-17T18:54:06.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae526f6012d511bf-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.625 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.626 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.626 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.627 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.627 DEBUG:boto:Data: 2023-10-17T18:54:06.628 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.628 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.628 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.629 DEBUG:boto:Params: {} 2023-10-17T18:54:06.629 DEBUG:boto:Token: None 2023-10-17T18:54:06.630 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.630 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd439af94ddb47f2-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.634 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:06.635 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:06.635 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.636 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:06.636 DEBUG:boto:Data: 2023-10-17T18:54:06.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.637 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.637 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.638 DEBUG:boto:Params: {} 2023-10-17T18:54:06.638 DEBUG:boto:Token: None 2023-10-17T18:54:06.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:06.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Roio/m+2KU8ZTG4TMRkDMZ2AgG4= 2023-10-17T18:54:06.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Roio/m+2KU8ZTG4TMRkDMZ2AgG4='} 2023-10-17T18:54:06.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a182924480ba998-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.683 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.683 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.684 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.684 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.685 DEBUG:boto:Data: 2023-10-17T18:54:06.685 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.686 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.686 DEBUG:boto:Params: {} 2023-10-17T18:54:06.687 DEBUG:boto:Token: None 2023-10-17T18:54:06.687 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.688 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082fec7713db6cb92-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.692 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:06.693 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:06.693 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.694 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:06.694 DEBUG:boto:Data: 2023-10-17T18:54:06.694 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.695 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.695 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.696 DEBUG:boto:Params: {} 2023-10-17T18:54:06.696 DEBUG:boto:Token: None 2023-10-17T18:54:06.697 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:06.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gWjxbrHefs5/qhdAFfDAEd1eTNI= 2023-10-17T18:54:06.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gWjxbrHefs5/qhdAFfDAEd1eTNI='} 2023-10-17T18:54:06.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c905ac2e82c2efa0-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.705 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.705 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.706 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.706 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.707 DEBUG:boto:Data: 2023-10-17T18:54:06.707 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.708 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.708 DEBUG:boto:Params: {} 2023-10-17T18:54:06.709 DEBUG:boto:Token: None 2023-10-17T18:54:06.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce3f6eff94263866-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.714 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:06.714 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:06.715 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.715 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:06.716 DEBUG:boto:Data: 2023-10-17T18:54:06.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.717 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.717 DEBUG:boto:Params: {} 2023-10-17T18:54:06.718 DEBUG:boto:Token: None 2023-10-17T18:54:06.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:06.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bqMHv4zk5JQxm3Qf3hobBSlzHDY= 2023-10-17T18:54:06.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bqMHv4zk5JQxm3Qf3hobBSlzHDY='} 2023-10-17T18:54:06.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6c2149a64a870ef-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.763 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.763 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.764 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.764 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.765 DEBUG:boto:Data: 2023-10-17T18:54:06.765 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.766 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.766 DEBUG:boto:Params: {} 2023-10-17T18:54:06.767 DEBUG:boto:Token: None 2023-10-17T18:54:06.767 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cb332d3ded0828d-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.772 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:06.772 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:06.773 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.773 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:06.774 DEBUG:boto:Data: 2023-10-17T18:54:06.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.775 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.775 DEBUG:boto:Params: {} 2023-10-17T18:54:06.776 DEBUG:boto:Token: None 2023-10-17T18:54:06.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:06.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xX+u2VJfjZA2Xao66zrbGqfG6zw= 2023-10-17T18:54:06.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xX+u2VJfjZA2Xao66zrbGqfG6zw='} 2023-10-17T18:54:06.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c16ae1ce1a75066e-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.783 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.784 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.784 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.784 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.785 DEBUG:boto:Data: 2023-10-17T18:54:06.785 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.786 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.786 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.786 DEBUG:boto:Params: {} 2023-10-17T18:54:06.787 DEBUG:boto:Token: None 2023-10-17T18:54:06.787 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cddaba9beebb28bc-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.792 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:06.793 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:06.793 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.794 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:06.794 DEBUG:boto:Data: 2023-10-17T18:54:06.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.795 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.795 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.796 DEBUG:boto:Params: {} 2023-10-17T18:54:06.796 DEBUG:boto:Token: None 2023-10-17T18:54:06.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:06.797 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L7un8cJfPWDuCQsU74r18babjdY= 2023-10-17T18:54:06.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L7un8cJfPWDuCQsU74r18babjdY='} 2023-10-17T18:54:06.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000023691028ffa7624-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.843 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.844 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.844 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.844 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.845 DEBUG:boto:Data: 2023-10-17T18:54:06.845 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.846 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.846 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.847 DEBUG:boto:Params: {} 2023-10-17T18:54:06.847 DEBUG:boto:Token: None 2023-10-17T18:54:06.848 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.848 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4c0ff0a624e8d12-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.852 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:06.853 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:06.853 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.854 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:06.854 DEBUG:boto:Data: 2023-10-17T18:54:06.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.855 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.855 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.856 DEBUG:boto:Params: {} 2023-10-17T18:54:06.856 DEBUG:boto:Token: None 2023-10-17T18:54:06.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:06.857 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VECQwHAdKjPr9t+0rzrijed7pHg= 2023-10-17T18:54:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VECQwHAdKjPr9t+0rzrijed7pHg='} 2023-10-17T18:54:06.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002498456b7f8abc68-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.863 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.864 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.864 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.864 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.865 DEBUG:boto:Data: 2023-10-17T18:54:06.865 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.866 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.866 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.866 DEBUG:boto:Params: {} 2023-10-17T18:54:06.867 DEBUG:boto:Token: None 2023-10-17T18:54:06.867 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.868 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2ac77df07351ba2-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.872 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:06.873 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:06.873 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.874 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:06.874 DEBUG:boto:Data: 2023-10-17T18:54:06.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.875 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.876 DEBUG:boto:Params: {} 2023-10-17T18:54:06.876 DEBUG:boto:Token: None 2023-10-17T18:54:06.877 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:06.877 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6DInmFG3UURiOIM6sTi1/HGUHiQ= 2023-10-17T18:54:06.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6DInmFG3UURiOIM6sTi1/HGUHiQ='} 2023-10-17T18:54:06.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000898636616ea1f54a-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.923 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:06.923 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:06.924 DEBUG:boto:Method: HEAD 2023-10-17T18:54:06.924 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:06.925 DEBUG:boto:Data: 2023-10-17T18:54:06.925 DEBUG:boto:Headers: {} 2023-10-17T18:54:06.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.926 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.926 DEBUG:boto:Params: {} 2023-10-17T18:54:06.927 DEBUG:boto:Token: None 2023-10-17T18:54:06.927 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/ 2023-10-17T18:54:06.928 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ= 2023-10-17T18:54:06.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jt/9GMRxnFVZaeRhfTD8E7juoeQ=', 'Content-Length': '0'} 2023-10-17T18:54:06.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4c3b3bbe58deef3-00652ed84e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.932 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:06.933 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:06.933 DEBUG:boto:Method: PUT 2023-10-17T18:54:06.934 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:06.934 DEBUG:boto:Data: 2023-10-17T18:54:06.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:06.935 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:06.936 DEBUG:boto:Port: 8000 2023-10-17T18:54:06.936 DEBUG:boto:Params: {} 2023-10-17T18:54:06.937 DEBUG:boto:Token: None 2023-10-17T18:54:06.937 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:06 GMT /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:06.938 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D7Qm+Yba7HTnrNzNiY19J4vS3NY= 2023-10-17T18:54:06.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D7Qm+Yba7HTnrNzNiY19J4vS3NY='} 2023-10-17T18:54:06.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f1aa470ffdbb9fe-00652ed84e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:06.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:06.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:06.946 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:07.001 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:07.001 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:07.053 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:54:07.053 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-58 2023-10-17T18:54:07.064 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 51 2023-10-17T18:54:07.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:07.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:07.125 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:07.178 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:07.179 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:07.223 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:07.223 INFO:teuthology.orchestra.run.smithi049.stdout: "objs": [ 2023-10-17T18:54:07.223 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000060", 2023-10-17T18:54:07.223 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000000", 2023-10-17T18:54:07.224 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000011", 2023-10-17T18:54:07.224 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000001", 2023-10-17T18:54:07.224 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000033", 2023-10-17T18:54:07.224 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000065", 2023-10-17T18:54:07.224 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000005", 2023-10-17T18:54:07.225 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000016", 2023-10-17T18:54:07.225 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000048", 2023-10-17T18:54:07.225 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000015", 2023-10-17T18:54:07.225 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000026", 2023-10-17T18:54:07.226 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000030", 2023-10-17T18:54:07.226 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000062", 2023-10-17T18:54:07.226 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000009", 2023-10-17T18:54:07.226 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000040", 2023-10-17T18:54:07.227 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000024", 2023-10-17T18:54:07.227 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000012", 2023-10-17T18:54:07.227 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000002", 2023-10-17T18:54:07.227 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000013", 2023-10-17T18:54:07.227 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000045", 2023-10-17T18:54:07.228 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000006", 2023-10-17T18:54:07.228 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000017", 2023-10-17T18:54:07.228 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000007", 2023-10-17T18:54:07.228 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000039", 2023-10-17T18:54:07.228 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000027", 2023-10-17T18:54:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000010", 2023-10-17T18:54:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000021", 2023-10-17T18:54:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000053", 2023-10-17T18:54:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000020", 2023-10-17T18:54:07.230 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000004", 2023-10-17T18:54:07.230 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000036", 2023-10-17T18:54:07.230 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000003", 2023-10-17T18:54:07.230 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000014", 2023-10-17T18:54:07.231 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000019", 2023-10-17T18:54:07.231 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000057", 2023-10-17T18:54:07.231 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000018", 2023-10-17T18:54:07.231 INFO:teuthology.orchestra.run.smithi049.stdout: "obj00000000000000000008" 2023-10-17T18:54:07.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:07.232 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:07.233 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.233 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.234 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.234 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.235 DEBUG:boto:Data: 2023-10-17T18:54:07.235 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.236 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.236 DEBUG:boto:Params: {} 2023-10-17T18:54:07.237 DEBUG:boto:Token: None 2023-10-17T18:54:07.237 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5c6158ec870c4b-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.242 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:07.243 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:07.243 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.244 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:07.244 DEBUG:boto:Data: 2023-10-17T18:54:07.244 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.245 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.246 DEBUG:boto:Params: {} 2023-10-17T18:54:07.246 DEBUG:boto:Token: None 2023-10-17T18:54:07.247 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:07.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q1l1pQM6p8OGBTGw2pb7iYYH0P0= 2023-10-17T18:54:07.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q1l1pQM6p8OGBTGw2pb7iYYH0P0='} 2023-10-17T18:54:07.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043f1240e892923fa-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.254 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.254 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.255 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.255 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.256 DEBUG:boto:Data: 2023-10-17T18:54:07.256 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.257 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.257 DEBUG:boto:Params: {} 2023-10-17T18:54:07.258 DEBUG:boto:Token: None 2023-10-17T18:54:07.258 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061de1d7c734ca678-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.265 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:07.265 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:07.266 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.266 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:07.267 DEBUG:boto:Data: 2023-10-17T18:54:07.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.268 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.268 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.268 DEBUG:boto:Params: {} 2023-10-17T18:54:07.269 DEBUG:boto:Token: None 2023-10-17T18:54:07.269 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:07.270 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hwkD/w1JCHWt7J2twhrbh7mGwL4= 2023-10-17T18:54:07.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hwkD/w1JCHWt7J2twhrbh7mGwL4='} 2023-10-17T18:54:07.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed3826684de6592b-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.315 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.315 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.316 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.316 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.316 DEBUG:boto:Data: 2023-10-17T18:54:07.317 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.318 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.318 DEBUG:boto:Params: {} 2023-10-17T18:54:07.319 DEBUG:boto:Token: None 2023-10-17T18:54:07.319 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b3d6f28afa52efa-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.324 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:07.324 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:07.325 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.325 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:07.325 DEBUG:boto:Data: 2023-10-17T18:54:07.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.327 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.327 DEBUG:boto:Params: {} 2023-10-17T18:54:07.328 DEBUG:boto:Token: None 2023-10-17T18:54:07.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:07.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n0Gkr0Np+bDaWBg9556VfqJcZFg= 2023-10-17T18:54:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n0Gkr0Np+bDaWBg9556VfqJcZFg='} 2023-10-17T18:54:07.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000501f892ccc5106ca-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.336 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.336 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.337 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.337 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.338 DEBUG:boto:Data: 2023-10-17T18:54:07.338 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.339 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.339 DEBUG:boto:Params: {} 2023-10-17T18:54:07.340 DEBUG:boto:Token: None 2023-10-17T18:54:07.340 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693db9d281f0228a-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.346 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:07.346 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:07.346 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.347 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:07.347 DEBUG:boto:Data: 2023-10-17T18:54:07.348 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.349 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.349 DEBUG:boto:Params: {} 2023-10-17T18:54:07.349 DEBUG:boto:Token: None 2023-10-17T18:54:07.350 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:07.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/W7kbMIs30dEsmctQRDaUrapzOg= 2023-10-17T18:54:07.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/W7kbMIs30dEsmctQRDaUrapzOg='} 2023-10-17T18:54:07.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fe4fce8659c7712-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.395 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.395 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.396 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.396 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.396 DEBUG:boto:Data: 2023-10-17T18:54:07.397 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.398 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.398 DEBUG:boto:Params: {} 2023-10-17T18:54:07.399 DEBUG:boto:Token: None 2023-10-17T18:54:07.399 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.400 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069259c03a3b6979b-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.404 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:07.405 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:07.405 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.406 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:07.406 DEBUG:boto:Data: 2023-10-17T18:54:07.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.407 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.408 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.408 DEBUG:boto:Params: {} 2023-10-17T18:54:07.408 DEBUG:boto:Token: None 2023-10-17T18:54:07.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:07.409 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rYoFA7pJ6bDuStOxTXP3e8tEeYw= 2023-10-17T18:54:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rYoFA7pJ6bDuStOxTXP3e8tEeYw='} 2023-10-17T18:54:07.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bb4a20d71262ba8-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.416 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.417 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.417 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.417 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.418 DEBUG:boto:Data: 2023-10-17T18:54:07.418 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.419 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.419 DEBUG:boto:Params: {} 2023-10-17T18:54:07.420 DEBUG:boto:Token: None 2023-10-17T18:54:07.420 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e1b5f6df5c0a964-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.426 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:07.426 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:07.426 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.427 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:07.427 DEBUG:boto:Data: 2023-10-17T18:54:07.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.428 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.429 DEBUG:boto:Params: {} 2023-10-17T18:54:07.429 DEBUG:boto:Token: None 2023-10-17T18:54:07.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:07.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hjef9+dkpotpdieVdYeTW08cp7E= 2023-10-17T18:54:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hjef9+dkpotpdieVdYeTW08cp7E='} 2023-10-17T18:54:07.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006287f9a5100f4601-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.475 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.476 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.476 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.476 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.477 DEBUG:boto:Data: 2023-10-17T18:54:07.477 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.478 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.478 DEBUG:boto:Params: {} 2023-10-17T18:54:07.479 DEBUG:boto:Token: None 2023-10-17T18:54:07.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d93e7859c3c99da-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.484 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:07.485 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:07.485 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.486 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:07.486 DEBUG:boto:Data: 2023-10-17T18:54:07.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.487 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.488 DEBUG:boto:Params: {} 2023-10-17T18:54:07.488 DEBUG:boto:Token: None 2023-10-17T18:54:07.489 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:07.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KUS2EooZfq1snFXcpxLor967VBo= 2023-10-17T18:54:07.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KUS2EooZfq1snFXcpxLor967VBo='} 2023-10-17T18:54:07.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd96642162b9798d-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.497 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.497 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.497 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.498 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.498 DEBUG:boto:Data: 2023-10-17T18:54:07.499 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.499 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.500 DEBUG:boto:Params: {} 2023-10-17T18:54:07.500 DEBUG:boto:Token: None 2023-10-17T18:54:07.500 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b3b978d7dc7e70b-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.505 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:07.505 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:07.506 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.506 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:07.507 DEBUG:boto:Data: 2023-10-17T18:54:07.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.508 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.508 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.508 DEBUG:boto:Params: {} 2023-10-17T18:54:07.509 DEBUG:boto:Token: None 2023-10-17T18:54:07.509 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:07.510 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kVDrHvypLAnSDperdW2joGY6rN8= 2023-10-17T18:54:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVDrHvypLAnSDperdW2joGY6rN8='} 2023-10-17T18:54:07.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080fe3fe6e433bbfb-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.559 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.560 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.560 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.560 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.561 DEBUG:boto:Data: 2023-10-17T18:54:07.561 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.562 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.562 DEBUG:boto:Params: {} 2023-10-17T18:54:07.563 DEBUG:boto:Token: None 2023-10-17T18:54:07.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.564 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094b66d0a568a489e-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.568 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:07.568 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:07.569 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.569 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:07.570 DEBUG:boto:Data: 2023-10-17T18:54:07.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.571 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.571 DEBUG:boto:Params: {} 2023-10-17T18:54:07.572 DEBUG:boto:Token: None 2023-10-17T18:54:07.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:07.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uH2QIYrL41/vHMMQmBoU5XVWkJg= 2023-10-17T18:54:07.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uH2QIYrL41/vHMMQmBoU5XVWkJg='} 2023-10-17T18:54:07.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000025ab6ef19d57226-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.580 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.580 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.580 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.581 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.581 DEBUG:boto:Data: 2023-10-17T18:54:07.581 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.582 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.583 DEBUG:boto:Params: {} 2023-10-17T18:54:07.583 DEBUG:boto:Token: None 2023-10-17T18:54:07.584 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008bbf9cfb72baf16-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.588 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:07.589 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:07.589 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.590 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:07.590 DEBUG:boto:Data: 2023-10-17T18:54:07.590 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.591 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.592 DEBUG:boto:Params: {} 2023-10-17T18:54:07.592 DEBUG:boto:Token: None 2023-10-17T18:54:07.593 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:07.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZjLwjwJRBxxWZzc3pDXELXrkD04= 2023-10-17T18:54:07.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZjLwjwJRBxxWZzc3pDXELXrkD04='} 2023-10-17T18:54:07.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdd9840ecf773cf9-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.639 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.639 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.640 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.640 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.641 DEBUG:boto:Data: 2023-10-17T18:54:07.641 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.642 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.642 DEBUG:boto:Params: {} 2023-10-17T18:54:07.643 DEBUG:boto:Token: None 2023-10-17T18:54:07.643 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bc201b34cf4bd1a-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.648 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:07.648 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:07.649 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.649 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:07.650 DEBUG:boto:Data: 2023-10-17T18:54:07.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.651 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.651 DEBUG:boto:Params: {} 2023-10-17T18:54:07.652 DEBUG:boto:Token: None 2023-10-17T18:54:07.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:07.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FhCk9SP1VoCL2/sLSLCJCWjwONA= 2023-10-17T18:54:07.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FhCk9SP1VoCL2/sLSLCJCWjwONA='} 2023-10-17T18:54:07.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a6b6299367f7a08-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.660 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.660 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.660 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.661 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.661 DEBUG:boto:Data: 2023-10-17T18:54:07.662 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.662 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.663 DEBUG:boto:Params: {} 2023-10-17T18:54:07.663 DEBUG:boto:Token: None 2023-10-17T18:54:07.664 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa13d3db6f286299-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.669 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:07.669 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:07.670 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.670 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:07.671 DEBUG:boto:Data: 2023-10-17T18:54:07.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.671 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.672 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.672 DEBUG:boto:Params: {} 2023-10-17T18:54:07.673 DEBUG:boto:Token: None 2023-10-17T18:54:07.673 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:07.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fn/ebWrESj+7LhoNsmoo7NwCUog= 2023-10-17T18:54:07.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fn/ebWrESj+7LhoNsmoo7NwCUog='} 2023-10-17T18:54:07.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000238c90ddb4c3e049-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.719 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.720 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.720 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.720 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.721 DEBUG:boto:Data: 2023-10-17T18:54:07.721 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.722 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.722 DEBUG:boto:Params: {} 2023-10-17T18:54:07.723 DEBUG:boto:Token: None 2023-10-17T18:54:07.723 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1e050cb94333726-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.728 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:07.729 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:07.729 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.730 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:07.730 DEBUG:boto:Data: 2023-10-17T18:54:07.730 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.731 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.732 DEBUG:boto:Params: {} 2023-10-17T18:54:07.732 DEBUG:boto:Token: None 2023-10-17T18:54:07.733 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:07.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CHYvELDSdedAIwM4SgnGMk1iHvM= 2023-10-17T18:54:07.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CHYvELDSdedAIwM4SgnGMk1iHvM='} 2023-10-17T18:54:07.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073736809ed9aef8e-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.739 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.739 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.740 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.740 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.741 DEBUG:boto:Data: 2023-10-17T18:54:07.741 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.742 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.742 DEBUG:boto:Params: {} 2023-10-17T18:54:07.743 DEBUG:boto:Token: None 2023-10-17T18:54:07.743 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024d63609c3932453-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.748 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:07.749 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:07.749 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.750 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:07.750 DEBUG:boto:Data: 2023-10-17T18:54:07.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.751 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.751 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.752 DEBUG:boto:Params: {} 2023-10-17T18:54:07.752 DEBUG:boto:Token: None 2023-10-17T18:54:07.753 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:07.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8exUQGLZgUrL4WJpAQtuoCgGV/o= 2023-10-17T18:54:07.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8exUQGLZgUrL4WJpAQtuoCgGV/o='} 2023-10-17T18:54:07.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d9595fe18fef710-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.799 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.800 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.800 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.800 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.801 DEBUG:boto:Data: 2023-10-17T18:54:07.801 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.802 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.802 DEBUG:boto:Params: {} 2023-10-17T18:54:07.803 DEBUG:boto:Token: None 2023-10-17T18:54:07.803 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e6e540460e45051-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.809 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:07.809 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:07.809 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.810 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:07.810 DEBUG:boto:Data: 2023-10-17T18:54:07.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.811 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.812 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.812 DEBUG:boto:Params: {} 2023-10-17T18:54:07.812 DEBUG:boto:Token: None 2023-10-17T18:54:07.813 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:07.813 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7EJ3RIMYURl6iws2luFvmh7lh+c= 2023-10-17T18:54:07.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7EJ3RIMYURl6iws2luFvmh7lh+c='} 2023-10-17T18:54:07.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca54faab78d9981a-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.821 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.821 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.821 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.822 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.822 DEBUG:boto:Data: 2023-10-17T18:54:07.823 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.823 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.823 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.824 DEBUG:boto:Params: {} 2023-10-17T18:54:07.824 DEBUG:boto:Token: None 2023-10-17T18:54:07.825 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c9fdc57f3f70809-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.829 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:07.830 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:07.830 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.831 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:07.831 DEBUG:boto:Data: 2023-10-17T18:54:07.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.832 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.832 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.833 DEBUG:boto:Params: {} 2023-10-17T18:54:07.833 DEBUG:boto:Token: None 2023-10-17T18:54:07.834 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:07.834 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vnktSgzdWYJJKtbWjLuO3gUc0xc= 2023-10-17T18:54:07.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vnktSgzdWYJJKtbWjLuO3gUc0xc='} 2023-10-17T18:54:07.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000825eaaa020af8384-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.879 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.879 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.880 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.880 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.881 DEBUG:boto:Data: 2023-10-17T18:54:07.881 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.881 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.882 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.882 DEBUG:boto:Params: {} 2023-10-17T18:54:07.883 DEBUG:boto:Token: None 2023-10-17T18:54:07.883 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2c69621ac3e44ee-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.888 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:07.889 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:07.889 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.889 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:07.890 DEBUG:boto:Data: 2023-10-17T18:54:07.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.891 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.891 DEBUG:boto:Params: {} 2023-10-17T18:54:07.892 DEBUG:boto:Token: None 2023-10-17T18:54:07.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:07.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xIj2umKCeo4sKWq+VC0XRe8Zjow= 2023-10-17T18:54:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xIj2umKCeo4sKWq+VC0XRe8Zjow='} 2023-10-17T18:54:07.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fe080c898ce84a3-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.900 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.900 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.901 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.901 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.901 DEBUG:boto:Data: 2023-10-17T18:54:07.902 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.902 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.903 DEBUG:boto:Params: {} 2023-10-17T18:54:07.903 DEBUG:boto:Token: None 2023-10-17T18:54:07.904 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000469f86ce8c19f23a-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.908 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:07.909 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:07.909 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.909 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:07.910 DEBUG:boto:Data: 2023-10-17T18:54:07.910 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.911 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.911 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.912 DEBUG:boto:Params: {} 2023-10-17T18:54:07.912 DEBUG:boto:Token: None 2023-10-17T18:54:07.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:07.913 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0RuPFVNvVlcG7txuG2t1h8ErZo8= 2023-10-17T18:54:07.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0RuPFVNvVlcG7txuG2t1h8ErZo8='} 2023-10-17T18:54:07.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b9e1a7d63423ec1-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.959 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.959 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.960 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.960 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.961 DEBUG:boto:Data: 2023-10-17T18:54:07.961 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.962 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.962 DEBUG:boto:Params: {} 2023-10-17T18:54:07.963 DEBUG:boto:Token: None 2023-10-17T18:54:07.963 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee566d151ef5b96-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.968 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:07.969 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:07.969 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.969 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:07.970 DEBUG:boto:Data: 2023-10-17T18:54:07.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.971 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.971 DEBUG:boto:Params: {} 2023-10-17T18:54:07.972 DEBUG:boto:Token: None 2023-10-17T18:54:07.972 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:07.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VotkUp73L1FUbLlmgl/I3Ez3ByE= 2023-10-17T18:54:07.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VotkUp73L1FUbLlmgl/I3Ez3ByE='} 2023-10-17T18:54:07.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c11f42c2d246d57-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.980 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:07.981 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:07.981 DEBUG:boto:Method: HEAD 2023-10-17T18:54:07.981 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:07.982 DEBUG:boto:Data: 2023-10-17T18:54:07.982 DEBUG:boto:Headers: {} 2023-10-17T18:54:07.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.983 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.983 DEBUG:boto:Params: {} 2023-10-17T18:54:07.984 DEBUG:boto:Token: None 2023-10-17T18:54:07.984 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/ 2023-10-17T18:54:07.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs= 2023-10-17T18:54:07.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HhUAhjENteWCiwv1o6zm1Pvv1Vs=', 'Content-Length': '0'} 2023-10-17T18:54:07.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918fb5af1d85726f-00652ed84f-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:07.989 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:07.989 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:07.990 DEBUG:boto:Method: PUT 2023-10-17T18:54:07.990 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:07.990 DEBUG:boto:Data: 2023-10-17T18:54:07.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:07.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:07.992 DEBUG:boto:Port: 8000 2023-10-17T18:54:07.992 DEBUG:boto:Params: {} 2023-10-17T18:54:07.992 DEBUG:boto:Token: None 2023-10-17T18:54:07.993 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:07 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:07.993 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nYYIHAmWp8tFWdcXEYH01h8h6e0= 2023-10-17T18:54:07.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nYYIHAmWp8tFWdcXEYH01h8h6e0='} 2023-10-17T18:54:08.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfa9fc9ca36ad24b-00652ed84f-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.039 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.040 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.040 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.040 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.041 DEBUG:boto:Data: 2023-10-17T18:54:08.041 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.042 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.042 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.043 DEBUG:boto:Params: {} 2023-10-17T18:54:08.043 DEBUG:boto:Token: None 2023-10-17T18:54:08.043 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.044 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b437e678b323017-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.049 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:08.049 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:08.049 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.050 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:08.050 DEBUG:boto:Data: 2023-10-17T18:54:08.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.051 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.051 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.052 DEBUG:boto:Params: {} 2023-10-17T18:54:08.052 DEBUG:boto:Token: None 2023-10-17T18:54:08.053 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:08.053 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i88rp9WeU27KYgjSOHEgxD5BI/A= 2023-10-17T18:54:08.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i88rp9WeU27KYgjSOHEgxD5BI/A='} 2023-10-17T18:54:08.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9f19a6d61e07eaa-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.059 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.060 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.060 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.060 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.061 DEBUG:boto:Data: 2023-10-17T18:54:08.061 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.062 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.062 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.063 DEBUG:boto:Params: {} 2023-10-17T18:54:08.063 DEBUG:boto:Token: None 2023-10-17T18:54:08.063 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.064 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089af51670357958b-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.069 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:08.069 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:08.070 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.070 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:08.070 DEBUG:boto:Data: 2023-10-17T18:54:08.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.071 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.072 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.072 DEBUG:boto:Params: {} 2023-10-17T18:54:08.073 DEBUG:boto:Token: None 2023-10-17T18:54:08.073 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:08.073 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uiQmQZ5Wc7/Dwx+/vskoyR9TY08= 2023-10-17T18:54:08.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uiQmQZ5Wc7/Dwx+/vskoyR9TY08='} 2023-10-17T18:54:08.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000150352a1463ede63-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.123 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.123 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.124 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.124 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.124 DEBUG:boto:Data: 2023-10-17T18:54:08.125 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.125 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.125 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.126 DEBUG:boto:Params: {} 2023-10-17T18:54:08.126 DEBUG:boto:Token: None 2023-10-17T18:54:08.127 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.127 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056df7106f36ea0ba-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.132 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:08.133 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:08.133 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.134 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:08.134 DEBUG:boto:Data: 2023-10-17T18:54:08.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.135 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.136 DEBUG:boto:Params: {} 2023-10-17T18:54:08.136 DEBUG:boto:Token: None 2023-10-17T18:54:08.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:08.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AUyzNWZtEITonVYObjeWBAcIO8Y= 2023-10-17T18:54:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AUyzNWZtEITonVYObjeWBAcIO8Y='} 2023-10-17T18:54:08.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000709bc20641512521-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.145 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.145 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.146 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.146 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.147 DEBUG:boto:Data: 2023-10-17T18:54:08.147 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.147 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.148 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.148 DEBUG:boto:Params: {} 2023-10-17T18:54:08.149 DEBUG:boto:Token: None 2023-10-17T18:54:08.149 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cddab4766b9ca7c-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.154 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:08.154 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:08.155 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.155 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:08.155 DEBUG:boto:Data: 2023-10-17T18:54:08.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.157 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.157 DEBUG:boto:Params: {} 2023-10-17T18:54:08.158 DEBUG:boto:Token: None 2023-10-17T18:54:08.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:08.159 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6LcBnC+sPI90OO6Tcld5BGRCabM= 2023-10-17T18:54:08.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6LcBnC+sPI90OO6Tcld5BGRCabM='} 2023-10-17T18:54:08.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000981d396eecd8cea8-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.207 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.208 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.208 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.208 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.209 DEBUG:boto:Data: 2023-10-17T18:54:08.209 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.210 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.210 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.210 DEBUG:boto:Params: {} 2023-10-17T18:54:08.211 DEBUG:boto:Token: None 2023-10-17T18:54:08.211 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c9a76d74f780660-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.216 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:08.217 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:08.217 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.218 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:08.218 DEBUG:boto:Data: 2023-10-17T18:54:08.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.219 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.219 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.220 DEBUG:boto:Params: {} 2023-10-17T18:54:08.220 DEBUG:boto:Token: None 2023-10-17T18:54:08.221 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:08.221 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1ySHAQaA8y0D2R82r9HMGDpVZt0= 2023-10-17T18:54:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1ySHAQaA8y0D2R82r9HMGDpVZt0='} 2023-10-17T18:54:08.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57d1043d57cfb8a-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.228 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.229 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.229 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.230 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.230 DEBUG:boto:Data: 2023-10-17T18:54:08.230 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.231 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.231 DEBUG:boto:Params: {} 2023-10-17T18:54:08.232 DEBUG:boto:Token: None 2023-10-17T18:54:08.232 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.233 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b41e9f6fbc85a30-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.237 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:08.238 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:08.238 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.238 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:08.239 DEBUG:boto:Data: 2023-10-17T18:54:08.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.240 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.240 DEBUG:boto:Params: {} 2023-10-17T18:54:08.241 DEBUG:boto:Token: None 2023-10-17T18:54:08.241 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:08.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yYSIQSUNs6ZFi4xW6Rfro54r1f8= 2023-10-17T18:54:08.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yYSIQSUNs6ZFi4xW6Rfro54r1f8='} 2023-10-17T18:54:08.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078c37b0bf07a7995-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.287 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.287 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.288 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.288 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.288 DEBUG:boto:Data: 2023-10-17T18:54:08.289 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.290 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.290 DEBUG:boto:Params: {} 2023-10-17T18:54:08.290 DEBUG:boto:Token: None 2023-10-17T18:54:08.291 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41a07ed2197fb8e-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.295 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:08.296 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:08.296 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.297 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:08.297 DEBUG:boto:Data: 2023-10-17T18:54:08.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.298 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.299 DEBUG:boto:Params: {} 2023-10-17T18:54:08.299 DEBUG:boto:Token: None 2023-10-17T18:54:08.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:08.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w8zPLfNSq7Agx78UU/TJqSMnas8= 2023-10-17T18:54:08.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w8zPLfNSq7Agx78UU/TJqSMnas8='} 2023-10-17T18:54:08.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005640a2fe0218767d-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.308 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.308 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.309 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.309 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.310 DEBUG:boto:Data: 2023-10-17T18:54:08.310 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.311 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.311 DEBUG:boto:Params: {} 2023-10-17T18:54:08.311 DEBUG:boto:Token: None 2023-10-17T18:54:08.312 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e473e40dd73690de-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.317 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:08.317 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:08.318 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.318 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:08.318 DEBUG:boto:Data: 2023-10-17T18:54:08.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.320 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.320 DEBUG:boto:Params: {} 2023-10-17T18:54:08.320 DEBUG:boto:Token: None 2023-10-17T18:54:08.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:08.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2C55WI0ybWW4lJRT7j5AfT5uQRY= 2023-10-17T18:54:08.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2C55WI0ybWW4lJRT7j5AfT5uQRY='} 2023-10-17T18:54:08.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000348ad9333d48dc94-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.367 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.367 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.368 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.368 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.369 DEBUG:boto:Data: 2023-10-17T18:54:08.369 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.370 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.370 DEBUG:boto:Params: {} 2023-10-17T18:54:08.371 DEBUG:boto:Token: None 2023-10-17T18:54:08.371 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001df96a8ecba05abb-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.376 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:08.376 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:08.377 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.377 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:08.377 DEBUG:boto:Data: 2023-10-17T18:54:08.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.378 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.379 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.379 DEBUG:boto:Params: {} 2023-10-17T18:54:08.379 DEBUG:boto:Token: None 2023-10-17T18:54:08.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:08.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yc4zBCp1uzbIbWC+PvM15pDb1tM= 2023-10-17T18:54:08.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yc4zBCp1uzbIbWC+PvM15pDb1tM='} 2023-10-17T18:54:08.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002221a06612cd35ca-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.388 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.388 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.388 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.389 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.389 DEBUG:boto:Data: 2023-10-17T18:54:08.390 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.390 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.391 DEBUG:boto:Params: {} 2023-10-17T18:54:08.391 DEBUG:boto:Token: None 2023-10-17T18:54:08.392 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4dc9c038b9ea066-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.396 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:08.397 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:08.397 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.398 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:08.398 DEBUG:boto:Data: 2023-10-17T18:54:08.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.399 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.400 DEBUG:boto:Params: {} 2023-10-17T18:54:08.400 DEBUG:boto:Token: None 2023-10-17T18:54:08.401 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:08.401 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e7Fn6LZ/84xDoyktWWxbfFLGkVM= 2023-10-17T18:54:08.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e7Fn6LZ/84xDoyktWWxbfFLGkVM='} 2023-10-17T18:54:08.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2889843c16a3819-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.447 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.447 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.448 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.448 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.449 DEBUG:boto:Data: 2023-10-17T18:54:08.449 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.449 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.450 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.450 DEBUG:boto:Params: {} 2023-10-17T18:54:08.451 DEBUG:boto:Token: None 2023-10-17T18:54:08.451 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.451 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd7eac1898b159ca-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.456 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:08.456 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:08.457 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.457 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:08.457 DEBUG:boto:Data: 2023-10-17T18:54:08.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.459 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.459 DEBUG:boto:Params: {} 2023-10-17T18:54:08.459 DEBUG:boto:Token: None 2023-10-17T18:54:08.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:08.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JR0Fb7QhMhWgoGUVrUrSIdycrFo= 2023-10-17T18:54:08.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JR0Fb7QhMhWgoGUVrUrSIdycrFo='} 2023-10-17T18:54:08.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001208795e0c8e8351-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.468 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.468 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.469 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.469 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.470 DEBUG:boto:Data: 2023-10-17T18:54:08.470 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.470 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.471 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.471 DEBUG:boto:Params: {} 2023-10-17T18:54:08.472 DEBUG:boto:Token: None 2023-10-17T18:54:08.472 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000091dce06630b2c61-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.477 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:08.478 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:08.478 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.478 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:08.479 DEBUG:boto:Data: 2023-10-17T18:54:08.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.480 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.480 DEBUG:boto:Params: {} 2023-10-17T18:54:08.481 DEBUG:boto:Token: None 2023-10-17T18:54:08.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:08.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9Zj5+ejD4qV4VbGDXLE1FNY6Cf8= 2023-10-17T18:54:08.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9Zj5+ejD4qV4VbGDXLE1FNY6Cf8='} 2023-10-17T18:54:08.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d6a71dc790a25a7-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.527 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.528 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.528 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.528 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.529 DEBUG:boto:Data: 2023-10-17T18:54:08.529 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.530 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.531 DEBUG:boto:Params: {} 2023-10-17T18:54:08.531 DEBUG:boto:Token: None 2023-10-17T18:54:08.531 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c9f34c7fe0961e2-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.537 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:08.537 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:08.538 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.538 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:08.539 DEBUG:boto:Data: 2023-10-17T18:54:08.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.540 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.540 DEBUG:boto:Params: {} 2023-10-17T18:54:08.541 DEBUG:boto:Token: None 2023-10-17T18:54:08.541 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:08.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xWEeoqdmTqhrhLO5j1/6ElgNgc4= 2023-10-17T18:54:08.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xWEeoqdmTqhrhLO5j1/6ElgNgc4='} 2023-10-17T18:54:08.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000394f1cd00c9d8234-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.550 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:08.550 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:08.551 DEBUG:boto:Method: HEAD 2023-10-17T18:54:08.551 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:08.552 DEBUG:boto:Data: 2023-10-17T18:54:08.552 DEBUG:boto:Headers: {} 2023-10-17T18:54:08.552 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.553 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.553 DEBUG:boto:Params: {} 2023-10-17T18:54:08.554 DEBUG:boto:Token: None 2023-10-17T18:54:08.554 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/ 2023-10-17T18:54:08.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw= 2023-10-17T18:54:08.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9l6TJC6wd05XA+5crhRqX47OIWw=', 'Content-Length': '0'} 2023-10-17T18:54:08.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057072237ce93ec0d-00652ed850-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.559 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:08.560 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:08.560 DEBUG:boto:Method: PUT 2023-10-17T18:54:08.560 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:08.561 DEBUG:boto:Data: 2023-10-17T18:54:08.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:54:08.562 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:08.562 DEBUG:boto:Port: 8000 2023-10-17T18:54:08.562 DEBUG:boto:Params: {} 2023-10-17T18:54:08.563 DEBUG:boto:Token: None 2023-10-17T18:54:08.563 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:54:08 GMT /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:08.564 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wmlg+Lb18piVNID3/OT4vcfeeiY= 2023-10-17T18:54:08.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:54:08 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wmlg+Lb18piVNID3/OT4vcfeeiY='} 2023-10-17T18:54:08.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000987918f60ac9810c-00652ed850-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:08.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:08.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:54:08.608 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:54:08.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:08.664 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:08.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 2 2023-10-17T18:54:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 2 2023-10-17T18:54:08.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-17T18:54:08.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.728+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 2 2023-10-17T18:54:08.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000001.59.6 2023-10-17T18:54:08.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 2 2023-10-17T18:54:08.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.729+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000001.35.6 2023-10-17T18:54:08.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 2 2023-10-17T18:54:08.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-17T18:54:08.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 2 2023-10-17T18:54:08.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000001.25.6 2023-10-17T18:54:08.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.731+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 2 2023-10-17T18:54:08.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.731+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 2 2023-10-17T18:54:08.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.731+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-17T18:54:08.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.731+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000001.62.6 2023-10-17T18:54:08.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 2 2023-10-17T18:54:08.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-17T18:54:08.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 2 2023-10-17T18:54:08.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position=00000000001.25.6 2023-10-17T18:54:08.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 2 2023-10-17T18:54:08.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-17T18:54:08.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.784+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.784+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.784+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.784+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:08.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:08.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:08.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.785+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:08.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:08.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:08.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.786+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:08.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:08.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:08.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:08.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.787+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:08.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:08.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-17T18:54:08.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2/obj00000000000000000024 2023-10-17T18:54:08.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: entry->timestamp=2023-10-17T18:54:01.626967+0000 2023-10-17T18:54:08.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.70.6 op=0 op_state=1 2023-10-17T18:54:08.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000024[0] 2023-10-17T18:54:08.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.788+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position=00000000001.70.6 2023-10-17T18:54:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:08.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.789+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:08.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:08.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000010 2023-10-17T18:54:08.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:01.876823+0000 2023-10-17T18:54:08.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-17T18:54:08.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:08.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-17T18:54:08.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: parsed entry: id=00000000002.121.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:08.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7/obj00000000000000000001 2023-10-17T18:54:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: entry->timestamp=2023-10-17T18:54:02.030508+0000 2023-10-17T18:54:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.121.6 op=0 op_state=1 2023-10-17T18:54:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000021 2023-10-17T18:54:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:01.703913+0000 2023-10-17T18:54:08.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-10-17T18:54:08.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-17T18:54:08.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000030 2023-10-17T18:54:08.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:01.933414+0000 2023-10-17T18:54:08.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.72.6 op=0 op_state=1 2023-10-17T18:54:08.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:08.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000010 2023-10-17T18:54:08.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000001[0] 2023-10-17T18:54:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position=00000000002.121.6 2023-10-17T18:54:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000021[0] 2023-10-17T18:54:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000001.29.6 2023-10-17T18:54:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000030[0] 2023-10-17T18:54:08.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.790+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000001.72.6 2023-10-17T18:54:08.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-17T18:54:08.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14/obj00000000000000000033 2023-10-17T18:54:08.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: entry->timestamp=2023-10-17T18:54:01.723393+0000 2023-10-17T18:54:08.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.70.6 op=0 op_state=1 2023-10-17T18:54:08.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000033[0] 2023-10-17T18:54:08.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.791+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position=00000000001.70.6 2023-10-17T18:54:08.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.796+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: a sync operation returned error 2023-10-17T18:54:08.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.796+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: backing out with sync_status=-5 2023-10-17T18:54:08.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.796+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:08.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.796+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:08.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.797+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.797+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:08.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.797+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.804+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000024]: success 2023-10-17T18:54:08.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.804+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2:1 marker=00000000001.70.6 timestamp=2023-10-17T18:54:01.626967+0000 2023-10-17T18:54:08.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.805+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000033]: success 2023-10-17T18:54:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.805+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14:1 marker=00000000001.70.6 timestamp=2023-10-17T18:54:01.723393+0000 2023-10-17T18:54:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.805+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000021]: success 2023-10-17T18:54:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.805+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:1 marker=00000000001.29.6 timestamp=2023-10-17T18:54:01.703913+0000 2023-10-17T18:54:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.806+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000030]: success 2023-10-17T18:54:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:1 marker=00000000001.72.6 timestamp=2023-10-17T18:54:01.933414+0000 2023-10-17T18:54:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.806+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000001]: success 2023-10-17T18:54:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7:1 marker=00000000002.121.6 timestamp=2023-10-17T18:54:02.030508+0000 2023-10-17T18:54:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-17T18:54:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000007 2023-10-17T18:54:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:01.951880+0000 2023-10-17T18:54:08.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-10-17T18:54:08.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000007[0] 2023-10-17T18:54:08.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000001.51.6 2023-10-17T18:54:08.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-17T18:54:08.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000020 2023-10-17T18:54:08.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:01.570534+0000 2023-10-17T18:54:08.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.68.6 op=0 op_state=1 2023-10-17T18:54:08.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000001.25.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-17T18:54:08.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000016 2023-10-17T18:54:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:01.783019+0000 2023-10-17T18:54:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.25.6 op=0 op_state=1 2023-10-17T18:54:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: : injecting data sync error on key=obj00000000000000000020 2023-10-17T18:54:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000001.68.6 2023-10-17T18:54:08.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000016[0] 2023-10-17T18:54:08.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000001.25.6 2023-10-17T18:54:08.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:08.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.827+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000020 2023-10-17T18:54:08.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000001.119.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-17T18:54:08.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000028 2023-10-17T18:54:08.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:01.550744+0000 2023-10-17T18:54:08.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.119.6 op=0 op_state=1 2023-10-17T18:54:08.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000028[0] 2023-10-17T18:54:08.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000001.119.6 2023-10-17T18:54:08.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:08.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000000 2023-10-17T18:54:08.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:01.645845+0000 2023-10-17T18:54:08.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-17T18:54:08.837 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-17T18:54:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-17T18:54:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000013 2023-10-17T18:54:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:01.969622+0000 2023-10-17T18:54:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-10-17T18:54:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2023-10-17T18:54:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.34.6 2023-10-17T18:54:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:08.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.830+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000013 2023-10-17T18:54:08.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:08.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000019 2023-10-17T18:54:08.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:01.801900+0000 2023-10-17T18:54:08.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-10-17T18:54:08.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000019[0] 2023-10-17T18:54:08.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-17T18:54:08.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-17T18:54:08.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000032 2023-10-17T18:54:08.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:01.857957+0000 2023-10-17T18:54:08.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-17T18:54:08.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-17T18:54:08.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15/obj00000000000000000004 2023-10-17T18:54:08.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: entry->timestamp=2023-10-17T18:54:02.050690+0000 2023-10-17T18:54:08.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.21.6 op=0 op_state=1 2023-10-17T18:54:08.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: : injecting data sync error on key=obj00000000000000000032 2023-10-17T18:54:08.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-17T18:54:08.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000004[0] 2023-10-17T18:54:08.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position=00000000002.21.6 2023-10-17T18:54:08.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:08.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.832+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000032 2023-10-17T18:54:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-17T18:54:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13/obj00000000000000000026 2023-10-17T18:54:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: entry->timestamp=2023-10-17T18:54:01.531408+0000 2023-10-17T18:54:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-10-17T18:54:08.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000026[0] 2023-10-17T18:54:08.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.834+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position=00000000001.28.6 2023-10-17T18:54:08.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:08.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000010 2023-10-17T18:54:08.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:01.876823+0000 2023-10-17T18:54:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-17T18:54:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-17T18:54:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.840+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000010 2023-10-17T18:54:08.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.842+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000019]: success 2023-10-17T18:54:08.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.842+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:1 marker=00000000001.45.6 timestamp=2023-10-17T18:54:01.801900+0000 2023-10-17T18:54:08.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.842+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000026]: success 2023-10-17T18:54:08.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.842+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13:1 marker=00000000001.28.6 timestamp=2023-10-17T18:54:01.531408+0000 2023-10-17T18:54:08.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: a sync operation returned error 2023-10-17T18:54:08.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: backing out with sync_status=-5 2023-10-17T18:54:08.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:08.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:08.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:08.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.870+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.872+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: a sync operation returned error 2023-10-17T18:54:08.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.872+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: backing out with sync_status=-5 2023-10-17T18:54:08.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.872+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:08.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.872+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:08.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.873+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.873+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:08.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.873+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000007]: success 2023-10-17T18:54:08.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:1 marker=00000000001.51.6 timestamp=2023-10-17T18:54:01.951880+0000 2023-10-17T18:54:08.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: a sync operation returned error 2023-10-17T18:54:08.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: backing out with sync_status=-5 2023-10-17T18:54:08.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:08.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:08.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000016]: success 2023-10-17T18:54:08.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.875+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:1 marker=00000000001.25.6 timestamp=2023-10-17T18:54:01.783019+0000 2023-10-17T18:54:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.876+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.877+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.877+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.877+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000028]: success 2023-10-17T18:54:08.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.877+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:1 marker=00000000001.119.6 timestamp=2023-10-17T18:54:01.550744+0000 2023-10-17T18:54:08.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.878+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000000]: success 2023-10-17T18:54:08.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.878+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:1 marker=00000000002.51.6 timestamp=2023-10-17T18:54:01.645845+0000 2023-10-17T18:54:08.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.881+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000004]: success 2023-10-17T18:54:08.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.881+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15:1 marker=00000000002.21.6 timestamp=2023-10-17T18:54:02.050690+0000 2023-10-17T18:54:08.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.882+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: a sync operation returned error 2023-10-17T18:54:08.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.882+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: backing out with sync_status=-5 2023-10-17T18:54:08.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.882+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:08.884 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.882+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:08.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.884+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.885+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:08.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.885+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-17T18:54:08.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000020 2023-10-17T18:54:08.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:01.570534+0000 2023-10-17T18:54:08.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.68.6 op=0 op_state=1 2023-10-17T18:54:08.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000020[0] 2023-10-17T18:54:08.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.912+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000001.68.6 2023-10-17T18:54:08.916 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-17T18:54:08.916 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000013 2023-10-17T18:54:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:01.969622+0000 2023-10-17T18:54:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-10-17T18:54:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000013[0] 2023-10-17T18:54:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.915+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.34.6 2023-10-17T18:54:08.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.918+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000020]: success 2023-10-17T18:54:08.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.918+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:1 marker=00000000001.68.6 timestamp=2023-10-17T18:54:01.570534+0000 2023-10-17T18:54:08.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-17T18:54:08.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000032 2023-10-17T18:54:08.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:01.857957+0000 2023-10-17T18:54:08.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-17T18:54:08.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000032[0] 2023-10-17T18:54:08.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.921+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-17T18:54:08.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.926+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: success 2023-10-17T18:54:08.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.926+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:1 marker=00000000001.39.6 timestamp=2023-10-17T18:54:01.857957+0000 2023-10-17T18:54:08.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:08.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000010 2023-10-17T18:54:08.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:01.876823+0000 2023-10-17T18:54:08.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-17T18:54:08.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000010[0] 2023-10-17T18:54:08.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.927+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-17T18:54:08.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.959+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000013]: success 2023-10-17T18:54:08.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.959+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:1 marker=00000000002.34.6 timestamp=2023-10-17T18:54:01.969622+0000 2023-10-17T18:54:08.972 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.971+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000010]: success 2023-10-17T18:54:08.973 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.971+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:1 marker=00000000002.42.6 timestamp=2023-10-17T18:54:01.876823+0000 2023-10-17T18:54:08.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:08.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:08.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:08.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.976+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:08.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:08.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.977+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:08.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:08.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.978+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:08.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:08.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-17T18:54:08.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000030 2023-10-17T18:54:08.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:02.943998+0000 2023-10-17T18:54:08.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-10-17T18:54:08.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000030[0] 2023-10-17T18:54:08.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.979+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-17T18:54:08.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-17T18:54:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000025 2023-10-17T18:54:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:02.763003+0000 2023-10-17T18:54:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-10-17T18:54:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000025[0] 2023-10-17T18:54:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-17T18:54:08.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-17T18:54:08.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000004 2023-10-17T18:54:08.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:02.438843+0000 2023-10-17T18:54:08.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-17T18:54:08.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000004[0] 2023-10-17T18:54:08.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-17T18:54:08.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.980+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:08.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:08.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:08.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:08.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:08.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000001.38.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-17T18:54:08.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000009 2023-10-17T18:54:08.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:02.843896+0000 2023-10-17T18:54:08.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.38.6 op=0 op_state=1 2023-10-17T18:54:08.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:08.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000009[0] 2023-10-17T18:54:08.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000001.38.6 2023-10-17T18:54:08.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.981+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket shard: 0 2023-10-17T18:54:09.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket shard: 0 2023-10-17T18:54:09.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.983+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-17T18:54:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000014 2023-10-17T18:54:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:02.626697+0000 2023-10-17T18:54:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-10-17T18:54:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000014[0] 2023-10-17T18:54:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-17T18:54:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000030]: success 2023-10-17T18:54:09.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.984+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:2 marker=00000000002.75.6 timestamp=2023-10-17T18:54:02.943998+0000 2023-10-17T18:54:09.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000004]: success 2023-10-17T18:54:09.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:2 marker=00000000002.47.6 timestamp=2023-10-17T18:54:02.438843+0000 2023-10-17T18:54:09.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-17T18:54:09.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000012 2023-10-17T18:54:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:03.028612+0000 2023-10-17T18:54:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-10-17T18:54:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000012[0] 2023-10-17T18:54:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.985+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000001.63.6 2023-10-17T18:54:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:09.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14/obj00000000000000000000 2023-10-17T18:54:09.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: entry->timestamp=2023-10-17T18:54:02.543755+0000 2023-10-17T18:54:09.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-17T18:54:09.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.986+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-17T18:54:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-17T18:54:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18/obj00000000000000000007 2023-10-17T18:54:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: entry->timestamp=2023-10-17T18:54:02.382468+0000 2023-10-17T18:54:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-10-17T18:54:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000007[0] 2023-10-17T18:54:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position=00000000002.33.6 2023-10-17T18:54:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000003 2023-10-17T18:54:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:02.703713+0000 2023-10-17T18:54:09.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-17T18:54:09.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000003[0] 2023-10-17T18:54:09.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.987+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-17T18:54:09.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-17T18:54:09.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000011 2023-10-17T18:54:09.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:02.783333+0000 2023-10-17T18:54:09.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-17T18:54:09.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000011[0] 2023-10-17T18:54:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-17T18:54:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13/obj00000000000000000002 2023-10-17T18:54:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: entry->timestamp=2023-10-17T18:54:03.007588+0000 2023-10-17T18:54:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-17T18:54:09.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: parsed entry: id=00000000002.127.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:09.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17/obj00000000000000000001 2023-10-17T18:54:09.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: entry->timestamp=2023-10-17T18:54:02.862687+0000 2023-10-17T18:54:09.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.127.6 op=0 op_state=1 2023-10-17T18:54:09.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000002[0] 2023-10-17T18:54:09.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-17T18:54:09.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000001[0] 2023-10-17T18:54:09.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.988+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position=00000000002.127.6 2023-10-17T18:54:09.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.992+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000000]: success 2023-10-17T18:54:09.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.992+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14:2 marker=00000000002.56.6 timestamp=2023-10-17T18:54:02.543755+0000 2023-10-17T18:54:09.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.993+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000007]: success 2023-10-17T18:54:09.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.993+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18:2 marker=00000000002.33.6 timestamp=2023-10-17T18:54:02.382468+0000 2023-10-17T18:54:09.014 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.993+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: success 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.993+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17:2 marker=00000000002.127.6 timestamp=2023-10-17T18:54:02.862687+0000 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.996+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000009]: success 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.996+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:2 marker=00000000001.38.6 timestamp=2023-10-17T18:54:02.843896+0000 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.997+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000002]: success 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.997+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13:2 marker=00000000002.43.6 timestamp=2023-10-17T18:54:03.007588+0000 2023-10-17T18:54:09.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.999+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000011]: success 2023-10-17T18:54:09.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:08.999+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16:2 marker=00000000002.60.6 timestamp=2023-10-17T18:54:02.783333+0000 2023-10-17T18:54:09.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.001+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000003]: success 2023-10-17T18:54:09.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.001+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:2 marker=00000000002.44.6 timestamp=2023-10-17T18:54:02.703713+0000 2023-10-17T18:54:09.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-17T18:54:09.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000005 2023-10-17T18:54:09.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:02.683668+0000 2023-10-17T18:54:09.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-10-17T18:54:09.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000005[0] 2023-10-17T18:54:09.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000002.32.6 2023-10-17T18:54:09.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:09.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000021 2023-10-17T18:54:09.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:02.923431+0000 2023-10-17T18:54:09.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-10-17T18:54:09.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000021[0] 2023-10-17T18:54:09.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.019+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000002.26.6 2023-10-17T18:54:09.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000010 2023-10-17T18:54:09.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:02.603735+0000 2023-10-17T18:54:09.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-17T18:54:09.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000010[0] 2023-10-17T18:54:09.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-17T18:54:09.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-17T18:54:09.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2/obj00000000000000000016 2023-10-17T18:54:09.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: entry->timestamp=2023-10-17T18:54:02.522946+0000 2023-10-17T18:54:09.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-10-17T18:54:09.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000016[0] 2023-10-17T18:54:09.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.020+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position=00000000002.75.6 2023-10-17T18:54:09.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:09.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000019 2023-10-17T18:54:09.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:02.459123+0000 2023-10-17T18:54:09.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-17T18:54:09.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000019[0] 2023-10-17T18:54:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-17T18:54:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000021]: success 2023-10-17T18:54:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.021+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:2 marker=00000000002.26.6 timestamp=2023-10-17T18:54:02.923431+0000 2023-10-17T18:54:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.022+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000016]: success 2023-10-17T18:54:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.022+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2:2 marker=00000000002.75.6 timestamp=2023-10-17T18:54:02.522946+0000 2023-10-17T18:54:09.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.023+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000010]: success 2023-10-17T18:54:09.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.023+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:2 marker=00000000002.79.6 timestamp=2023-10-17T18:54:02.603735+0000 2023-10-17T18:54:09.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.023+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000019]: success 2023-10-17T18:54:09.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.023+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:2 marker=00000000002.58.6 timestamp=2023-10-17T18:54:02.459123+0000 2023-10-17T18:54:09.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-17T18:54:09.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15/obj00000000000000000006 2023-10-17T18:54:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: entry->timestamp=2023-10-17T18:54:02.361794+0000 2023-10-17T18:54:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-17T18:54:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000006[0] 2023-10-17T18:54:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.025+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-17T18:54:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.029+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000025]: success 2023-10-17T18:54:09.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.029+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:2 marker=00000000001.41.6 timestamp=2023-10-17T18:54:02.763003+0000 2023-10-17T18:54:09.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.035+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000014]: success 2023-10-17T18:54:09.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.035+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:2 marker=00000000001.45.6 timestamp=2023-10-17T18:54:02.626697+0000 2023-10-17T18:54:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.037+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000012]: success 2023-10-17T18:54:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.037+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:2 marker=00000000001.63.6 timestamp=2023-10-17T18:54:03.028612+0000 2023-10-17T18:54:09.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.065+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000005]: success 2023-10-17T18:54:09.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.065+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:2 marker=00000000002.32.6 timestamp=2023-10-17T18:54:02.683668+0000 2023-10-17T18:54:09.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.072+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000006]: success 2023-10-17T18:54:09.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.072+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15:2 marker=00000000002.31.6 timestamp=2023-10-17T18:54:02.361794+0000 2023-10-17T18:54:09.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:09.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:09.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.077+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:09.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:09.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:09.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.078+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:09.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:09.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.079+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:09.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.080+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-17T18:54:09.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000032 2023-10-17T18:54:09.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:04.095688+0000 2023-10-17T18:54:09.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-17T18:54:09.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000032[0] 2023-10-17T18:54:09.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-17T18:54:09.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:09.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:09.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:09.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.081+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-17T18:54:09.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000039 2023-10-17T18:54:09.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:03.956453+0000 2023-10-17T18:54:09.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.73.6 op=0 op_state=1 2023-10-17T18:54:09.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7/obj00000000000000000002 2023-10-17T18:54:09.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: entry->timestamp=2023-10-17T18:54:04.175767+0000 2023-10-17T18:54:09.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-17T18:54:09.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000039[0] 2023-10-17T18:54:09.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000001.73.6 2023-10-17T18:54:09.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000002[0] 2023-10-17T18:54:09.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.082+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-17T18:54:09.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.083+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:09.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.083+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:09.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.083+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.083+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.083+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket shard: 0 2023-10-17T18:54:09.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket shard: 0 2023-10-17T18:54:09.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket shard: 0 2023-10-17T18:54:09.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket shard: 0 2023-10-17T18:54:09.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket shard: 0 2023-10-17T18:54:09.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000032]: success 2023-10-17T18:54:09.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:3 marker=00000000002.47.6 timestamp=2023-10-17T18:54:04.095688+0000 2023-10-17T18:54:09.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-17T18:54:09.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000040 2023-10-17T18:54:09.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:03.709681+0000 2023-10-17T18:54:09.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-10-17T18:54:09.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-17T18:54:09.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000009 2023-10-17T18:54:09.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:03.435488+0000 2023-10-17T18:54:09.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-17T18:54:09.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.085+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-17T18:54:09.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15/obj00000000000000000012 2023-10-17T18:54:09.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: entry->timestamp=2023-10-17T18:54:03.542348+0000 2023-10-17T18:54:09.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-17T18:54:09.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000040[0] 2023-10-17T18:54:09.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000001.31.6 2023-10-17T18:54:09.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000009[0] 2023-10-17T18:54:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-17T18:54:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000012[0] 2023-10-17T18:54:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-17T18:54:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000002]: success 2023-10-17T18:54:09.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7:3 marker=00000000002.35.6 timestamp=2023-10-17T18:54:04.175767+0000 2023-10-17T18:54:09.108 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.086+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket shard: 0 2023-10-17T18:54:09.108 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.087+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000009]: success 2023-10-17T18:54:09.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:3 marker=00000000002.31.6 timestamp=2023-10-17T18:54:03.435488+0000 2023-10-17T18:54:09.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000012]: success 2023-10-17T18:54:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15:3 marker=00000000002.37.6 timestamp=2023-10-17T18:54:03.542348+0000 2023-10-17T18:54:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000003 2023-10-17T18:54:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: entry->timestamp=2023-10-17T18:54:03.770811+0000 2023-10-17T18:54:09.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-17T18:54:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000003[0] 2023-10-17T18:54:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-17T18:54:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-17T18:54:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22/obj00000000000000000008 2023-10-17T18:54:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: entry->timestamp=2023-10-17T18:54:03.354825+0000 2023-10-17T18:54:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-10-17T18:54:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000008[0] 2023-10-17T18:54:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.090+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position=00000000002.80.6 2023-10-17T18:54:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.093+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000003]: success 2023-10-17T18:54:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.093+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20:3 marker=00000000002.56.6 timestamp=2023-10-17T18:54:03.770811+0000 2023-10-17T18:54:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.097+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000008]: success 2023-10-17T18:54:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.097+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22:3 marker=00000000002.80.6 timestamp=2023-10-17T18:54:03.354825+0000 2023-10-17T18:54:09.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-17T18:54:09.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2/obj00000000000000000011 2023-10-17T18:54:09.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: entry->timestamp=2023-10-17T18:54:04.117487+0000 2023-10-17T18:54:09.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-17T18:54:09.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000011[0] 2023-10-17T18:54:09.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.120+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-17T18:54:09.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-10-17T18:54:09.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000058 2023-10-17T18:54:09.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:03.456624+0000 2023-10-17T18:54:09.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.35.6 op=0 op_state=1 2023-10-17T18:54:09.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000058[0] 2023-10-17T18:54:09.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000001.35.6 2023-10-17T18:54:09.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-17T18:54:09.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000026 2023-10-17T18:54:09.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:03.520097+0000 2023-10-17T18:54:09.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-17T18:54:09.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-17T18:54:09.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000014 2023-10-17T18:54:09.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:03.873755+0000 2023-10-17T18:54:09.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-17T18:54:09.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000026[0] 2023-10-17T18:54:09.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-17T18:54:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000014[0] 2023-10-17T18:54:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.121+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-17T18:54:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-10-17T18:54:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000046 2023-10-17T18:54:09.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:03.374892+0000 2023-10-17T18:54:09.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.33.6 op=0 op_state=1 2023-10-17T18:54:09.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000046[0] 2023-10-17T18:54:09.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000001.33.6 2023-10-17T18:54:09.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000001.65.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-17T18:54:09.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000050 2023-10-17T18:54:09.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:03.624215+0000 2023-10-17T18:54:09.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.65.6 op=0 op_state=1 2023-10-17T18:54:09.130 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000050[0] 2023-10-17T18:54:09.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000001.65.6 2023-10-17T18:54:09.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000011]: success 2023-10-17T18:54:09.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.122+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2:3 marker=00000000002.39.6 timestamp=2023-10-17T18:54:04.117487+0000 2023-10-17T18:54:09.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-17T18:54:09.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000007 2023-10-17T18:54:09.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:03.793971+0000 2023-10-17T18:54:09.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-17T18:54:09.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-17T18:54:09.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000006 2023-10-17T18:54:09.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.124+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:03.685267+0000 2023-10-17T18:54:09.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-17T18:54:09.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000026]: success 2023-10-17T18:54:09.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:3 marker=00000000002.49.6 timestamp=2023-10-17T18:54:03.520097+0000 2023-10-17T18:54:09.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000007[0] 2023-10-17T18:54:09.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-17T18:54:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000014]: success 2023-10-17T18:54:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:3 marker=00000000002.31.6 timestamp=2023-10-17T18:54:03.873755+0000 2023-10-17T18:54:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000006[0] 2023-10-17T18:54:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-17T18:54:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-17T18:54:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13/obj00000000000000000045 2023-10-17T18:54:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: entry->timestamp=2023-10-17T18:54:04.036288+0000 2023-10-17T18:54:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.33.6 op=0 op_state=1 2023-10-17T18:54:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000045[0] 2023-10-17T18:54:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.125+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position=00000000001.33.6 2023-10-17T18:54:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17/obj00000000000000000001 2023-10-17T18:54:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: entry->timestamp=2023-10-17T18:54:03.935953+0000 2023-10-17T18:54:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-10-17T18:54:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000001[0] 2023-10-17T18:54:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position=00000000002.26.6 2023-10-17T18:54:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18/obj00000000000000000000 2023-10-17T18:54:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: entry->timestamp=2023-10-17T18:54:03.602637+0000 2023-10-17T18:54:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-10-17T18:54:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.128+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position=00000000002.34.6 2023-10-17T18:54:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-10-17T18:54:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21/obj00000000000000000055 2023-10-17T18:54:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: entry->timestamp=2023-10-17T18:54:03.852931+0000 2023-10-17T18:54:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.36.6 op=0 op_state=1 2023-10-17T18:54:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000055[0] 2023-10-17T18:54:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position=00000000001.36.6 2023-10-17T18:54:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-17T18:54:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000033 2023-10-17T18:54:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:04.015962+0000 2023-10-17T18:54:09.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-17T18:54:09.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000033[0] 2023-10-17T18:54:09.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.129+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-17T18:54:09.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.131+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000006]: success 2023-10-17T18:54:09.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.131+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:3 marker=00000000002.68.6 timestamp=2023-10-17T18:54:03.685267+0000 2023-10-17T18:54:09.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.132+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000007]: success 2023-10-17T18:54:09.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.132+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:3 marker=00000000002.52.6 timestamp=2023-10-17T18:54:03.793971+0000 2023-10-17T18:54:09.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000046]: success 2023-10-17T18:54:09.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:3 marker=00000000001.33.6 timestamp=2023-10-17T18:54:03.374892+0000 2023-10-17T18:54:09.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000050]: success 2023-10-17T18:54:09.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:3 marker=00000000001.65.6 timestamp=2023-10-17T18:54:03.624215+0000 2023-10-17T18:54:09.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000039]: success 2023-10-17T18:54:09.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:3 marker=00000000001.73.6 timestamp=2023-10-17T18:54:03.956453+0000 2023-10-17T18:54:09.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000033]: success 2023-10-17T18:54:09.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16:3 marker=00000000002.50.6 timestamp=2023-10-17T18:54:04.015962+0000 2023-10-17T18:54:09.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000001]: success 2023-10-17T18:54:09.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.133+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17:3 marker=00000000002.26.6 timestamp=2023-10-17T18:54:03.935953+0000 2023-10-17T18:54:09.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.134+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000000]: success 2023-10-17T18:54:09.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.134+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18:3 marker=00000000002.34.6 timestamp=2023-10-17T18:54:03.602637+0000 2023-10-17T18:54:09.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.138+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000040]: success 2023-10-17T18:54:09.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.138+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:3 marker=00000000001.31.6 timestamp=2023-10-17T18:54:03.709681+0000 2023-10-17T18:54:09.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.167+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000058]: success 2023-10-17T18:54:09.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.167+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:3 marker=00000000001.35.6 timestamp=2023-10-17T18:54:03.456624+0000 2023-10-17T18:54:09.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.173+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000045]: success 2023-10-17T18:54:09.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.173+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13:3 marker=00000000001.33.6 timestamp=2023-10-17T18:54:04.036288+0000 2023-10-17T18:54:09.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.177+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000055]: success 2023-10-17T18:54:09.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.177+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21:3 marker=00000000001.36.6 timestamp=2023-10-17T18:54:03.852931+0000 2023-10-17T18:54:09.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.198+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.198+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.198+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.198+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:09.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:09.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:09.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:09.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.199+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:09.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:09.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.200+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:09.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:09.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:09.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:09.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.201+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:09.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:09.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.202+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:09.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket shard: 0 2023-10-17T18:54:09.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.204+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket shard: 0 2023-10-17T18:54:09.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:09.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket shard: 0 2023-10-17T18:54:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.205+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket shard: 0 2023-10-17T18:54:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18/obj00000000000000000003 2023-10-17T18:54:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: entry->timestamp=2023-10-17T18:54:05.359579+0000 2023-10-17T18:54:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-17T18:54:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000003[0] 2023-10-17T18:54:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.206+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-17T18:54:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket shard: 0 2023-10-17T18:54:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket shard: 0 2023-10-17T18:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket shard: 0 2023-10-17T18:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket shard: 0 2023-10-17T18:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket shard: 0 2023-10-17T18:54:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket shard: 0 2023-10-17T18:54:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket shard: 0 2023-10-17T18:54:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.207+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.208+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-17T18:54:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000005 2023-10-17T18:54:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:04.614541+0000 2023-10-17T18:54:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-17T18:54:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-10-17T18:54:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-17T18:54:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-10-17T18:54:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19/obj00000000000000000042 2023-10-17T18:54:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: entry->timestamp=2023-10-17T18:54:05.520429+0000 2023-10-17T18:54:09.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-17T18:54:09.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-17T18:54:09.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000008 2023-10-17T18:54:09.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:05.339564+0000 2023-10-17T18:54:09.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-10-17T18:54:09.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000010 2023-10-17T18:54:09.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: entry->timestamp=2023-10-17T18:54:04.695970+0000 2023-10-17T18:54:09.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-17T18:54:09.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000005 2023-10-17T18:54:09.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000042[0] 2023-10-17T18:54:09.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-17T18:54:09.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000008[0] 2023-10-17T18:54:09.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000002.85.6 2023-10-17T18:54:09.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:09.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-17T18:54:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.209+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000010 2023-10-17T18:54:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-17T18:54:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27/obj00000000000000000030 2023-10-17T18:54:09.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: entry->timestamp=2023-10-17T18:54:05.441713+0000 2023-10-17T18:54:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-10-17T18:54:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000030[0] 2023-10-17T18:54:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.211+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position=00000000002.78.6 2023-10-17T18:54:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-17T18:54:09.234 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000014 2023-10-17T18:54:09.234 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:05.280347+0000 2023-10-17T18:54:09.234 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-10-17T18:54:09.235 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: : injecting data sync error on key=obj00000000000000000014 2023-10-17T18:54:09.235 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.80.6 2023-10-17T18:54:09.235 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: a sync operation returned error 2023-10-17T18:54:09.235 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000003]: success 2023-10-17T18:54:09.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18:4 marker=00000000002.45.6 timestamp=2023-10-17T18:54:05.359579+0000 2023-10-17T18:54:09.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000014 2023-10-17T18:54:09.237 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.212+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: backing out with sync_status=-5 2023-10-17T18:54:09.237 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.237 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-17T18:54:09.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22/obj00000000000000000009 2023-10-17T18:54:09.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: entry->timestamp=2023-10-17T18:54:05.096582+0000 2023-10-17T18:54:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-17T18:54:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000009[0] 2023-10-17T18:54:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-17T18:54:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: a sync operation returned error 2023-10-17T18:54:09.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: backing out with sync_status=-5 2023-10-17T18:54:09.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.213+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.214+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.214+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000030]: success 2023-10-17T18:54:09.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.214+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27:4 marker=00000000002.78.6 timestamp=2023-10-17T18:54:05.441713+0000 2023-10-17T18:54:09.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket shard: 0 2023-10-17T18:54:09.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000008]: success 2023-10-17T18:54:09.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:4 marker=00000000002.85.6 timestamp=2023-10-17T18:54:05.339564+0000 2023-10-17T18:54:09.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.215+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.216+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000009]: success 2023-10-17T18:54:09.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.216+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22:4 marker=00000000002.54.6 timestamp=2023-10-17T18:54:05.096582+0000 2023-10-17T18:54:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.218+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000042]: success 2023-10-17T18:54:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.219+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19:4 marker=00000000001.43.6 timestamp=2023-10-17T18:54:05.520429+0000 2023-10-17T18:54:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-17T18:54:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2/obj00000000000000000027 2023-10-17T18:54:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: entry->timestamp=2023-10-17T18:54:04.531609+0000 2023-10-17T18:54:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-17T18:54:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000027[0] 2023-10-17T18:54:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-17T18:54:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-17T18:54:09.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000006 2023-10-17T18:54:09.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:04.775738+0000 2023-10-17T18:54:09.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-17T18:54:09.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-10-17T18:54:09.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000059 2023-10-17T18:54:09.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:04.856098+0000 2023-10-17T18:54:09.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.35.6 op=0 op_state=1 2023-10-17T18:54:09.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-17T18:54:09.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000011 2023-10-17T18:54:09.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:04.935527+0000 2023-10-17T18:54:09.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-17T18:54:09.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000006[0] 2023-10-17T18:54:09.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-17T18:54:09.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000059[0] 2023-10-17T18:54:09.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000001.35.6 2023-10-17T18:54:09.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000011[0] 2023-10-17T18:54:09.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.241+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-17T18:54:09.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-17T18:54:09.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000004 2023-10-17T18:54:09.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:05.420149+0000 2023-10-17T18:54:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-17T18:54:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000004[0] 2023-10-17T18:54:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.242+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-17T18:54:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000019 2023-10-17T18:54:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:04.795928+0000 2023-10-17T18:54:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-17T18:54:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-10-17T18:54:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-17T18:54:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-10-17T18:54:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000031 2023-10-17T18:54:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:05.037045+0000 2023-10-17T18:54:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-10-17T18:54:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-10-17T18:54:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7/obj00000000000000000049 2023-10-17T18:54:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: entry->timestamp=2023-10-17T18:54:04.553443+0000 2023-10-17T18:54:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-10-17T18:54:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000019 2023-10-17T18:54:09.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000031[0] 2023-10-17T18:54:09.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000001.34.6 2023-10-17T18:54:09.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000049[0] 2023-10-17T18:54:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.244+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position=00000000001.83.6 2023-10-17T18:54:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000006]: success 2023-10-17T18:54:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:4 marker=00000000002.40.6 timestamp=2023-10-17T18:54:04.775738+0000 2023-10-17T18:54:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000011]: success 2023-10-17T18:54:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:4 marker=00000000002.52.6 timestamp=2023-10-17T18:54:04.935527+0000 2023-10-17T18:54:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000004]: success 2023-10-17T18:54:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:4 marker=00000000002.44.6 timestamp=2023-10-17T18:54:05.420149+0000 2023-10-17T18:54:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000002 2023-10-17T18:54:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:05.180811+0000 2023-10-17T18:54:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-17T18:54:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-17T18:54:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000015 2023-10-17T18:54:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:05.117542+0000 2023-10-17T18:54:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-17T18:54:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-17T18:54:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000017 2023-10-17T18:54:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:05.499828+0000 2023-10-17T18:54:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-10-17T18:54:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-17T18:54:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000022 2023-10-17T18:54:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.245+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:04.876037+0000 2023-10-17T18:54:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-17T18:54:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-17T18:54:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14/obj00000000000000000020 2023-10-17T18:54:09.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: entry->timestamp=2023-10-17T18:54:04.472045+0000 2023-10-17T18:54:09.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-17T18:54:09.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000002[0] 2023-10-17T18:54:09.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-17T18:54:09.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: : injecting data sync error on key=obj00000000000000000015 2023-10-17T18:54:09.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-17T18:54:09.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000017[0] 2023-10-17T18:54:09.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000001.47.6 2023-10-17T18:54:09.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000022[0] 2023-10-17T18:54:09.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-17T18:54:09.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000020[0] 2023-10-17T18:54:09.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-17T18:54:09.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.246+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000015 2023-10-17T18:54:09.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-17T18:54:09.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17/obj00000000000000000012 2023-10-17T18:54:09.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: entry->timestamp=2023-10-17T18:54:05.015451+0000 2023-10-17T18:54:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-17T18:54:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000012[0] 2023-10-17T18:54:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.247+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-17T18:54:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-17T18:54:09.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13/obj00000000000000000013 2023-10-17T18:54:09.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: entry->timestamp=2023-10-17T18:54:04.635327+0000 2023-10-17T18:54:09.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-17T18:54:09.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000013[0] 2023-10-17T18:54:09.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.248+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-17T18:54:09.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.250+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000002]: success 2023-10-17T18:54:09.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.250+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:4 marker=00000000002.56.6 timestamp=2023-10-17T18:54:05.180811+0000 2023-10-17T18:54:09.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-17T18:54:09.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24/obj00000000000000000016 2023-10-17T18:54:09.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: entry->timestamp=2023-10-17T18:54:05.201557+0000 2023-10-17T18:54:09.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-17T18:54:09.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000016[0] 2023-10-17T18:54:09.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-17T18:54:09.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:09.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25/obj00000000000000000000 2023-10-17T18:54:09.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: entry->timestamp=2023-10-17T18:54:05.259394+0000 2023-10-17T18:54:09.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-17T18:54:09.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:09.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.251+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-17T18:54:09.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:09.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28/obj00000000000000000021 2023-10-17T18:54:09.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: entry->timestamp=2023-10-17T18:54:04.956425+0000 2023-10-17T18:54:09.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-17T18:54:09.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000021[0] 2023-10-17T18:54:09.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.252+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-17T18:54:09.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.253+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000012]: success 2023-10-17T18:54:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.253+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17:4 marker=00000000002.57.6 timestamp=2023-10-17T18:54:05.015451+0000 2023-10-17T18:54:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.253+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000020]: success 2023-10-17T18:54:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.253+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14:4 marker=00000000002.38.6 timestamp=2023-10-17T18:54:04.472045+0000 2023-10-17T18:54:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-17T18:54:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23/obj00000000000000000025 2023-10-17T18:54:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: entry->timestamp=2023-10-17T18:54:04.715529+0000 2023-10-17T18:54:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-10-17T18:54:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000025[0] 2023-10-17T18:54:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position=00000000002.73.6 2023-10-17T18:54:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000000]: success 2023-10-17T18:54:09.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.254+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25:4 marker=00000000002.43.6 timestamp=2023-10-17T18:54:05.259394+0000 2023-10-17T18:54:09.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: a sync operation returned error 2023-10-17T18:54:09.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000016]: success 2023-10-17T18:54:09.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24:4 marker=00000000002.52.6 timestamp=2023-10-17T18:54:05.201557+0000 2023-10-17T18:54:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: backing out with sync_status=-5 2023-10-17T18:54:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.255+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000022]: success 2023-10-17T18:54:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:4 marker=00000000002.56.6 timestamp=2023-10-17T18:54:04.876037+0000 2023-10-17T18:54:09.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000010 2023-10-17T18:54:09.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: entry->timestamp=2023-10-17T18:54:04.695970+0000 2023-10-17T18:54:09.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-17T18:54:09.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000010[0] 2023-10-17T18:54:09.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-17T18:54:09.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000025]: success 2023-10-17T18:54:09.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.257+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23:4 marker=00000000002.73.6 timestamp=2023-10-17T18:54:04.715529+0000 2023-10-17T18:54:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.258+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000013]: success 2023-10-17T18:54:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.258+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13:4 marker=00000000002.42.6 timestamp=2023-10-17T18:54:04.635327+0000 2023-10-17T18:54:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.258+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000021]: success 2023-10-17T18:54:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.258+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28:4 marker=00000000002.41.6 timestamp=2023-10-17T18:54:04.956425+0000 2023-10-17T18:54:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-17T18:54:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000005 2023-10-17T18:54:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:04.614541+0000 2023-10-17T18:54:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-17T18:54:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000005[0] 2023-10-17T18:54:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.259+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-17T18:54:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.261+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.261+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000010]: success 2023-10-17T18:54:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.261+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20:4 marker=00000000002.30.6 timestamp=2023-10-17T18:54:04.695970+0000 2023-10-17T18:54:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.261+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.261+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.263+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000005]: success 2023-10-17T18:54:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.263+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16:4 marker=00000000002.61.6 timestamp=2023-10-17T18:54:04.614541+0000 2023-10-17T18:54:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.263+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000017]: success 2023-10-17T18:54:09.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.263+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:4 marker=00000000001.47.6 timestamp=2023-10-17T18:54:05.499828+0000 2023-10-17T18:54:09.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: a sync operation returned error 2023-10-17T18:54:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: backing out with sync_status=-5 2023-10-17T18:54:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000059]: success 2023-10-17T18:54:09.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.288+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:4 marker=00000000001.35.6 timestamp=2023-10-17T18:54:04.856098+0000 2023-10-17T18:54:09.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: a sync operation returned error 2023-10-17T18:54:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: backing out with sync_status=-5 2023-10-17T18:54:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000027]: success 2023-10-17T18:54:09.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.290+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2:4 marker=00000000001.49.6 timestamp=2023-10-17T18:54:04.531609+0000 2023-10-17T18:54:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.291+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.291+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.291+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.292+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.292+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.293+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.294+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000031]: success 2023-10-17T18:54:09.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.294+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:4 marker=00000000001.34.6 timestamp=2023-10-17T18:54:05.037045+0000 2023-10-17T18:54:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.298+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000049]: success 2023-10-17T18:54:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.298+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7:4 marker=00000000001.83.6 timestamp=2023-10-17T18:54:04.553443+0000 2023-10-17T18:54:09.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-17T18:54:09.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000014 2023-10-17T18:54:09.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:05.280347+0000 2023-10-17T18:54:09.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-10-17T18:54:09.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000014[0] 2023-10-17T18:54:09.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.304+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.80.6 2023-10-17T18:54:09.307 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.306+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000014]: success 2023-10-17T18:54:09.307 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.306+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26:4 marker=00000000002.80.6 timestamp=2023-10-17T18:54:05.280347+0000 2023-10-17T18:54:09.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:09.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000019 2023-10-17T18:54:09.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:04.795928+0000 2023-10-17T18:54:09.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-17T18:54:09.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000019[0] 2023-10-17T18:54:09.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-17T18:54:09.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-17T18:54:09.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000015 2023-10-17T18:54:09.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:05.117542+0000 2023-10-17T18:54:09.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-17T18:54:09.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000015[0] 2023-10-17T18:54:09.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.335+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-17T18:54:09.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.337+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000019]: success 2023-10-17T18:54:09.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.337+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:4 marker=00000000002.54.6 timestamp=2023-10-17T18:54:04.795928+0000 2023-10-17T18:54:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.341+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000015]: success 2023-10-17T18:54:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.341+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:4 marker=00000000002.36.6 timestamp=2023-10-17T18:54:05.117542+0000 2023-10-17T18:54:09.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.384+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.384+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:09.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:09.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.385+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.386+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:09.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.386+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.386+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.386+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.386+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-10-17T18:54:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2/obj00000000000000000467 2023-10-17T18:54:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: entry->timestamp=2023-10-17T18:54:06.940754+0000 2023-10-17T18:54:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.81.6 op=0 op_state=1 2023-10-17T18:54:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000467[0] 2023-10-17T18:54:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position=00000000001.81.6 2023-10-17T18:54:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.387+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000002.140.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-17T18:54:09.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000004 2023-10-17T18:54:09.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:06.158874+0000 2023-10-17T18:54:09.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.140.6 op=0 op_state=1 2023-10-17T18:54:09.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000004[0] 2023-10-17T18:54:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000002.140.6 2023-10-17T18:54:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-17T18:54:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000005 2023-10-17T18:54:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:06.399451+0000 2023-10-17T18:54:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-10-17T18:54:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000005[0] 2023-10-17T18:54:09.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.388+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000002.88.6 2023-10-17T18:54:09.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:09.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000021 2023-10-17T18:54:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:05.975845+0000 2023-10-17T18:54:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.138.6 op=0 op_state=1 2023-10-17T18:54:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2023-10-17T18:54:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.138.6 2023-10-17T18:54:09.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000021 2023-10-17T18:54:09.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:09.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:09.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.391+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000004]: success 2023-10-17T18:54:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:5 marker=00000000002.140.6 timestamp=2023-10-17T18:54:06.158874+0000 2023-10-17T18:54:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:09.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.392+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-17T18:54:09.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000022 2023-10-17T18:54:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:06.079533+0000 2023-10-17T18:54:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-17T18:54:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000022[0] 2023-10-17T18:54:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-17T18:54:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000005]: success 2023-10-17T18:54:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:5 marker=00000000002.88.6 timestamp=2023-10-17T18:54:06.399451+0000 2023-10-17T18:54:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket shard: 0 2023-10-17T18:54:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket shard: 0 2023-10-17T18:54:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.393+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket shard: 0 2023-10-17T18:54:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket shard: 0 2023-10-17T18:54:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket shard: 0 2023-10-17T18:54:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket shard: 0 2023-10-17T18:54:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket shard: 0 2023-10-17T18:54:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: sync status for source bucket shard: 0 2023-10-17T18:54:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket shard: 0 2023-10-17T18:54:09.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.394+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: sync status for source bucket shard: 0 2023-10-17T18:54:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket shard: 0 2023-10-17T18:54:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket shard: 0 2023-10-17T18:54:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.395+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:09.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket shard: 0 2023-10-17T18:54:09.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.396+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-17T18:54:09.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14/obj00000000000000000018 2023-10-17T18:54:09.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: entry->timestamp=2023-10-17T18:54:06.799518+0000 2023-10-17T18:54:09.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-17T18:54:09.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-17T18:54:09.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000009 2023-10-17T18:54:09.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:05.996848+0000 2023-10-17T18:54:09.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-17T18:54:09.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000018[0] 2023-10-17T18:54:09.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-17T18:54:09.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000009[0] 2023-10-17T18:54:09.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.398+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-17T18:54:09.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: parsed entry: id=00000000002.89.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-17T18:54:09.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13/obj00000000000000000008 2023-10-17T18:54:09.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: entry->timestamp=2023-10-17T18:54:06.240044+0000 2023-10-17T18:54:09.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.89.6 op=0 op_state=1 2023-10-17T18:54:09.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19/obj00000000000000000003 2023-10-17T18:54:09.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: entry->timestamp=2023-10-17T18:54:06.059001+0000 2023-10-17T18:54:09.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-17T18:54:09.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:09.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23/obj00000000000000000001 2023-10-17T18:54:09.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: entry->timestamp=2023-10-17T18:54:06.379135+0000 2023-10-17T18:54:09.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-17T18:54:09.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-17T18:54:09.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18/obj00000000000000000016 2023-10-17T18:54:09.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: entry->timestamp=2023-10-17T18:54:06.721762+0000 2023-10-17T18:54:09.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-17T18:54:09.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-17T18:54:09.445 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24/obj00000000000000000011 2023-10-17T18:54:09.445 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: entry->timestamp=2023-10-17T18:54:06.299380+0000 2023-10-17T18:54:09.445 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-17T18:54:09.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-17T18:54:09.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28/obj00000000000000000050 2023-10-17T18:54:09.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: entry->timestamp=2023-10-17T18:54:06.700149+0000 2023-10-17T18:54:09.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-17T18:54:09.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: parsed entry: id=00000000001.87.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-10-17T18:54:09.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30/obj00000000000000000133 2023-10-17T18:54:09.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: entry->timestamp=2023-10-17T18:54:05.912310+0000 2023-10-17T18:54:09.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.87.6 op=0 op_state=1 2023-10-17T18:54:09.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:06.481446+0000 2023-10-17T18:54:09.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-17T18:54:09.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-10-17T18:54:09.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29/obj00000000000000000029 2023-10-17T18:54:09.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: entry->timestamp=2023-10-17T18:54:06.320451+0000 2023-10-17T18:54:09.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-10-17T18:54:09.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-10-17T18:54:09.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25/obj00000000000000000283 2023-10-17T18:54:09.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: entry->timestamp=2023-10-17T18:54:06.859629+0000 2023-10-17T18:54:09.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-10-17T18:54:09.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21/obj00000000000000000002 2023-10-17T18:54:09.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: entry->timestamp=2023-10-17T18:54:06.219574+0000 2023-10-17T18:54:09.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-17T18:54:09.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-17T18:54:09.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15/obj00000000000000000007 2023-10-17T18:54:09.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: entry->timestamp=2023-10-17T18:54:06.138458+0000 2023-10-17T18:54:09.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-17T18:54:09.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-17T18:54:09.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27/obj00000000000000000061 2023-10-17T18:54:09.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: entry->timestamp=2023-10-17T18:54:06.879743+0000 2023-10-17T18:54:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-10-17T18:54:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:entry[obj00000000000000000467]: success 2023-10-17T18:54:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2:5 marker=00000000001.81.6 timestamp=2023-10-17T18:54:06.940754+0000 2023-10-17T18:54:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-17T18:54:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000423 2023-10-17T18:54:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: entry->timestamp=2023-10-17T18:54:06.562585+0000 2023-10-17T18:54:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-10-17T18:54:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000008[0] 2023-10-17T18:54:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position=00000000002.89.6 2023-10-17T18:54:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-17T18:54:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000017 2023-10-17T18:54:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:06.460048+0000 2023-10-17T18:54:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-17T18:54:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-10-17T18:54:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-17T18:54:09.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-10-17T18:54:09.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-17T18:54:09.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000022]: success 2023-10-17T18:54:09.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:5 marker=00000000002.53.6 timestamp=2023-10-17T18:54:06.079533+0000 2023-10-17T18:54:09.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000016[0] 2023-10-17T18:54:09.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-17T18:54:09.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2023-10-17T18:54:09.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-17T18:54:09.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000050[0] 2023-10-17T18:54:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-17T18:54:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000133[0] 2023-10-17T18:54:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: listing bilog for incremental sync; position=00000000001.87.6 2023-10-17T18:54:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000009]: success 2023-10-17T18:54:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:5 marker=00000000002.79.6 timestamp=2023-10-17T18:54:05.996848+0000 2023-10-17T18:54:09.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:09.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-17T18:54:09.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000029[0] 2023-10-17T18:54:09.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: listing bilog for incremental sync; position=00000000001.32.6 2023-10-17T18:54:09.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000283[0] 2023-10-17T18:54:09.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position=00000000001.57.6 2023-10-17T18:54:09.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-10-17T18:54:09.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-17T18:54:09.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000007[0] 2023-10-17T18:54:09.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-17T18:54:09.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: : injecting data sync error on key=obj00000000000000000061 2023-10-17T18:54:09.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-17T18:54:09.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000423[0] 2023-10-17T18:54:09.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-17T18:54:09.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-10-17T18:54:09.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-17T18:54:09.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19/obj00000000000000000003 2023-10-17T18:54:09.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23/obj00000000000000000001 2023-10-17T18:54:09.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24/obj00000000000000000011 2023-10-17T18:54:09.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21/obj00000000000000000002 2023-10-17T18:54:09.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27/obj00000000000000000061 2023-10-17T18:54:09.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.402+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000017 2023-10-17T18:54:09.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.410+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:entry[obj00000000000000000008]: success 2023-10-17T18:54:09.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.410+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13:5 marker=00000000002.89.6 timestamp=2023-10-17T18:54:06.240044+0000 2023-10-17T18:54:09.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.411+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000050]: success 2023-10-17T18:54:09.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.411+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28:5 marker=00000000002.57.6 timestamp=2023-10-17T18:54:06.700149+0000 2023-10-17T18:54:09.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.411+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000007]: success 2023-10-17T18:54:09.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.411+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15:5 marker=00000000002.41.6 timestamp=2023-10-17T18:54:06.138458+0000 2023-10-17T18:54:09.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.413+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000016]: success 2023-10-17T18:54:09.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.413+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18:5 marker=00000000002.51.6 timestamp=2023-10-17T18:54:06.721762+0000 2023-10-17T18:54:09.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.416+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000133]: success 2023-10-17T18:54:09.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.416+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30:5 marker=00000000001.87.6 timestamp=2023-10-17T18:54:05.912310+0000 2023-10-17T18:54:09.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.418+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000283]: success 2023-10-17T18:54:09.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.418+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25:5 marker=00000000001.57.6 timestamp=2023-10-17T18:54:06.859629+0000 2023-10-17T18:54:09.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.419+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000423]: success 2023-10-17T18:54:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.419+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20:5 marker=00000000001.42.6 timestamp=2023-10-17T18:54:06.562585+0000 2023-10-17T18:54:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000019 2023-10-17T18:54:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:06.641794+0000 2023-10-17T18:54:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-17T18:54:09.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000019[0] 2023-10-17T18:54:09.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.428+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-17T18:54:09.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-10-17T18:54:09.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000056 2023-10-17T18:54:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:06.779567+0000 2023-10-17T18:54:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-10-17T18:54:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000056[0] 2023-10-17T18:54:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-17T18:54:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000002.136.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-17T18:54:09.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000006 2023-10-17T18:54:09.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:05.892356+0000 2023-10-17T18:54:09.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.136.6 op=0 op_state=1 2023-10-17T18:54:09.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000006[0] 2023-10-17T18:54:09.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.429+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000002.136.6 2023-10-17T18:54:09.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.430+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000019]: success 2023-10-17T18:54:09.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.430+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:5 marker=00000000002.55.6 timestamp=2023-10-17T18:54:06.641794+0000 2023-10-17T18:54:09.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-10-17T18:54:09.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000023 2023-10-17T18:54:09.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:05.872917+0000 2023-10-17T18:54:09.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-17T18:54:09.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000023[0] 2023-10-17T18:54:09.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-17T18:54:09.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000006]: success 2023-10-17T18:54:09.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.431+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:5 marker=00000000002.136.6 timestamp=2023-10-17T18:54:05.892356+0000 2023-10-17T18:54:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:09.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000000 2023-10-17T18:54:09.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:06.620553+0000 2023-10-17T18:54:09.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-17T18:54:09.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:09.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.437+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-17T18:54:09.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.438+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000000]: success 2023-10-17T18:54:09.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.438+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:5 marker=00000000002.59.6 timestamp=2023-10-17T18:54:06.620553+0000 2023-10-17T18:54:09.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-10-17T18:54:09.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000271 2023-10-17T18:54:09.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:06.541152+0000 2023-10-17T18:54:09.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-17T18:54:09.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000271[0] 2023-10-17T18:54:09.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.439+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-17T18:54:09.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: a sync operation returned error 2023-10-17T18:54:09.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: backing out with sync_status=-5 2023-10-17T18:54:09.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.440+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: a sync operation returned error 2023-10-17T18:54:09.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: backing out with sync_status=-5 2023-10-17T18:54:09.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.445+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000018]: success 2023-10-17T18:54:09.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.446+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14:5 marker=00000000002.62.6 timestamp=2023-10-17T18:54:06.799518+0000 2023-10-17T18:54:09.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.446+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket shard: 0 2023-10-17T18:54:09.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.446+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: a sync operation returned error 2023-10-17T18:54:09.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: backing out with sync_status=-5 2023-10-17T18:54:09.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket shard: 0 2023-10-17T18:54:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.449+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.450+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: a sync operation returned error 2023-10-17T18:54:09.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.450+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: backing out with sync_status=-5 2023-10-17T18:54:09.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.450+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.450+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.450+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: a sync operation returned error 2023-10-17T18:54:09.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: a sync operation returned error 2023-10-17T18:54:09.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: a sync operation returned error 2023-10-17T18:54:09.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: backing out with sync_status=-5 2023-10-17T18:54:09.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: backing out with sync_status=-5 2023-10-17T18:54:09.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: backing out with sync_status=-5 2023-10-17T18:54:09.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.451+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: a sync operation returned error 2023-10-17T18:54:09.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000029]: success 2023-10-17T18:54:09.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29:5 marker=00000000001.32.6 timestamp=2023-10-17T18:54:06.320451+0000 2023-10-17T18:54:09.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: backing out with sync_status=-5 2023-10-17T18:54:09.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.453+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.454+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.454+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.454+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket shard: 0 2023-10-17T18:54:09.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.454+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket shard: 0 2023-10-17T18:54:09.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket shard: 0 2023-10-17T18:54:09.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.455+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.457+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.457+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.474+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000056]: success 2023-10-17T18:54:09.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.474+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:5 marker=00000000001.56.6 timestamp=2023-10-17T18:54:06.779567+0000 2023-10-17T18:54:09.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.477+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000023]: success 2023-10-17T18:54:09.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.477+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:5 marker=00000000002.59.6 timestamp=2023-10-17T18:54:05.872917+0000 2023-10-17T18:54:09.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.138.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:09.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000021 2023-10-17T18:54:09.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:05.975845+0000 2023-10-17T18:54:09.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.138.6 op=0 op_state=1 2023-10-17T18:54:09.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000021[0] 2023-10-17T18:54:09.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.482+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.138.6 2023-10-17T18:54:09.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.483+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000021]: success 2023-10-17T18:54:09.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.483+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:5 marker=00000000002.138.6 timestamp=2023-10-17T18:54:05.975845+0000 2023-10-17T18:54:09.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.484+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000271]: success 2023-10-17T18:54:09.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.484+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:5 marker=00000000001.53.6 timestamp=2023-10-17T18:54:06.541152+0000 2023-10-17T18:54:09.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19/obj00000000000000000003 2023-10-17T18:54:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: entry->timestamp=2023-10-17T18:54:06.059001+0000 2023-10-17T18:54:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-17T18:54:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000003[0] 2023-10-17T18:54:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.489+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-17T18:54:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.490+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000003]: success 2023-10-17T18:54:09.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.490+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19:5 marker=00000000002.51.6 timestamp=2023-10-17T18:54:06.059001+0000 2023-10-17T18:54:09.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:09.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23/obj00000000000000000001 2023-10-17T18:54:09.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: entry->timestamp=2023-10-17T18:54:06.379135+0000 2023-10-17T18:54:09.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-17T18:54:09.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000001[0] 2023-10-17T18:54:09.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.492+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-17T18:54:09.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.493+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000001]: success 2023-10-17T18:54:09.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.493+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23:5 marker=00000000002.37.6 timestamp=2023-10-17T18:54:06.379135+0000 2023-10-17T18:54:09.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-17T18:54:09.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000017 2023-10-17T18:54:09.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:06.460048+0000 2023-10-17T18:54:09.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-17T18:54:09.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000017[0] 2023-10-17T18:54:09.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.494+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-17T18:54:09.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.495+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000017]: success 2023-10-17T18:54:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.495+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16:5 marker=00000000002.39.6 timestamp=2023-10-17T18:54:06.460048+0000 2023-10-17T18:54:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21/obj00000000000000000002 2023-10-17T18:54:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: entry->timestamp=2023-10-17T18:54:06.219574+0000 2023-10-17T18:54:09.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-17T18:54:09.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000002[0] 2023-10-17T18:54:09.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-17T18:54:09.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-17T18:54:09.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27/obj00000000000000000061 2023-10-17T18:54:09.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: entry->timestamp=2023-10-17T18:54:06.879743+0000 2023-10-17T18:54:09.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-10-17T18:54:09.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000061[0] 2023-10-17T18:54:09.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.497+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-17T18:54:09.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-17T18:54:09.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24/obj00000000000000000011 2023-10-17T18:54:09.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: entry->timestamp=2023-10-17T18:54:06.299380+0000 2023-10-17T18:54:09.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-17T18:54:09.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000011[0] 2023-10-17T18:54:09.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.498+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-17T18:54:09.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000002]: success 2023-10-17T18:54:09.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21:5 marker=00000000002.54.6 timestamp=2023-10-17T18:54:06.219574+0000 2023-10-17T18:54:09.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:06.481446+0000 2023-10-17T18:54:09.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-17T18:54:09.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:09.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-17T18:54:09.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.499+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.500+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000011]: success 2023-10-17T18:54:09.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.500+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24:5 marker=00000000002.61.6 timestamp=2023-10-17T18:54:06.299380+0000 2023-10-17T18:54:09.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.542+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: a sync operation returned error 2023-10-17T18:54:09.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.542+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: backing out with sync_status=-5 2023-10-17T18:54:09.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.542+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.542+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.542+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.543+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:entry[obj00000000000000000061]: success 2023-10-17T18:54:09.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.543+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27:5 marker=00000000001.40.6 timestamp=2023-10-17T18:54:06.879743+0000 2023-10-17T18:54:09.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.543+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.543+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:06.481446+0000 2023-10-17T18:54:09.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-17T18:54:09.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-17T18:54:09.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-17T18:54:09.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.585+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.627+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: a sync operation returned error 2023-10-17T18:54:09.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.627+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: backing out with sync_status=-5 2023-10-17T18:54:09.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.627+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.627+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.627+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.628+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.628+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000010 2023-10-17T18:54:09.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:06.481446+0000 2023-10-17T18:54:09.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-17T18:54:09.672 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000010[0] 2023-10-17T18:54:09.672 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.669+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-17T18:54:09.673 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.671+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000010]: success 2023-10-17T18:54:09.673 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.671+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26:5 marker=00000000002.40.6 timestamp=2023-10-17T18:54:06.481446+0000 2023-10-17T18:54:09.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.715+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.715+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.715+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.715+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.715+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: sync status for source bucket shard: 0 2023-10-17T18:54:09.720 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.720 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: sync status for source bucket shard: 0 2023-10-17T18:54:09.720 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: sync status for source bucket shard: 0 2023-10-17T18:54:09.720 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:2]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: sync status for source bucket shard: 0 2023-10-17T18:54:09.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: sync status for source bucket shard: 0 2023-10-17T18:54:09.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.716+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: sync status for source bucket shard: 0 2023-10-17T18:54:09.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.717+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: sync status for source bucket shard: 0 2023-10-17T18:54:09.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:09.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: sync status for source bucket shard: 0 2023-10-17T18:54:09.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-10-17T18:54:09.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0/obj00000000000000000060 2023-10-17T18:54:09.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: entry->timestamp=2023-10-17T18:54:07.735719+0000 2023-10-17T18:54:09.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.48.6 op=0 op_state=1 2023-10-17T18:54:09.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000060[0] 2023-10-17T18:54:09.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: listing bilog for incremental sync; position=00000000001.48.6 2023-10-17T18:54:09.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: sync status for source bucket shard: 0 2023-10-17T18:54:09.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.718+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: sync status for source bucket shard: 0 2023-10-17T18:54:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: sync status for source bucket shard: 0 2023-10-17T18:54:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: sync status for source bucket shard: 0 2023-10-17T18:54:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-10-17T18:54:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5/obj00000000000000000065 2023-10-17T18:54:09.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: entry->timestamp=2023-10-17T18:54:07.915772+0000 2023-10-17T18:54:09.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-10-17T18:54:09.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000065[0] 2023-10-17T18:54:09.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.719+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: listing bilog for incremental sync; position=00000000001.71.6 2023-10-17T18:54:09.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: sync status for source bucket shard: 0 2023-10-17T18:54:09.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: sync status for source bucket shard: 0 2023-10-17T18:54:09.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:13]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-17T18:54:09.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3/obj00000000000000000001 2023-10-17T18:54:09.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: entry->timestamp=2023-10-17T18:54:08.323466+0000 2023-10-17T18:54:09.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.720+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.100.6 op=0 op_state=1 2023-10-17T18:54:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000001[0] 2023-10-17T18:54:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: listing bilog for incremental sync; position=00000000002.100.6 2023-10-17T18:54:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: sync status for source bucket shard: 0 2023-10-17T18:54:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.721+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: sync status for source bucket shard: 0 2023-10-17T18:54:09.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-17T18:54:09.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7/obj00000000000000000016 2023-10-17T18:54:09.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: entry->timestamp=2023-10-17T18:54:07.996140+0000 2023-10-17T18:54:09.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.100.6 op=0 op_state=1 2023-10-17T18:54:09.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000016[0] 2023-10-17T18:54:09.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: listing bilog for incremental sync; position=00000000002.100.6 2023-10-17T18:54:09.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-10-17T18:54:09.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12/obj00000000000000000062 2023-10-17T18:54:09.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: entry->timestamp=2023-10-17T18:54:08.302970+0000 2023-10-17T18:54:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-10-17T18:54:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000062[0] 2023-10-17T18:54:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-17T18:54:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.722+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: sync status for source bucket shard: 0 2023-10-17T18:54:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: sync status for source bucket shard: 0 2023-10-17T18:54:09.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: sync status for source bucket shard: 0 2023-10-17T18:54:09.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: sync status for source bucket shard: 0 2023-10-17T18:54:09.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: sync status for source bucket shard: 0 2023-10-17T18:54:09.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: sync status for source bucket shard: 0 2023-10-17T18:54:09.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.723+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:27]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: sync status for source bucket shard: 0 2023-10-17T18:54:09.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-17T18:54:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11/obj00000000000000000030 2023-10-17T18:54:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: entry->timestamp=2023-10-17T18:54:07.975365+0000 2023-10-17T18:54:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.98.6 op=0 op_state=1 2023-10-17T18:54:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-17T18:54:09.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15/obj00000000000000000024 2023-10-17T18:54:09.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: entry->timestamp=2023-10-17T18:54:07.895409+0000 2023-10-17T18:54:09.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-17T18:54:09.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: parsed entry: id=00000000002.87.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-17T18:54:09.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9/obj00000000000000000015 2023-10-17T18:54:09.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: entry->timestamp=2023-10-17T18:54:08.223824+0000 2023-10-17T18:54:09.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.87.6 op=0 op_state=1 2023-10-17T18:54:09.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-17T18:54:09.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10/obj00000000000000000026 2023-10-17T18:54:09.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: entry->timestamp=2023-10-17T18:54:07.676288+0000 2023-10-17T18:54:09.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-17T18:54:09.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000030[0] 2023-10-17T18:54:09.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: listing bilog for incremental sync; position=00000000002.98.6 2023-10-17T18:54:09.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000024[0] 2023-10-17T18:54:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-17T18:54:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000015[0] 2023-10-17T18:54:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: listing bilog for incremental sync; position=00000000002.87.6 2023-10-17T18:54:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000026[0] 2023-10-17T18:54:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.726+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-17T18:54:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: sync status for source bucket shard: 0 2023-10-17T18:54:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: sync status for source bucket shard: 0 2023-10-17T18:54:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.755 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: sync status for source bucket shard: 0 2023-10-17T18:54:09.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: sync status for source bucket shard: 0 2023-10-17T18:54:09.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:entry[obj00000000000000000016]: success 2023-10-17T18:54:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:7:6 marker=00000000002.100.6 timestamp=2023-10-17T18:54:07.996140+0000 2023-10-17T18:54:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: sync status for source bucket shard: 0 2023-10-17T18:54:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.727+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: sync status for source bucket shard: 0 2023-10-17T18:54:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: sync status for source bucket shard: 0 2023-10-17T18:54:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: sync status for source bucket shard: 0 2023-10-17T18:54:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: sync status for source bucket shard: 0 2023-10-17T18:54:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: sync status for source bucket shard: 0 2023-10-17T18:54:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: sync status for source bucket shard: 0 2023-10-17T18:54:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: sync status for source bucket shard: 0 2023-10-17T18:54:09.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-17T18:54:09.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14/obj00000000000000000040 2023-10-17T18:54:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: entry->timestamp=2023-10-17T18:54:08.076371+0000 2023-10-17T18:54:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-17T18:54:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-17T18:54:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19/obj00000000000000000045 2023-10-17T18:54:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: entry->timestamp=2023-10-17T18:54:07.655165+0000 2023-10-17T18:54:09.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-17T18:54:09.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-17T18:54:09.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22/obj00000000000000000007 2023-10-17T18:54:09.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: entry->timestamp=2023-10-17T18:54:07.353389+0000 2023-10-17T18:54:09.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-17T18:54:09.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-17T18:54:09.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17/obj00000000000000000002 2023-10-17T18:54:09.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: entry->timestamp=2023-10-17T18:54:07.512320+0000 2023-10-17T18:54:09.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-17T18:54:09.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-17T18:54:09.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25/obj00000000000000000010 2023-10-17T18:54:09.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: entry->timestamp=2023-10-17T18:54:08.243934+0000 2023-10-17T18:54:09.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-17T18:54:09.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000040[0] 2023-10-17T18:54:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-17T18:54:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-17T18:54:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24/obj00000000000000000027 2023-10-17T18:54:09.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: entry->timestamp=2023-10-17T18:54:07.249589+0000 2023-10-17T18:54:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-17T18:54:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000045[0] 2023-10-17T18:54:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-17T18:54:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000007[0] 2023-10-17T18:54:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-17T18:54:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000002[0] 2023-10-17T18:54:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-17T18:54:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000010[0] 2023-10-17T18:54:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-17T18:54:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000027[0] 2023-10-17T18:54:09.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.730+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-17T18:54:09.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: sync status for source bucket shard: 0 2023-10-17T18:54:09.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:entry[obj00000000000000000001]: success 2023-10-17T18:54:09.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:3:6 marker=00000000002.100.6 timestamp=2023-10-17T18:54:08.323466+0000 2023-10-17T18:54:09.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:entry[obj00000000000000000024]: success 2023-10-17T18:54:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:15:6 marker=00000000002.55.6 timestamp=2023-10-17T18:54:07.895409+0000 2023-10-17T18:54:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.732+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.733+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:entry[obj00000000000000000060]: success 2023-10-17T18:54:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.733+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:0:6 marker=00000000001.48.6 timestamp=2023-10-17T18:54:07.735719+0000 2023-10-17T18:54:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.734+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:entry[obj00000000000000000015]: success 2023-10-17T18:54:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.734+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:9:6 marker=00000000002.87.6 timestamp=2023-10-17T18:54:08.223824+0000 2023-10-17T18:54:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-17T18:54:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36/obj00000000000000000008 2023-10-17T18:54:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: entry->timestamp=2023-10-17T18:54:08.139508+0000 2023-10-17T18:54:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-17T18:54:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-10-17T18:54:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-17T18:54:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36/obj00000000000000000008 2023-10-17T18:54:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:entry[obj00000000000000000040]: success 2023-10-17T18:54:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:14:6 marker=00000000002.41.6 timestamp=2023-10-17T18:54:08.076371+0000 2023-10-17T18:54:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:entry[obj00000000000000000027]: success 2023-10-17T18:54:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.737+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:24:6 marker=00000000002.46.6 timestamp=2023-10-17T18:54:07.249589+0000 2023-10-17T18:54:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.738+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:entry[obj00000000000000000007]: success 2023-10-17T18:54:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.738+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:22:6 marker=00000000002.67.6 timestamp=2023-10-17T18:54:07.353389+0000 2023-10-17T18:54:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-17T18:54:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32/obj00000000000000000014 2023-10-17T18:54:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: entry->timestamp=2023-10-17T18:54:08.463058+0000 2023-10-17T18:54:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-17T18:54:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000014[0] 2023-10-17T18:54:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-17T18:54:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:entry[obj00000000000000000026]: success 2023-10-17T18:54:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:10:6 marker=00000000002.69.6 timestamp=2023-10-17T18:54:07.676288+0000 2023-10-17T18:54:09.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:entry[obj00000000000000000030]: success 2023-10-17T18:54:09.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:11:6 marker=00000000002.98.6 timestamp=2023-10-17T18:54:07.975365+0000 2023-10-17T18:54:09.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:entry[obj00000000000000000002]: success 2023-10-17T18:54:09.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:17:6 marker=00000000002.69.6 timestamp=2023-10-17T18:54:07.512320+0000 2023-10-17T18:54:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:entry[obj00000000000000000010]: success 2023-10-17T18:54:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.739+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:25:6 marker=00000000002.57.6 timestamp=2023-10-17T18:54:08.243934+0000 2023-10-17T18:54:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-17T18:54:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28/obj00000000000000000020 2023-10-17T18:54:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: entry->timestamp=2023-10-17T18:54:07.432687+0000 2023-10-17T18:54:09.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-10-17T18:54:09.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000020[0] 2023-10-17T18:54:09.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: listing bilog for incremental sync; position=00000000002.85.6 2023-10-17T18:54:09.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: parsed entry: id=00000000002.98.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-17T18:54:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33/obj00000000000000000019 2023-10-17T18:54:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: entry->timestamp=2023-10-17T18:54:07.575098+0000 2023-10-17T18:54:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.98.6 op=0 op_state=1 2023-10-17T18:54:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000019[0] 2023-10-17T18:54:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: listing bilog for incremental sync; position=00000000002.98.6 2023-10-17T18:54:09.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:entry[obj00000000000000000045]: success 2023-10-17T18:54:09.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.741+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:19:6 marker=00000000002.48.6 timestamp=2023-10-17T18:54:07.655165+0000 2023-10-17T18:54:09.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-17T18:54:09.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35/obj00000000000000000018 2023-10-17T18:54:09.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: entry->timestamp=2023-10-17T18:54:07.816127+0000 2023-10-17T18:54:09.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-17T18:54:09.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000018[0] 2023-10-17T18:54:09.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-17T18:54:09.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:entry[obj00000000000000000065]: success 2023-10-17T18:54:09.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.743+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:5:6 marker=00000000001.71.6 timestamp=2023-10-17T18:54:07.915772+0000 2023-10-17T18:54:09.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.745+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:entry[obj00000000000000000062]: success 2023-10-17T18:54:09.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.745+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:12:6 marker=00000000001.67.6 timestamp=2023-10-17T18:54:08.302970+0000 2023-10-17T18:54:09.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:entry[obj00000000000000000020]: success 2023-10-17T18:54:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:28:6 marker=00000000002.85.6 timestamp=2023-10-17T18:54:07.432687+0000 2023-10-17T18:54:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:entry[obj00000000000000000019]: success 2023-10-17T18:54:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:33:6 marker=00000000002.98.6 timestamp=2023-10-17T18:54:07.575098+0000 2023-10-17T18:54:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:entry[obj00000000000000000014]: success 2023-10-17T18:54:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.746+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:32:6 marker=00000000002.47.6 timestamp=2023-10-17T18:54:08.463058+0000 2023-10-17T18:54:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.747+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:entry[obj00000000000000000018]: success 2023-10-17T18:54:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.747+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:35:6 marker=00000000002.70.6 timestamp=2023-10-17T18:54:07.816127+0000 2023-10-17T18:54:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-17T18:54:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1/obj00000000000000000000 2023-10-17T18:54:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: entry->timestamp=2023-10-17T18:54:08.161271+0000 2023-10-17T18:54:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-17T18:54:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000000[0] 2023-10-17T18:54:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.759+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-17T18:54:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.760+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:entry[obj00000000000000000000]: success 2023-10-17T18:54:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.760+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:1:6 marker=00000000002.43.6 timestamp=2023-10-17T18:54:08.161271+0000 2023-10-17T18:54:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-17T18:54:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6/obj00000000000000000005 2023-10-17T18:54:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: entry->timestamp=2023-10-17T18:54:07.755869+0000 2023-10-17T18:54:09.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-17T18:54:09.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000005[0] 2023-10-17T18:54:09.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.761+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-17T18:54:09.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: parsed entry: id=00000000001.96.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-10-17T18:54:09.792 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8/obj00000000000000000048 2023-10-17T18:54:09.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: entry->timestamp=2023-10-17T18:54:07.836975+0000 2023-10-17T18:54:09.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.96.6 op=0 op_state=1 2023-10-17T18:54:09.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000048[0] 2023-10-17T18:54:09.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: listing bilog for incremental sync; position=00000000001.96.6 2023-10-17T18:54:09.793 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:entry[obj00000000000000000005]: success 2023-10-17T18:54:09.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.762+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:6:6 marker=00000000002.53.6 timestamp=2023-10-17T18:54:07.755869+0000 2023-10-17T18:54:09.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-17T18:54:09.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000033 2023-10-17T18:54:09.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:08.403649+0000 2023-10-17T18:54:09.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-17T18:54:09.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: : injecting data sync error on key=obj00000000000000000033 2023-10-17T18:54:09.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-17T18:54:09.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: failed, retcode=-5 ((5) Input/output error) 2023-10-17T18:54:09.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.764+0000 7f80b1b90780 0 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: ERROR: failed to sync object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000033 2023-10-17T18:54:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-17T18:54:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16/obj00000000000000000012 2023-10-17T18:54:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: entry->timestamp=2023-10-17T18:54:08.383225+0000 2023-10-17T18:54:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-17T18:54:09.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000012[0] 2023-10-17T18:54:09.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.769+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-17T18:54:09.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.770+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:entry[obj00000000000000000012]: success 2023-10-17T18:54:09.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.770+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:16:6 marker=00000000002.45.6 timestamp=2023-10-17T18:54:08.383225+0000 2023-10-17T18:54:09.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-17T18:54:09.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18/obj00000000000000000013 2023-10-17T18:54:09.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: entry->timestamp=2023-10-17T18:54:07.412036+0000 2023-10-17T18:54:09.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-17T18:54:09.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000013[0] 2023-10-17T18:54:09.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-17T18:54:09.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-17T18:54:09.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23/obj00000000000000000039 2023-10-17T18:54:09.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: entry->timestamp=2023-10-17T18:54:07.272515+0000 2023-10-17T18:54:09.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-17T18:54:09.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-17T18:54:09.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20/obj00000000000000000006 2023-10-17T18:54:09.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: entry->timestamp=2023-10-17T18:54:08.484756+0000 2023-10-17T18:54:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-10-17T18:54:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000039[0] 2023-10-17T18:54:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-17T18:54:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000006[0] 2023-10-17T18:54:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.771+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: listing bilog for incremental sync; position=00000000002.93.6 2023-10-17T18:54:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-17T18:54:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29/obj00000000000000000004 2023-10-17T18:54:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: entry->timestamp=2023-10-17T18:54:08.544568+0000 2023-10-17T18:54:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-17T18:54:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000004[0] 2023-10-17T18:54:09.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.772+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-17T18:54:09.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.773+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:entry[obj00000000000000000006]: success 2023-10-17T18:54:09.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.773+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:20:6 marker=00000000002.93.6 timestamp=2023-10-17T18:54:08.484756+0000 2023-10-17T18:54:09.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.773+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:entry[obj00000000000000000013]: success 2023-10-17T18:54:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.773+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:18:6 marker=00000000002.66.6 timestamp=2023-10-17T18:54:07.412036+0000 2023-10-17T18:54:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.773+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:entry[obj00000000000000000039]: success 2023-10-17T18:54:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:23:6 marker=00000000002.59.6 timestamp=2023-10-17T18:54:07.272515+0000 2023-10-17T18:54:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-17T18:54:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26/obj00000000000000000021 2023-10-17T18:54:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: entry->timestamp=2023-10-17T18:54:07.331498+0000 2023-10-17T18:54:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-17T18:54:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000021[0] 2023-10-17T18:54:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-17T18:54:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-17T18:54:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21/obj00000000000000000017 2023-10-17T18:54:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: entry->timestamp=2023-10-17T18:54:07.491710+0000 2023-10-17T18:54:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-17T18:54:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000017[0] 2023-10-17T18:54:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-17T18:54:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:entry[obj00000000000000000004]: success 2023-10-17T18:54:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.774+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:29:6 marker=00000000002.71.6 timestamp=2023-10-17T18:54:08.544568+0000 2023-10-17T18:54:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.777+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:entry[obj00000000000000000021]: success 2023-10-17T18:54:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.777+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:26:6 marker=00000000002.65.6 timestamp=2023-10-17T18:54:07.331498+0000 2023-10-17T18:54:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.777+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:entry[obj00000000000000000017]: success 2023-10-17T18:54:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.777+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:21:6 marker=00000000002.43.6 timestamp=2023-10-17T18:54:07.491710+0000 2023-10-17T18:54:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-10-17T18:54:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30/obj00000000000000000036 2023-10-17T18:54:09.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: entry->timestamp=2023-10-17T18:54:08.055850+0000 2023-10-17T18:54:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-10-17T18:54:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000036[0] 2023-10-17T18:54:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: listing bilog for incremental sync; position=00000000001.63.6 2023-10-17T18:54:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-17T18:54:09.809 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31/obj00000000000000000003 2023-10-17T18:54:09.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: entry->timestamp=2023-10-17T18:54:07.595722+0000 2023-10-17T18:54:09.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-17T18:54:09.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000003[0] 2023-10-17T18:54:09.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-17T18:54:09.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: a sync operation returned error 2023-10-17T18:54:09.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: backing out with sync_status=-5 2023-10-17T18:54:09.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.781+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-10-17T18:54:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34/obj00000000000000000057 2023-10-17T18:54:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: entry->timestamp=2023-10-17T18:54:08.566698+0000 2023-10-17T18:54:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-17T18:54:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000057[0] 2023-10-17T18:54:09.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-17T18:54:09.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: sync status for source bucket shard: 0 2023-10-17T18:54:09.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.782+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.783+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:entry[obj00000000000000000003]: success 2023-10-17T18:54:09.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.783+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:31:6 marker=00000000002.67.6 timestamp=2023-10-17T18:54:07.595722+0000 2023-10-17T18:54:09.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: a sync operation returned error 2023-10-17T18:54:09.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: backing out with sync_status=-5 2023-10-17T18:54:09.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: incremental sync on bucket failed, retcode=-5 2023-10-17T18:54:09.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: ERROR: incremental sync failed. error: -5 2023-10-17T18:54:09.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-17T18:54:09.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:entry[obj00000000000000000048]: success 2023-10-17T18:54:09.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:8:6 marker=00000000001.96.6 timestamp=2023-10-17T18:54:07.836975+0000 2023-10-17T18:54:09.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: sync status for source bucket shard: 0 2023-10-17T18:54:09.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.806+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position= 2023-10-17T18:54:09.827 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-17T18:54:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36/obj00000000000000000008 2023-10-17T18:54:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: entry->timestamp=2023-10-17T18:54:08.139508+0000 2023-10-17T18:54:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-17T18:54:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000008[0] 2023-10-17T18:54:09.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-17T18:54:09.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:entry[obj00000000000000000036]: success 2023-10-17T18:54:09.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.826+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:30:6 marker=00000000001.63.6 timestamp=2023-10-17T18:54:08.055850+0000 2023-10-17T18:54:09.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.828+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:entry[obj00000000000000000008]: success 2023-10-17T18:54:09.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.828+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:36:6 marker=00000000002.41.6 timestamp=2023-10-17T18:54:08.139508+0000 2023-10-17T18:54:09.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.829+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:entry[obj00000000000000000057]: success 2023-10-17T18:54:09.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.829+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:34:6 marker=00000000001.49.6 timestamp=2023-10-17T18:54:08.566698+0000 2023-10-17T18:54:09.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-17T18:54:09.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: syncing object: hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4/obj00000000000000000033 2023-10-17T18:54:09.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: entry->timestamp=2023-10-17T18:54:08.403649+0000 2023-10-17T18:54:09.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b=bucket sync single entry (source_zone=4c84373e-d451-49bb-a8c9-c37d5e5e2160) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-17T18:54:09.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 5 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 4c84373e-d451-49bb-a8c9-c37d5e5e2160/:hbmuog-58[4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23])/obj00000000000000000033[0] 2023-10-17T18:54:09.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.848+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-17T18:54:09.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.849+0000 7f80b1b90780 10 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:entry[obj00000000000000000033]: success 2023-10-17T18:54:09.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:09.849+0000 7f80b1b90780 20 RGW-SYNC:bucket[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23<-hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]:inc_sync[hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4]: updating marker marker_oid=bucket.sync-status.4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23:4:6 marker=00000000002.69.6 timestamp=2023-10-17T18:54:08.403649+0000 2023-10-17T18:54:09.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:09.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:09.913 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:10.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:10.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:10.041 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:54:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000000", 2023-10-17T18:54:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14 2023-10-17T18:54:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.161271Z", 2023-10-17T18:54:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.10960980216672472744", 2023-10-17T18:54:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000001", 2023-10-17T18:54:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 16 2023-10-17T18:54:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.323466Z", 2023-10-17T18:54:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.3786077250853067924", 2023-10-17T18:54:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000002", 2023-10-17T18:54:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 18 2023-10-17T18:54:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.512320Z", 2023-10-17T18:54:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.9294936941842906107", 2023-10-17T18:54:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000003", 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 13 2023-10-17T18:54:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.595722Z", 2023-10-17T18:54:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14833032046866545913", 2023-10-17T18:54:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000004", 2023-10-17T18:54:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 20 2023-10-17T18:54:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.544568Z", 2023-10-17T18:54:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.4129551063212458548", 2023-10-17T18:54:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000005", 2023-10-17T18:54:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.755869Z", 2023-10-17T18:54:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2131774917203392272", 2023-10-17T18:54:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000006", 2023-10-17T18:54:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 20 2023-10-17T18:54:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.484756Z", 2023-10-17T18:54:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.6731317784796079527", 2023-10-17T18:54:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000007", 2023-10-17T18:54:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.353389Z", 2023-10-17T18:54:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2298239784933160722", 2023-10-17T18:54:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000008", 2023-10-17T18:54:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.139508Z", 2023-10-17T18:54:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.8114292485759509793", 2023-10-17T18:54:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000009", 2023-10-17T18:54:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.996848Z", 2023-10-17T18:54:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.17362399132183523169", 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000010", 2023-10-17T18:54:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14 2023-10-17T18:54:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.243934Z", 2023-10-17T18:54:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.8701934196219345301", 2023-10-17T18:54:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000011", 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.299380Z", 2023-10-17T18:54:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.487345991767047120", 2023-10-17T18:54:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000012", 2023-10-17T18:54:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.383225Z", 2023-10-17T18:54:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2459423231783548362", 2023-10-17T18:54:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000013", 2023-10-17T18:54:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.412036Z", 2023-10-17T18:54:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.11219770750303349672", 2023-10-17T18:54:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000014", 2023-10-17T18:54:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.463058Z", 2023-10-17T18:54:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.1299421937340941137", 2023-10-17T18:54:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000015", 2023-10-17T18:54:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.223824Z", 2023-10-17T18:54:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.16536391290304920458", 2023-10-17T18:54:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000016", 2023-10-17T18:54:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 13 2023-10-17T18:54:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.996140Z", 2023-10-17T18:54:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.16116690491208815179", 2023-10-17T18:54:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000017", 2023-10-17T18:54:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 15 2023-10-17T18:54:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.491710Z", 2023-10-17T18:54:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14814138118834846093", 2023-10-17T18:54:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000018", 2023-10-17T18:54:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.816127Z", 2023-10-17T18:54:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14579553508079015962", 2023-10-17T18:54:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000019", 2023-10-17T18:54:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 11 2023-10-17T18:54:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.575098Z", 2023-10-17T18:54:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.169649074026672678", 2023-10-17T18:54:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000020", 2023-10-17T18:54:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.432687Z", 2023-10-17T18:54:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.7099917824907232769", 2023-10-17T18:54:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000021", 2023-10-17T18:54:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.331498Z", 2023-10-17T18:54:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.5773484072811693770", 2023-10-17T18:54:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000022", 2023-10-17T18:54:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14 2023-10-17T18:54:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.079533Z", 2023-10-17T18:54:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.11081167753170042757", 2023-10-17T18:54:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000023", 2023-10-17T18:54:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.872917Z", 2023-10-17T18:54:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.5413678364886180364", 2023-10-17T18:54:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000024", 2023-10-17T18:54:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 13 2023-10-17T18:54:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.895409Z", 2023-10-17T18:54:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.11520349445859214499", 2023-10-17T18:54:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000025", 2023-10-17T18:54:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.715529Z", 2023-10-17T18:54:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14468188470383027333", 2023-10-17T18:54:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000026", 2023-10-17T18:54:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.676288Z", 2023-10-17T18:54:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2561581569961877577", 2023-10-17T18:54:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000027", 2023-10-17T18:54:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 18 2023-10-17T18:54:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.249589Z", 2023-10-17T18:54:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.4895733914777756666", 2023-10-17T18:54:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000028", 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:01.550744Z", 2023-10-17T18:54:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14544956412483886300", 2023-10-17T18:54:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000029", 2023-10-17T18:54:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.320451Z", 2023-10-17T18:54:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.6931972650755103335", 2023-10-17T18:54:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000030", 2023-10-17T18:54:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.975365Z", 2023-10-17T18:54:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.5481430692059704663", 2023-10-17T18:54:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000031", 2023-10-17T18:54:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.037045Z", 2023-10-17T18:54:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.3612346652497674527", 2023-10-17T18:54:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000032", 2023-10-17T18:54:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.095688Z", 2023-10-17T18:54:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14130021187478313995", 2023-10-17T18:54:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000033", 2023-10-17T18:54:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.403649Z", 2023-10-17T18:54:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.15170542761572710425", 2023-10-17T18:54:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000036", 2023-10-17T18:54:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.055850Z", 2023-10-17T18:54:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.18010346179415539370", 2023-10-17T18:54:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000039", 2023-10-17T18:54:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.272515Z", 2023-10-17T18:54:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.17093454615110113579", 2023-10-17T18:54:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000040", 2023-10-17T18:54:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14 2023-10-17T18:54:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.076371Z", 2023-10-17T18:54:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.1514144752348356195", 2023-10-17T18:54:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000042", 2023-10-17T18:54:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.520429Z", 2023-10-17T18:54:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.3911163884429353445", 2023-10-17T18:54:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000045", 2023-10-17T18:54:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.655165Z", 2023-10-17T18:54:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.3056645182284855816", 2023-10-17T18:54:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000046", 2023-10-17T18:54:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.374892Z", 2023-10-17T18:54:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.6389260822368147384", 2023-10-17T18:54:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000048", 2023-10-17T18:54:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.836975Z", 2023-10-17T18:54:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.9394133477461296004", 2023-10-17T18:54:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000049", 2023-10-17T18:54:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.553443Z", 2023-10-17T18:54:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.6517732238783647333", 2023-10-17T18:54:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000050", 2023-10-17T18:54:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.700149Z", 2023-10-17T18:54:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.14485173092269354912", 2023-10-17T18:54:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000055", 2023-10-17T18:54:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.852931Z", 2023-10-17T18:54:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.13217439736039828054", 2023-10-17T18:54:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000056", 2023-10-17T18:54:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.779567Z", 2023-10-17T18:54:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.13937200272174810734", 2023-10-17T18:54:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000057", 2023-10-17T18:54:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.566698Z", 2023-10-17T18:54:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.10986840210969035020", 2023-10-17T18:54:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000058", 2023-10-17T18:54:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.456624Z", 2023-10-17T18:54:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.12158850325798057256", 2023-10-17T18:54:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000059", 2023-10-17T18:54:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.856098Z", 2023-10-17T18:54:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.18178793074098536806", 2023-10-17T18:54:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000060", 2023-10-17T18:54:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.735719Z", 2023-10-17T18:54:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.8319107328533786510", 2023-10-17T18:54:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000061", 2023-10-17T18:54:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.879743Z", 2023-10-17T18:54:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.9909667822171190602", 2023-10-17T18:54:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000062", 2023-10-17T18:54:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.302970Z", 2023-10-17T18:54:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.6215146697611835005", 2023-10-17T18:54:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000065", 2023-10-17T18:54:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.915772Z", 2023-10-17T18:54:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.1990057211182857921", 2023-10-17T18:54:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000133", 2023-10-17T18:54:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.912310Z", 2023-10-17T18:54:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.15930446089882967468", 2023-10-17T18:54:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000271", 2023-10-17T18:54:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 17 2023-10-17T18:54:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.541152Z", 2023-10-17T18:54:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.18054643386379943791", 2023-10-17T18:54:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000283", 2023-10-17T18:54:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 12 2023-10-17T18:54:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.859629Z", 2023-10-17T18:54:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2636933909829041256", 2023-10-17T18:54:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000423", 2023-10-17T18:54:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.365 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.365 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.365 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.562585Z", 2023-10-17T18:54:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.3121098804879840092", 2023-10-17T18:54:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000467", 2023-10-17T18:54:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2023-10-17T18:54:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 18 2023-10-17T18:54:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.940754Z", 2023-10-17T18:54:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.10311735102054054398", 2023-10-17T18:54:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:10.375 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:54:10.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:10.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:10.377 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:10.432 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:10.432 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:10.466 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:54:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000000", 2023-10-17T18:54:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.161271Z", 2023-10-17T18:54:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_-68CQeJdc5Aj9WxBk0mP_Q4F1a66nvE", 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000001", 2023-10-17T18:54:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.323466Z", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_mWBNNE_oED61MrdXGKGTaj4pLSMJPnA", 2023-10-17T18:54:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000002", 2023-10-17T18:54:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.512320Z", 2023-10-17T18:54:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_0fIIZOKoJUYOD9SEd_tMX92MpxEBfx7", 2023-10-17T18:54:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000003", 2023-10-17T18:54:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.595722Z", 2023-10-17T18:54:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_FfYkcfupZ2Wyunx7fusghZ8s_BAoUNl", 2023-10-17T18:54:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000004", 2023-10-17T18:54:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14 2023-10-17T18:54:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.544568Z", 2023-10-17T18:54:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_I74Bn7wB8ECWDkuPl2nilJ0i-zVuLFh", 2023-10-17T18:54:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000005", 2023-10-17T18:54:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.755869Z", 2023-10-17T18:54:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "__EJ9saRO0yXgF8xSJEKHwnt34LaBiR5", 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000006", 2023-10-17T18:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 9 2023-10-17T18:54:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.484756Z", 2023-10-17T18:54:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_sbaGF9jhmjEcLNPzClTTGnvJ8YYMLI0", 2023-10-17T18:54:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000007", 2023-10-17T18:54:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.353389Z", 2023-10-17T18:54:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_H4pxmvgn7zZY29YBGm0Wz5QB4kOXYm9", 2023-10-17T18:54:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000008", 2023-10-17T18:54:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.139508Z", 2023-10-17T18:54:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_fdzIqA15iHj5AkmlRNzepJDCjhXdqMu", 2023-10-17T18:54:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000009", 2023-10-17T18:54:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.996848Z", 2023-10-17T18:54:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_r9RmZdOaQ6wLV1rMtGz53P_tG3o422G", 2023-10-17T18:54:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000010", 2023-10-17T18:54:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.243934Z", 2023-10-17T18:54:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_tKDiwhAcL_wMgOgAqbwNXeB1Y6xPAxQ", 2023-10-17T18:54:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000011", 2023-10-17T18:54:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.299380Z", 2023-10-17T18:54:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_VIzjbNOIiMiPAdHLzkCf5R6TWFaBRjM", 2023-10-17T18:54:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000012", 2023-10-17T18:54:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.383225Z", 2023-10-17T18:54:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_OkSHJPX6ZAiKOOeHDEjmng5NVXe5kVc", 2023-10-17T18:54:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000013", 2023-10-17T18:54:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.412036Z", 2023-10-17T18:54:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_KNDKLkv3cUWQ87b4X0ZAAp0DZIEGN0n", 2023-10-17T18:54:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000014", 2023-10-17T18:54:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.463058Z", 2023-10-17T18:54:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_ddK1DPS8xfU5U_XsoDISs6No22yLXPX", 2023-10-17T18:54:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000015", 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.223824Z", 2023-10-17T18:54:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_jBOueH7W66eKMVBgJ_HH1LF2gLqBkPF", 2023-10-17T18:54:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000016", 2023-10-17T18:54:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.996140Z", 2023-10-17T18:54:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_9fi7kFhothpDeJUK4IWIOheuo2txdAz", 2023-10-17T18:54:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000017", 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.491710Z", 2023-10-17T18:54:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_ro3LYZk_OkyBGKAYrzbWA4iXULS08x2", 2023-10-17T18:54:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000018", 2023-10-17T18:54:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 13 2023-10-17T18:54:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.816127Z", 2023-10-17T18:54:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_MCefVpLdJWKy2KtYmdSb-LIPheRHH4k", 2023-10-17T18:54:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000019", 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.575098Z", 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_h7Hu_hHwEf72Wnk-PR9Rh3gpwjSIHVM", 2023-10-17T18:54:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000020", 2023-10-17T18:54:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.432687Z", 2023-10-17T18:54:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_38MsR3avGIZfKC32nFn3OekoLwxh_70", 2023-10-17T18:54:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000021", 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.331498Z", 2023-10-17T18:54:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_GUaYoGk8E-v547CtbKb_VFHqGsM_eXL", 2023-10-17T18:54:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000022", 2023-10-17T18:54:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 15 2023-10-17T18:54:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.079533Z", 2023-10-17T18:54:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_uj0Dix6JVUlsB0aKthTaSB9ITkoOtgW", 2023-10-17T18:54:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000023", 2023-10-17T18:54:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.872917Z", 2023-10-17T18:54:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_QtwUEwrBhqblLHX9kvr-Nj5JcqYaNsu", 2023-10-17T18:54:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000024", 2023-10-17T18:54:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.895409Z", 2023-10-17T18:54:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_zm-9oZdNmWkwstk4KzdxEwmyWUKNvXO", 2023-10-17T18:54:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000025", 2023-10-17T18:54:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.715529Z", 2023-10-17T18:54:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_shcPPDv1BEhxENVWv8TrKoA_YTUSdlg", 2023-10-17T18:54:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000026", 2023-10-17T18:54:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.676288Z", 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_jaJFDAK3ArTei7gX8t3AkCvrjFyR1rd", 2023-10-17T18:54:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000027", 2023-10-17T18:54:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 10 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.618 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.618 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.249589Z", 2023-10-17T18:54:10.618 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.618 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.618 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.619 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.619 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.619 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.619 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.619 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.620 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "__8sg1mLIfgBWsg1qghJg2WPWRO8YhNs", 2023-10-17T18:54:10.620 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.620 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.620 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000028", 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.621 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.622 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.622 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.622 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:01.550744Z", 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.623 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.624 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.624 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.624 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.624 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.624 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.625 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.625 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_sF68-_tmobV9RVBS78fLw5dJ00Q1JGN", 2023-10-17T18:54:10.625 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.626 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.626 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000029", 2023-10-17T18:54:10.627 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.627 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.627 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.627 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.628 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.628 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.628 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.628 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.628 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.629 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.320451Z", 2023-10-17T18:54:10.629 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.629 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.629 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.630 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.630 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.630 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.630 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.630 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.631 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_K8dRvpsl9FO3zXBiLmk_IJvkn0E3bBh", 2023-10-17T18:54:10.631 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.631 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.631 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.632 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000030", 2023-10-17T18:54:10.632 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.633 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.633 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.633 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.633 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.634 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.634 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.634 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.634 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.634 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.975365Z", 2023-10-17T18:54:10.635 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.635 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.635 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.635 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.635 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.636 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.636 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.636 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.636 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_owmLQOc4Q-YHRqOb7ydYDTMOEppYOeU", 2023-10-17T18:54:10.637 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.637 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.638 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000031", 2023-10-17T18:54:10.638 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.638 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.638 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.639 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.639 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.639 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.640 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.640 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.640 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.640 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.037045Z", 2023-10-17T18:54:10.640 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.641 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.641 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.641 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.641 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.641 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_ujezxF_4eFVJZ2oPkDnKI5Buy8Sr4Qb", 2023-10-17T18:54:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000032", 2023-10-17T18:54:10.644 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.644 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.644 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.644 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.645 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.645 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.645 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.645 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.646 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.646 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.095688Z", 2023-10-17T18:54:10.646 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.646 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.646 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.647 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.647 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.647 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.647 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.647 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.648 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_CXGvqG9PfUAdPF5U2wfBuU16S2TCcxr", 2023-10-17T18:54:10.648 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.649 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.649 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.650 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000033", 2023-10-17T18:54:10.650 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.650 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.650 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.651 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.651 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.651 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.652 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.652 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.652 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.652 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.403649Z", 2023-10-17T18:54:10.652 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.653 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.653 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.653 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.653 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.653 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.654 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.654 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.654 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_CwZ50YbsjvgPHkFHMj-Qpfci5zF7pbO", 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.655 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000036", 2023-10-17T18:54:10.656 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.656 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.656 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.656 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.657 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.658 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.055850Z", 2023-10-17T18:54:10.658 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.658 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.658 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.659 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.659 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.659 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.659 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.659 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.660 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_S4NpIwnNOEHq0wipFqUt1_-Cdvkzrf3", 2023-10-17T18:54:10.660 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.660 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.660 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.661 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000039", 2023-10-17T18:54:10.661 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.662 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.662 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.662 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.663 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.663 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.663 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.663 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.663 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.664 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.272515Z", 2023-10-17T18:54:10.664 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.664 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.664 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.664 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.665 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.665 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.665 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.665 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.666 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_Ram3ltgIbd3OP9AqHUXcSeJRjjhi9i5", 2023-10-17T18:54:10.666 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.666 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.666 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.667 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000040", 2023-10-17T18:54:10.667 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.667 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.668 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.668 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.668 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.668 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.669 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.669 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.669 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.669 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.076371Z", 2023-10-17T18:54:10.669 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.670 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.670 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.670 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.670 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.671 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.671 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.671 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.671 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_HY8R_VYskSiUkyqA8xSalTpyz43qiEF", 2023-10-17T18:54:10.672 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.672 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.672 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.673 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000042", 2023-10-17T18:54:10.673 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.673 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.673 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.673 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.674 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.674 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.674 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.674 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.675 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.675 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.520429Z", 2023-10-17T18:54:10.675 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.675 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.675 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.676 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.676 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.676 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.676 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.676 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.677 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_fkUzFke54NQ-Ro-zbgvoFgz6fIGup12", 2023-10-17T18:54:10.677 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.677 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.677 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.678 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000045", 2023-10-17T18:54:10.678 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.678 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.679 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.679 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.679 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.679 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.655165Z", 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.680 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.681 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.681 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.681 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.681 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.681 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.682 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.682 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_h6g8U0yn3k2jBFmWgutgZZb_oYFEsfT", 2023-10-17T18:54:10.682 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.682 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000046", 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.683 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.684 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.684 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.684 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.684 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.374892Z", 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.685 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_G2aH3a1AP_Zmt_xmrS5d-iwEu9cH0Tc", 2023-10-17T18:54:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000048", 2023-10-17T18:54:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.836975Z", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_GnL8UIkn5mJniXppwPTqePfOF9CpWK9", 2023-10-17T18:54:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000049", 2023-10-17T18:54:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.553443Z", 2023-10-17T18:54:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_OV-HCbTqtOsFnBq-xlDj-8kJs_h6tGy", 2023-10-17T18:54:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.699 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000050", 2023-10-17T18:54:10.699 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.699 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.699 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.700 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.700 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.700 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.700 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.700149Z", 2023-10-17T18:54:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.701 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.702 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.702 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.702 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.702 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.702 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.703 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.703 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_VQoRkr5h_IXutRm2GGqHX7iTjYrXVCn", 2023-10-17T18:54:10.703 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.703 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.704 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.704 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000055", 2023-10-17T18:54:10.704 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.705 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.705 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.705 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.705 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.706 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.706 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.706 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.706 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.706 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.852931Z", 2023-10-17T18:54:10.707 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.707 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.707 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.707 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.707 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.708 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.708 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.708 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.708 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_dxikKmj1ED7SJhRWrJ55fi3K3im8jpy", 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.709 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000056", 2023-10-17T18:54:10.710 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.710 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.710 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.710 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 3 2023-10-17T18:54:10.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.711 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.711 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.711 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.711 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.711 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.779567Z", 2023-10-17T18:54:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.713 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.713 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.713 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.713 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.714 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_yX9kAqO4HyD7LFxa0TP0BpGtuWErlPY", 2023-10-17T18:54:10.714 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.714 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.714 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.715 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000057", 2023-10-17T18:54:10.715 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.715 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.715 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.717 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.717 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.566698Z", 2023-10-17T18:54:10.717 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.717 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.717 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.718 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.718 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.718 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.718 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.718 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.719 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_qDd6nKxnfC2P4HvFLRIVKH8FJwmxYJt", 2023-10-17T18:54:10.719 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.719 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.719 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.720 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000058", 2023-10-17T18:54:10.720 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.720 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.721 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 7 2023-10-17T18:54:10.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.721 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.721 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:03.456624Z", 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.722 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.723 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.723 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.723 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.723 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.723 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.724 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.724 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_mjZQFCGT2A9Xajp9z5hfxUmN5MWmmla", 2023-10-17T18:54:10.724 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.724 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.725 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.725 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000059", 2023-10-17T18:54:10.725 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.726 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.726 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.726 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.726 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:04.856098Z", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.728 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.729 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.729 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.729 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_4lwIi-TZ58Zw6Oy6YSpCB79esExgL-V", 2023-10-17T18:54:10.729 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.730 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.730 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.730 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000060", 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.731 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.732 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.732 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.732 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.732 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.732 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.735719Z", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.733 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.734 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.734 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.734 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_BQknilELz6s7hm6cQyuMkzotZgj3lWy", 2023-10-17T18:54:10.734 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.735 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.735 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.735 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000061", 2023-10-17T18:54:10.737 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.737 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.737 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.737 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.738 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.738 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.738 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.738 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.739 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.739 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.879743Z", 2023-10-17T18:54:10.739 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.739 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.739 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.740 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.740 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.740 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.740 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.740 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_QcerMS5yty7N_QX16iDUS5zoETZ0GoN", 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.742 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000062", 2023-10-17T18:54:10.742 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.742 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.742 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.743 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 8 2023-10-17T18:54:10.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.743 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.743 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.743 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.744 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.744 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:08.302970Z", 2023-10-17T18:54:10.744 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.745 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.745 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.745 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.745 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.745 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.746 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.746 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.746 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_OfgYx2ApCgULpSmgNpC305zl1U0wu7W", 2023-10-17T18:54:10.746 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.747 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000065", 2023-10-17T18:54:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 6 2023-10-17T18:54:10.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.749 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 5, 2023-10-17T18:54:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:07.915772Z", 2023-10-17T18:54:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.750 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.751 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_-ZgaF1EQLIQ_IYtaJ-v0wXpbtweEME6", 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.752 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000133", 2023-10-17T18:54:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.753 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 5 2023-10-17T18:54:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.754 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:05.912310Z", 2023-10-17T18:54:10.755 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.756 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_hAM7GABaOEG-oaAe1t7OWLe5PN-iu9T", 2023-10-17T18:54:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000271", 2023-10-17T18:54:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 17 2023-10-17T18:54:10.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.760 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.541152Z", 2023-10-17T18:54:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.761 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_hHxY0IyCDyWKxLwEYpBqPEeAahMU9qB", 2023-10-17T18:54:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000283", 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.764 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 4 2023-10-17T18:54:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.765 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 6, 2023-10-17T18:54:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.859629Z", 2023-10-17T18:54:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.767 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_mxWJdKdu_1nOdq4xBP-cfZs5Zlw4cfe", 2023-10-17T18:54:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.768 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000423", 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.769 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 2 2023-10-17T18:54:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.770 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 16, 2023-10-17T18:54:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.562585Z", 2023-10-17T18:54:10.771 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.772 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_eHQDY_eJELRoASgUbuQAuMD-UQ7eSqs", 2023-10-17T18:54:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.773 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:10.775 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "obj00000000000000000467", 2023-10-17T18:54:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "instance": "", 2023-10-17T18:54:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ver": { 2023-10-17T18:54:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 13, 2023-10-17T18:54:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 16 2023-10-17T18:54:10.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "locator": "", 2023-10-17T18:54:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "exists": true, 2023-10-17T18:54:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "meta": { 2023-10-17T18:54:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "category": 1, 2023-10-17T18:54:10.777 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2023-10-17T18:54:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "mtime": "2023-10-17T18:54:06.940754Z", 2023-10-17T18:54:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-17T18:54:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": "", 2023-10-17T18:54:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "owner": "rgw-multisite-test-user", 2023-10-17T18:54:10.778 INFO:teuthology.orchestra.run.smithi049.stdout: "owner_display_name": "Multisite Test User", 2023-10-17T18:54:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "content_type": "application/octet-stream", 2023-10-17T18:54:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "accounted_size": 3, 2023-10-17T18:54:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "user_data": "", 2023-10-17T18:54:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: "appendable": false 2023-10-17T18:54:10.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "_fqZC9CHqooRc67VUrcc80nIcYUYB9tt", 2023-10-17T18:54:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2023-10-17T18:54:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "pending_map": [], 2023-10-17T18:54:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: "versioned_epoch": 0 2023-10-17T18:54:10.780 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:10.781 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:54:10.781 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T18:54:10.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T18:54:10.784 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T18:54:10.785 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T18:54:10.808 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:54:10.808 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:54:10.808 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T18:54:11.809 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:11.836 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T18:54:11.836 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T18:54:11.838 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T18:54:11.839 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T18:54:11.839 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-58 2023-10-17T18:54:11.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:54:11.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:54:11.840 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:54:11.938 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:11.939 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:11.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:11.978+0000 7fdf490ab780 1 bucket sync caught up with source: 2023-10-17T18:54:11.980 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.48.6, 00000000002.43.6, , 00000000002.100.6, 00000000002.69.6, 00000000001.71.6, 00000000002.53.6, 00000000002.100.6, 00000000001.96.6, 00000000002.87.6, 00000000002.69.6, 00000000002.98.6, 00000000001.67.6, , 00000000002.41.6, 00000000002.55.6, 00000000002.45.6, 00000000002.69.6, 00000000002.66.6, 00000000002.48.6, 00000000002.93.6, 00000000002.43.6, 00000000002.67.6, 00000000002.59.6, 00000000002.46.6, 00000000002.57.6, 00000000002.65.6, , 00000000002.85.6, 00000000002.71.6, 00000000001.63.6, 00000000002.67.6, 00000000002.47.6, 00000000002.98.6, 00000000001.49.6, 00000000002.70.6, 00000000002.41.6] 2023-10-17T18:54:11.980 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.48.6, 00000000002.43.6, , 00000000002.100.6, 00000000002.69.6, 00000000001.71.6, 00000000002.53.6, 00000000002.100.6, 00000000001.96.6, 00000000002.87.6, 00000000002.69.6, 00000000002.98.6, 00000000001.67.6, , 00000000002.41.6, 00000000002.55.6, 00000000002.45.6, 00000000002.69.6, 00000000002.66.6, 00000000002.48.6, 00000000002.93.6, 00000000002.43.6, 00000000002.67.6, 00000000002.59.6, 00000000002.46.6, 00000000002.57.6, 00000000002.65.6, , 00000000002.85.6, 00000000002.71.6, 00000000001.63.6, 00000000002.67.6, 00000000002.47.6, 00000000002.98.6, 00000000001.49.6, 00000000002.70.6, 00000000002.41.6] 2023-10-17T18:54:11.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:11.978+0000 7fdf490ab780 0 bucket checkpoint complete 2023-10-17T18:54:11.993 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-58 2023-10-17T18:54:11.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:54:11.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:54:11.995 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:54:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:12.053 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:12.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:12.087+0000 7f2bc863f780 1 bucket sync caught up with source: 2023-10-17T18:54:12.089 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000005.190.6, 00000000007.176.6, 00000000005.180.6, 00000000009.357.6, 00000000005.265.6, 00000000006.132.6, 00000000008.131.6, 00000000005.45.6, 00000000007.311.6, 00000000006.249.6, 00000000004.256.6] 2023-10-17T18:54:12.089 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000005.190.6, 00000000007.176.6, 00000000005.180.6, 00000000009.357.6, 00000000005.265.6, 00000000006.132.6, 00000000008.131.6, 00000000005.45.6, 00000000007.311.6, 00000000006.249.6, 00000000004.256.6] 2023-10-17T18:54:12.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:12.087+0000 7f2bc863f780 0 bucket checkpoint complete 2023-10-17T18:54:12.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-58 zones={a2, a1} 2023-10-17T18:54:12.103 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:12.103 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:12.104 DEBUG:boto:Method: HEAD 2023-10-17T18:54:12.104 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:12.104 DEBUG:boto:Data: 2023-10-17T18:54:12.105 DEBUG:boto:Headers: {} 2023-10-17T18:54:12.105 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:12.106 DEBUG:boto:Port: 8001 2023-10-17T18:54:12.106 DEBUG:boto:Params: {} 2023-10-17T18:54:12.107 DEBUG:boto:Token: None 2023-10-17T18:54:12.107 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:12 GMT /hbmuog-58/ 2023-10-17T18:54:12.107 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4= 2023-10-17T18:54:12.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4=', 'Content-Length': '0'} 2023-10-17T18:54:12.109 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-17T18:54:12.143 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:54:12.932 DEBUG:boto:Token: None 2023-10-17T18:54:12.933 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:12 GMT /hbmuog-58/ 2023-10-17T18:54:12.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4= 2023-10-17T18:54:12.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4=', 'Content-Length': '0'} 2023-10-17T18:54:12.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000861d41503ce3263b-00652ed854-1b46-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:12.941 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:12.941 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:12.941 DEBUG:boto:Method: HEAD 2023-10-17T18:54:12.942 DEBUG:boto:Path: /hbmuog-58/ 2023-10-17T18:54:12.942 DEBUG:boto:Data: 2023-10-17T18:54:12.943 DEBUG:boto:Headers: {} 2023-10-17T18:54:12.943 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:12.943 DEBUG:boto:Port: 8000 2023-10-17T18:54:12.944 DEBUG:boto:Params: {} 2023-10-17T18:54:12.944 DEBUG:boto:Token: None 2023-10-17T18:54:12.945 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:12 GMT /hbmuog-58/ 2023-10-17T18:54:12.945 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4= 2023-10-17T18:54:12.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l0mKT4RsqOpOa3bC9PLOk47NxH4=', 'Content-Length': '0'} 2023-10-17T18:54:12.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000615fd5efb02f11da-00652ed854-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:12.950 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:54:12.951 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:12.951 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:12.952 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.952 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.952 DEBUG:boto:Method: GET 2023-10-17T18:54:12.953 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.953 DEBUG:boto:Data: 2023-10-17T18:54:12.954 DEBUG:boto:Headers: {} 2023-10-17T18:54:12.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:12.955 DEBUG:boto:Port: 8001 2023-10-17T18:54:12.955 DEBUG:boto:Params: {} 2023-10-17T18:54:12.955 DEBUG:boto:Token: None 2023-10-17T18:54:12.956 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:12 GMT /hbmuog-58/?versions 2023-10-17T18:54:12.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K1TXzyZk3oKwioytTwUb7BqX6sI= 2023-10-17T18:54:12.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K1TXzyZk3oKwioytTwUb7BqX6sI=', 'Content-Length': '0'} 2023-10-17T18:54:12.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000085555341344873a-00652ed854-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:12.960 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:12.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-17T18:54:12.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-17T18:54:12.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-17T18:54:12.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-17T18:54:12.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-17T18:54:12.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-17T18:54:12.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-17T18:54:12.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-17T18:54:12.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-17T18:54:12.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-17T18:54:12.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-17T18:54:12.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-17T18:54:12.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-17T18:54:12.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-17T18:54:12.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-17T18:54:12.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-17T18:54:12.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-17T18:54:12.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-17T18:54:12.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-17T18:54:12.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-17T18:54:12.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-17T18:54:12.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-17T18:54:12.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-17T18:54:12.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-17T18:54:12.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-17T18:54:12.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-17T18:54:12.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-17T18:54:12.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-17T18:54:12.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-17T18:54:12.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-17T18:54:12.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-17T18:54:12.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-17T18:54:12.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-17T18:54:12.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-17T18:54:12.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-17T18:54:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-17T18:54:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-17T18:54:12.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-17T18:54:12.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-17T18:54:12.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-17T18:54:12.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-17T18:54:12.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-17T18:54:12.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-17T18:54:12.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-17T18:54:12.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-10-17T18:54:12.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-17T18:54:12.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-17T18:54:12.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-17T18:54:12.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-17T18:54:12.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-17T18:54:12.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-17T18:54:12.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-17T18:54:12.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-17T18:54:12.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-17T18:54:12.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-17T18:54:12.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-17T18:54:12.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-17T18:54:12.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:54:12.985 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:12.986 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:12.986 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.987 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.987 DEBUG:boto:Method: GET 2023-10-17T18:54:12.987 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:12.988 DEBUG:boto:Data: 2023-10-17T18:54:12.988 DEBUG:boto:Headers: {} 2023-10-17T18:54:12.989 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:12.989 DEBUG:boto:Port: 8000 2023-10-17T18:54:12.989 DEBUG:boto:Params: {} 2023-10-17T18:54:12.990 DEBUG:boto:Token: None 2023-10-17T18:54:12.990 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:12 GMT /hbmuog-58/?versions 2023-10-17T18:54:12.991 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K1TXzyZk3oKwioytTwUb7BqX6sI= 2023-10-17T18:54:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K1TXzyZk3oKwioytTwUb7BqX6sI=', 'Content-Length': '0'} 2023-10-17T18:54:12.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000737909a9d0f087d4-00652ed854-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:12.997 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-17T18:54:13.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-17T18:54:13.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-17T18:54:13.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-17T18:54:13.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-17T18:54:13.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-17T18:54:13.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-17T18:54:13.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-17T18:54:13.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-17T18:54:13.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-17T18:54:13.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-17T18:54:13.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-17T18:54:13.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-17T18:54:13.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-17T18:54:13.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-17T18:54:13.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-17T18:54:13.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-17T18:54:13.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-17T18:54:13.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-17T18:54:13.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-17T18:54:13.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-17T18:54:13.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-17T18:54:13.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-17T18:54:13.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-17T18:54:13.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-17T18:54:13.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-17T18:54:13.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-17T18:54:13.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-17T18:54:13.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-17T18:54:13.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-17T18:54:13.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-17T18:54:13.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-17T18:54:13.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-17T18:54:13.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-17T18:54:13.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-17T18:54:13.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-17T18:54:13.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-17T18:54:13.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-17T18:54:13.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-17T18:54:13.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-17T18:54:13.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-17T18:54:13.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-17T18:54:13.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-17T18:54:13.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-17T18:54:13.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-10-17T18:54:13.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-17T18:54:13.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-17T18:54:13.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-17T18:54:13.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-17T18:54:13.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-17T18:54:13.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-17T18:54:13.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-17T18:54:13.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-17T18:54:13.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-17T18:54:13.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-17T18:54:13.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-17T18:54:13.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-17T18:54:13.019 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.020 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.020 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.021 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.021 DEBUG:boto:Method: GET 2023-10-17T18:54:13.021 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.022 DEBUG:boto:Data: 2023-10-17T18:54:13.022 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.023 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.023 DEBUG:boto:Params: {} 2023-10-17T18:54:13.024 DEBUG:boto:Token: None 2023-10-17T18:54:13.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/?versions 2023-10-17T18:54:13.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NJpst0vH9JJYpAUh+eH57jI2m+w= 2023-10-17T18:54:13.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NJpst0vH9JJYpAUh+eH57jI2m+w=', 'Content-Length': '0'} 2023-10-17T18:54:13.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a6bd6f832419a4-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.029 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.031 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.032 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.032 DEBUG:boto:path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.033 DEBUG:boto:auth_path=/hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.033 DEBUG:boto:Method: GET 2023-10-17T18:54:13.033 DEBUG:boto:Path: /hbmuog-58/?versions&max-keys=999 2023-10-17T18:54:13.034 DEBUG:boto:Data: 2023-10-17T18:54:13.034 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.035 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.035 DEBUG:boto:Params: {} 2023-10-17T18:54:13.036 DEBUG:boto:Token: None 2023-10-17T18:54:13.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/?versions 2023-10-17T18:54:13.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NJpst0vH9JJYpAUh+eH57jI2m+w= 2023-10-17T18:54:13.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NJpst0vH9JJYpAUh+eH57jI2m+w=', 'Content-Length': '0'} 2023-10-17T18:54:13.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de69d74c73dda25-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.042 DEBUG:boto:b'hbmuog-58999falseobj00000000000000000000nulltrue2023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:54:13.046 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.047 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.047 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.047 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.048 DEBUG:boto:Method: GET 2023-10-17T18:54:13.048 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.049 DEBUG:boto:Data: 2023-10-17T18:54:13.049 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.049 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.050 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.050 DEBUG:boto:Params: {} 2023-10-17T18:54:13.051 DEBUG:boto:Token: None 2023-10-17T18:54:13.051 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.052 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s= 2023-10-17T18:54:13.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s=', 'Content-Length': '0'} 2023-10-17T18:54:13.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db37c41ee781131-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.055 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.056 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.056 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.057 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.057 DEBUG:boto:Method: GET 2023-10-17T18:54:13.058 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.058 DEBUG:boto:Data: 2023-10-17T18:54:13.058 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.059 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.059 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.060 DEBUG:boto:Params: {} 2023-10-17T18:54:13.060 DEBUG:boto:Token: None 2023-10-17T18:54:13.061 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.061 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s= 2023-10-17T18:54:13.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s=', 'Content-Length': '0'} 2023-10-17T18:54:13.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003528666c8a17daaf-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.064 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.065 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.065 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.066 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.066 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.066 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.067 DEBUG:boto:Data: 2023-10-17T18:54:13.067 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.068 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.068 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.068 DEBUG:boto:Params: {} 2023-10-17T18:54:13.069 DEBUG:boto:Token: None 2023-10-17T18:54:13.069 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.070 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HRUjYnBRNJMF4w67tlsmZ2sPdNs= 2023-10-17T18:54:13.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HRUjYnBRNJMF4w67tlsmZ2sPdNs=', 'Content-Length': '0'} 2023-10-17T18:54:13.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009767497d9e78dd9e-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.073 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.073 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.074 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.074 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.074 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.075 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.075 DEBUG:boto:Data: 2023-10-17T18:54:13.076 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.076 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.077 DEBUG:boto:Params: {} 2023-10-17T18:54:13.077 DEBUG:boto:Token: None 2023-10-17T18:54:13.078 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HRUjYnBRNJMF4w67tlsmZ2sPdNs= 2023-10-17T18:54:13.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HRUjYnBRNJMF4w67tlsmZ2sPdNs=', 'Content-Length': '0'} 2023-10-17T18:54:13.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000041e408d2bc715a1-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:54:13.082 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.082 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.083 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.083 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.084 DEBUG:boto:Method: GET 2023-10-17T18:54:13.084 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.084 DEBUG:boto:Data: 2023-10-17T18:54:13.085 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.085 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.086 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.086 DEBUG:boto:Params: {} 2023-10-17T18:54:13.087 DEBUG:boto:Token: None 2023-10-17T18:54:13.087 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.087 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s= 2023-10-17T18:54:13.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s=', 'Content-Length': '0'} 2023-10-17T18:54:13.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c5229a77b362dd-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.130 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.131 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.131 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.131 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.132 DEBUG:boto:Method: GET 2023-10-17T18:54:13.132 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.133 DEBUG:boto:Data: 2023-10-17T18:54:13.133 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.133 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.134 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.134 DEBUG:boto:Params: {} 2023-10-17T18:54:13.135 DEBUG:boto:Token: None 2023-10-17T18:54:13.135 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000?versionId=null 2023-10-17T18:54:13.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s= 2023-10-17T18:54:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FkSo7K0JbBF8wz1957nOE9mlR3s=', 'Content-Length': '0'} 2023-10-17T18:54:13.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093efa098e2751a83-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.139 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.139 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.140 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.140 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.141 DEBUG:boto:Method: GET 2023-10-17T18:54:13.141 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.141 DEBUG:boto:Data: 2023-10-17T18:54:13.142 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.142 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.143 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.143 DEBUG:boto:Params: {} 2023-10-17T18:54:13.143 DEBUG:boto:Token: None 2023-10-17T18:54:13.144 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.144 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511ecd51b5f967dc-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.148 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.149 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.149 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.149 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.150 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.150 DEBUG:boto:Method: GET 2023-10-17T18:54:13.151 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.151 DEBUG:boto:Data: 2023-10-17T18:54:13.151 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.152 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.153 DEBUG:boto:Params: {} 2023-10-17T18:54:13.153 DEBUG:boto:Token: None 2023-10-17T18:54:13.154 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3be1cc41daf937-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.159 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.160 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.160 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.161 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.161 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.161 DEBUG:boto:Data: 2023-10-17T18:54:13.162 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.162 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.163 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.163 DEBUG:boto:Params: {} 2023-10-17T18:54:13.164 DEBUG:boto:Token: None 2023-10-17T18:54:13.164 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fx+x0pa5ZpIth9nPiJQ9kC0EWDM= 2023-10-17T18:54:13.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fx+x0pa5ZpIth9nPiJQ9kC0EWDM=', 'Content-Length': '0'} 2023-10-17T18:54:13.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006703c2a833b3ee78-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.167 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.168 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.168 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.169 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.169 DEBUG:boto:Data: 2023-10-17T18:54:13.169 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.170 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.171 DEBUG:boto:Params: {} 2023-10-17T18:54:13.171 DEBUG:boto:Token: None 2023-10-17T18:54:13.172 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fx+x0pa5ZpIth9nPiJQ9kC0EWDM= 2023-10-17T18:54:13.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fx+x0pa5ZpIth9nPiJQ9kC0EWDM=', 'Content-Length': '0'} 2023-10-17T18:54:13.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41774248f3a0c8f-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-17T18:54:13.175 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.176 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.176 DEBUG:boto:Method: GET 2023-10-17T18:54:13.177 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.177 DEBUG:boto:Data: 2023-10-17T18:54:13.178 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.178 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.179 DEBUG:boto:Params: {} 2023-10-17T18:54:13.179 DEBUG:boto:Token: None 2023-10-17T18:54:13.180 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oPfQIPE5p6PozIuamzyVJMkGkUU= 2023-10-17T18:54:13.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oPfQIPE5p6PozIuamzyVJMkGkUU=', 'Content-Length': '0'} 2023-10-17T18:54:13.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a3308dc504b5c1-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.226 DEBUG:boto:path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.227 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.227 DEBUG:boto:Method: GET 2023-10-17T18:54:13.228 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.228 DEBUG:boto:Data: 2023-10-17T18:54:13.229 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.229 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.229 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.230 DEBUG:boto:Params: {} 2023-10-17T18:54:13.230 DEBUG:boto:Token: None 2023-10-17T18:54:13.231 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000000 2023-10-17T18:54:13.231 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oPfQIPE5p6PozIuamzyVJMkGkUU= 2023-10-17T18:54:13.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oPfQIPE5p6PozIuamzyVJMkGkUU=', 'Content-Length': '0'} 2023-10-17T18:54:13.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dbbfa0949498365-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.234 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.235 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.235 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.236 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.236 DEBUG:boto:Method: GET 2023-10-17T18:54:13.236 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.237 DEBUG:boto:Data: 2023-10-17T18:54:13.237 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.238 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.238 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.238 DEBUG:boto:Params: {} 2023-10-17T18:54:13.239 DEBUG:boto:Token: None 2023-10-17T18:54:13.239 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.240 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad3e9a371d4072e-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.243 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.244 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.245 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.245 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.245 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.246 DEBUG:boto:Method: GET 2023-10-17T18:54:13.246 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000000 2023-10-17T18:54:13.247 DEBUG:boto:Data: 2023-10-17T18:54:13.247 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.248 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.248 DEBUG:boto:Params: {} 2023-10-17T18:54:13.249 DEBUG:boto:Token: None 2023-10-17T18:54:13.249 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.250 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636bd30b6a2b5066-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.255 DEBUG:boto:b'hbmuog-58obj000000000000000000001000falseobj000000000000000000002023-10-17T18:54:08.161Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:54:13.256 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.257 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.257 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.258 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.258 DEBUG:boto:Method: GET 2023-10-17T18:54:13.259 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.259 DEBUG:boto:Data: 2023-10-17T18:54:13.260 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.260 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.261 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.261 DEBUG:boto:Params: {} 2023-10-17T18:54:13.262 DEBUG:boto:Token: None 2023-10-17T18:54:13.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o= 2023-10-17T18:54:13.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o=', 'Content-Length': '0'} 2023-10-17T18:54:13.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1709a8b29babef-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.266 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.266 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.267 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.267 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.267 DEBUG:boto:Method: GET 2023-10-17T18:54:13.268 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.268 DEBUG:boto:Data: 2023-10-17T18:54:13.269 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.269 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.269 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.270 DEBUG:boto:Params: {} 2023-10-17T18:54:13.270 DEBUG:boto:Token: None 2023-10-17T18:54:13.271 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.271 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o= 2023-10-17T18:54:13.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o=', 'Content-Length': '0'} 2023-10-17T18:54:13.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcce1a348b0a880-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.274 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.275 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.275 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.276 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.276 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.277 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.277 DEBUG:boto:Data: 2023-10-17T18:54:13.277 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.278 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.279 DEBUG:boto:Params: {} 2023-10-17T18:54:13.279 DEBUG:boto:Token: None 2023-10-17T18:54:13.280 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G/W89MNpE1TSkexfAZDUeNXHVS0= 2023-10-17T18:54:13.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G/W89MNpE1TSkexfAZDUeNXHVS0=', 'Content-Length': '0'} 2023-10-17T18:54:13.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61d14a9e5fe73aa-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.283 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.283 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.284 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.284 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.285 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.285 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.285 DEBUG:boto:Data: 2023-10-17T18:54:13.286 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.287 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.287 DEBUG:boto:Params: {} 2023-10-17T18:54:13.287 DEBUG:boto:Token: None 2023-10-17T18:54:13.288 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:G/W89MNpE1TSkexfAZDUeNXHVS0= 2023-10-17T18:54:13.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:G/W89MNpE1TSkexfAZDUeNXHVS0=', 'Content-Length': '0'} 2023-10-17T18:54:13.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c6107cc3b385da-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:54:13.292 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.292 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.293 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.293 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.293 DEBUG:boto:Method: GET 2023-10-17T18:54:13.294 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.294 DEBUG:boto:Data: 2023-10-17T18:54:13.295 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.295 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.296 DEBUG:boto:Params: {} 2023-10-17T18:54:13.296 DEBUG:boto:Token: None 2023-10-17T18:54:13.297 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o= 2023-10-17T18:54:13.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o=', 'Content-Length': '0'} 2023-10-17T18:54:13.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb5774a767fd42dd-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.342 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.343 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.343 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.344 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.344 DEBUG:boto:Method: GET 2023-10-17T18:54:13.345 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.345 DEBUG:boto:Data: 2023-10-17T18:54:13.345 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.346 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.347 DEBUG:boto:Params: {} 2023-10-17T18:54:13.347 DEBUG:boto:Token: None 2023-10-17T18:54:13.348 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001?versionId=null 2023-10-17T18:54:13.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o= 2023-10-17T18:54:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkiZijtuGeq3QCbdk54V7S0no9o=', 'Content-Length': '0'} 2023-10-17T18:54:13.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de32f31e4d2656b9-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.352 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.352 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.353 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.353 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.354 DEBUG:boto:Method: GET 2023-10-17T18:54:13.354 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.355 DEBUG:boto:Data: 2023-10-17T18:54:13.355 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.355 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.356 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.356 DEBUG:boto:Params: {} 2023-10-17T18:54:13.357 DEBUG:boto:Token: None 2023-10-17T18:54:13.357 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.358 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fef08c7c49702920-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.361 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.362 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.363 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.363 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.364 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.364 DEBUG:boto:Method: GET 2023-10-17T18:54:13.364 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.365 DEBUG:boto:Data: 2023-10-17T18:54:13.365 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.366 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.366 DEBUG:boto:Params: {} 2023-10-17T18:54:13.367 DEBUG:boto:Token: None 2023-10-17T18:54:13.367 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002069e79c10226ba3-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.373 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.374 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.375 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.375 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.375 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.376 DEBUG:boto:Data: 2023-10-17T18:54:13.376 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.377 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.377 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.378 DEBUG:boto:Params: {} 2023-10-17T18:54:13.378 DEBUG:boto:Token: None 2023-10-17T18:54:13.378 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.379 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVHVeMoFRLoNsy7i8kGhcX9nWfk= 2023-10-17T18:54:13.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVHVeMoFRLoNsy7i8kGhcX9nWfk=', 'Content-Length': '0'} 2023-10-17T18:54:13.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3c9e4a3887a9bd-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.382 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.382 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.383 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.383 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.384 DEBUG:boto:Data: 2023-10-17T18:54:13.384 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.385 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.385 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.385 DEBUG:boto:Params: {} 2023-10-17T18:54:13.386 DEBUG:boto:Token: None 2023-10-17T18:54:13.386 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVHVeMoFRLoNsy7i8kGhcX9nWfk= 2023-10-17T18:54:13.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVHVeMoFRLoNsy7i8kGhcX9nWfk=', 'Content-Length': '0'} 2023-10-17T18:54:13.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34bfeb0174e6390-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-17T18:54:13.390 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.391 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.391 DEBUG:boto:Method: GET 2023-10-17T18:54:13.391 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.392 DEBUG:boto:Data: 2023-10-17T18:54:13.392 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.393 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.393 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.393 DEBUG:boto:Params: {} 2023-10-17T18:54:13.394 DEBUG:boto:Token: None 2023-10-17T18:54:13.394 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.395 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GSjqNl0K5jq7oMEhDTYw3e7d3P8= 2023-10-17T18:54:13.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GSjqNl0K5jq7oMEhDTYw3e7d3P8=', 'Content-Length': '0'} 2023-10-17T18:54:13.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085dc399a6324c95c-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.398 DEBUG:boto:path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.398 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.398 DEBUG:boto:Method: GET 2023-10-17T18:54:13.399 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.399 DEBUG:boto:Data: 2023-10-17T18:54:13.401 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.402 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.402 DEBUG:boto:Params: {} 2023-10-17T18:54:13.403 DEBUG:boto:Token: None 2023-10-17T18:54:13.403 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000001 2023-10-17T18:54:13.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GSjqNl0K5jq7oMEhDTYw3e7d3P8= 2023-10-17T18:54:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GSjqNl0K5jq7oMEhDTYw3e7d3P8=', 'Content-Length': '0'} 2023-10-17T18:54:13.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd5ae5883fd65ef-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.450 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.451 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.451 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.452 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.452 DEBUG:boto:Method: GET 2023-10-17T18:54:13.453 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.453 DEBUG:boto:Data: 2023-10-17T18:54:13.454 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.455 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.455 DEBUG:boto:Params: {} 2023-10-17T18:54:13.456 DEBUG:boto:Token: None 2023-10-17T18:54:13.456 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f1f8ea5c51ee19e-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.460 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.461 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.462 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.462 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.463 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.463 DEBUG:boto:Method: GET 2023-10-17T18:54:13.463 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000001 2023-10-17T18:54:13.464 DEBUG:boto:Data: 2023-10-17T18:54:13.464 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.465 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.466 DEBUG:boto:Params: {} 2023-10-17T18:54:13.466 DEBUG:boto:Token: None 2023-10-17T18:54:13.467 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.467 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b6de97b0c58dd7-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.473 DEBUG:boto:b'hbmuog-58obj000000000000000000011000falseobj000000000000000000012023-10-17T18:54:08.323Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:54:13.474 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.474 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.475 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.475 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.476 DEBUG:boto:Method: GET 2023-10-17T18:54:13.476 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.476 DEBUG:boto:Data: 2023-10-17T18:54:13.477 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.478 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.478 DEBUG:boto:Params: {} 2023-10-17T18:54:13.479 DEBUG:boto:Token: None 2023-10-17T18:54:13.479 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo= 2023-10-17T18:54:13.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo=', 'Content-Length': '0'} 2023-10-17T18:54:13.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d715d97e5aa0072d-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.484 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.484 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.484 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.485 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.485 DEBUG:boto:Method: GET 2023-10-17T18:54:13.486 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.486 DEBUG:boto:Data: 2023-10-17T18:54:13.486 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.487 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.488 DEBUG:boto:Params: {} 2023-10-17T18:54:13.488 DEBUG:boto:Token: None 2023-10-17T18:54:13.489 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo= 2023-10-17T18:54:13.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo=', 'Content-Length': '0'} 2023-10-17T18:54:13.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ba231605c5209e-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.493 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.493 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.493 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.494 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.494 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.495 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.495 DEBUG:boto:Data: 2023-10-17T18:54:13.496 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.496 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.496 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.497 DEBUG:boto:Params: {} 2023-10-17T18:54:13.497 DEBUG:boto:Token: None 2023-10-17T18:54:13.498 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.498 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jLV0hF5kFKCR8F2dOpxXTCefCDQ= 2023-10-17T18:54:13.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jLV0hF5kFKCR8F2dOpxXTCefCDQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004045e664591b0d6d-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.501 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.502 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.502 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.502 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.503 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.503 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.504 DEBUG:boto:Data: 2023-10-17T18:54:13.504 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.505 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.505 DEBUG:boto:Params: {} 2023-10-17T18:54:13.506 DEBUG:boto:Token: None 2023-10-17T18:54:13.506 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jLV0hF5kFKCR8F2dOpxXTCefCDQ= 2023-10-17T18:54:13.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jLV0hF5kFKCR8F2dOpxXTCefCDQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcbff298c8a33a06-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:54:13.510 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.510 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.511 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.511 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.512 DEBUG:boto:Method: GET 2023-10-17T18:54:13.512 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.512 DEBUG:boto:Data: 2023-10-17T18:54:13.513 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.513 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.514 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.514 DEBUG:boto:Params: {} 2023-10-17T18:54:13.515 DEBUG:boto:Token: None 2023-10-17T18:54:13.515 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.516 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo= 2023-10-17T18:54:13.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo=', 'Content-Length': '0'} 2023-10-17T18:54:13.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab70668213119d3-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.519 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.519 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.520 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.520 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.521 DEBUG:boto:Method: GET 2023-10-17T18:54:13.521 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.522 DEBUG:boto:Data: 2023-10-17T18:54:13.522 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.523 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.523 DEBUG:boto:Params: {} 2023-10-17T18:54:13.524 DEBUG:boto:Token: None 2023-10-17T18:54:13.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002?versionId=null 2023-10-17T18:54:13.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo= 2023-10-17T18:54:13.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zqsq81bgCH8VoXh21oYLUsSeZpo=', 'Content-Length': '0'} 2023-10-17T18:54:13.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5950a0461384c4f-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.528 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.528 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.529 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.529 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.529 DEBUG:boto:Method: GET 2023-10-17T18:54:13.530 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.530 DEBUG:boto:Data: 2023-10-17T18:54:13.531 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.532 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.532 DEBUG:boto:Params: {} 2023-10-17T18:54:13.532 DEBUG:boto:Token: None 2023-10-17T18:54:13.533 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c7915653100511-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.578 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.579 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.580 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.580 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.581 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.581 DEBUG:boto:Method: GET 2023-10-17T18:54:13.582 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.582 DEBUG:boto:Data: 2023-10-17T18:54:13.583 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.583 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.583 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.584 DEBUG:boto:Params: {} 2023-10-17T18:54:13.584 DEBUG:boto:Token: None 2023-10-17T18:54:13.585 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.585 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f68e6303a1d7388c-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.591 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.592 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.592 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.593 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.593 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.594 DEBUG:boto:Data: 2023-10-17T18:54:13.594 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.594 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.595 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.595 DEBUG:boto:Params: {} 2023-10-17T18:54:13.596 DEBUG:boto:Token: None 2023-10-17T18:54:13.596 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y9ltnAUGIL/1Tt5sxb5kCy672Dc= 2023-10-17T18:54:13.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y9ltnAUGIL/1Tt5sxb5kCy672Dc=', 'Content-Length': '0'} 2023-10-17T18:54:13.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddead35c500ce25b-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.600 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.600 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.601 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.601 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.601 DEBUG:boto:Data: 2023-10-17T18:54:13.602 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.602 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.603 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.603 DEBUG:boto:Params: {} 2023-10-17T18:54:13.604 DEBUG:boto:Token: None 2023-10-17T18:54:13.604 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.605 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y9ltnAUGIL/1Tt5sxb5kCy672Dc= 2023-10-17T18:54:13.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y9ltnAUGIL/1Tt5sxb5kCy672Dc=', 'Content-Length': '0'} 2023-10-17T18:54:13.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658f547133a5ec14-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-17T18:54:13.608 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.608 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.609 DEBUG:boto:Method: GET 2023-10-17T18:54:13.609 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.610 DEBUG:boto:Data: 2023-10-17T18:54:13.610 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.611 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.611 DEBUG:boto:Params: {} 2023-10-17T18:54:13.612 DEBUG:boto:Token: None 2023-10-17T18:54:13.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lxba3RaIJhc7eqLi9pru7OwOaGM= 2023-10-17T18:54:13.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lxba3RaIJhc7eqLi9pru7OwOaGM=', 'Content-Length': '0'} 2023-10-17T18:54:13.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089822232d72012af-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.616 DEBUG:boto:path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.616 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.617 DEBUG:boto:Method: GET 2023-10-17T18:54:13.617 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.617 DEBUG:boto:Data: 2023-10-17T18:54:13.618 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.619 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.619 DEBUG:boto:Params: {} 2023-10-17T18:54:13.620 DEBUG:boto:Token: None 2023-10-17T18:54:13.620 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000002 2023-10-17T18:54:13.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lxba3RaIJhc7eqLi9pru7OwOaGM= 2023-10-17T18:54:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lxba3RaIJhc7eqLi9pru7OwOaGM=', 'Content-Length': '0'} 2023-10-17T18:54:13.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f450e6b585755db4-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.666 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.667 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.667 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.668 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.668 DEBUG:boto:Method: GET 2023-10-17T18:54:13.668 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.669 DEBUG:boto:Data: 2023-10-17T18:54:13.669 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.670 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.670 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.671 DEBUG:boto:Params: {} 2023-10-17T18:54:13.671 DEBUG:boto:Token: None 2023-10-17T18:54:13.672 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.672 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005176e4f5e3896b44-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.676 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.676 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.677 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.677 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.678 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.678 DEBUG:boto:Method: GET 2023-10-17T18:54:13.678 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000002 2023-10-17T18:54:13.679 DEBUG:boto:Data: 2023-10-17T18:54:13.679 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.680 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.680 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.681 DEBUG:boto:Params: {} 2023-10-17T18:54:13.681 DEBUG:boto:Token: None 2023-10-17T18:54:13.682 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfea2c480e5d5ca0-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.688 DEBUG:boto:b'hbmuog-58obj000000000000000000021000falseobj000000000000000000022023-10-17T18:54:07.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:54:13.689 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.689 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.690 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.690 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.691 DEBUG:boto:Method: GET 2023-10-17T18:54:13.691 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.691 DEBUG:boto:Data: 2023-10-17T18:54:13.692 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.693 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.693 DEBUG:boto:Params: {} 2023-10-17T18:54:13.694 DEBUG:boto:Token: None 2023-10-17T18:54:13.694 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.695 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ= 2023-10-17T18:54:13.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60d688cfc642e47-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.738 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.739 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.739 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.740 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.740 DEBUG:boto:Method: GET 2023-10-17T18:54:13.740 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.741 DEBUG:boto:Data: 2023-10-17T18:54:13.741 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.742 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.743 DEBUG:boto:Params: {} 2023-10-17T18:54:13.743 DEBUG:boto:Token: None 2023-10-17T18:54:13.744 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ= 2023-10-17T18:54:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0abcb02fa9ee71-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.747 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.748 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.748 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.749 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.749 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.749 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.750 DEBUG:boto:Data: 2023-10-17T18:54:13.750 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.751 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.751 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.752 DEBUG:boto:Params: {} 2023-10-17T18:54:13.752 DEBUG:boto:Token: None 2023-10-17T18:54:13.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ao7z4qkWJ4Y8X/rbZso5bKgyY7Y= 2023-10-17T18:54:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ao7z4qkWJ4Y8X/rbZso5bKgyY7Y=', 'Content-Length': '0'} 2023-10-17T18:54:13.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49e07a9139c784e-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.756 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.756 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.757 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.757 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.757 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.758 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.758 DEBUG:boto:Data: 2023-10-17T18:54:13.759 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.759 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.760 DEBUG:boto:Params: {} 2023-10-17T18:54:13.760 DEBUG:boto:Token: None 2023-10-17T18:54:13.761 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ao7z4qkWJ4Y8X/rbZso5bKgyY7Y= 2023-10-17T18:54:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ao7z4qkWJ4Y8X/rbZso5bKgyY7Y=', 'Content-Length': '0'} 2023-10-17T18:54:13.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5afcb7439f0d8da-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:54:13.765 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.765 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.766 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.766 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.766 DEBUG:boto:Method: GET 2023-10-17T18:54:13.767 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.767 DEBUG:boto:Data: 2023-10-17T18:54:13.768 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.769 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.769 DEBUG:boto:Params: {} 2023-10-17T18:54:13.770 DEBUG:boto:Token: None 2023-10-17T18:54:13.770 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.771 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ= 2023-10-17T18:54:13.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034d03129cd576d9c-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.774 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.774 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.775 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.775 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.776 DEBUG:boto:Method: GET 2023-10-17T18:54:13.776 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.776 DEBUG:boto:Data: 2023-10-17T18:54:13.777 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.777 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.778 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.778 DEBUG:boto:Params: {} 2023-10-17T18:54:13.779 DEBUG:boto:Token: None 2023-10-17T18:54:13.779 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003?versionId=null 2023-10-17T18:54:13.780 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ= 2023-10-17T18:54:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N3Lq7NvHFwmLEsCx6tqm3KDuMrQ=', 'Content-Length': '0'} 2023-10-17T18:54:13.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e6de7da496a97fe-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.783 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.783 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.784 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.784 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.784 DEBUG:boto:Method: GET 2023-10-17T18:54:13.785 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.785 DEBUG:boto:Data: 2023-10-17T18:54:13.786 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.786 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.787 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.787 DEBUG:boto:Params: {} 2023-10-17T18:54:13.788 DEBUG:boto:Token: None 2023-10-17T18:54:13.788 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.789 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dc92fe04fa981a9-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.792 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.793 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.793 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.794 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.794 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.794 DEBUG:boto:Method: GET 2023-10-17T18:54:13.795 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.795 DEBUG:boto:Data: 2023-10-17T18:54:13.796 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.797 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.797 DEBUG:boto:Params: {} 2023-10-17T18:54:13.797 DEBUG:boto:Token: None 2023-10-17T18:54:13.798 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065469b3667d72fbb-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.850 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.851 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.852 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.852 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.852 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.853 DEBUG:boto:Data: 2023-10-17T18:54:13.853 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.854 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.854 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.855 DEBUG:boto:Params: {} 2023-10-17T18:54:13.855 DEBUG:boto:Token: None 2023-10-17T18:54:13.855 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OjV1FqEYz5dr0tgiFQlESZkR4/w= 2023-10-17T18:54:13.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OjV1FqEYz5dr0tgiFQlESZkR4/w=', 'Content-Length': '0'} 2023-10-17T18:54:13.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0faee547081992f-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.860 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.860 DEBUG:boto:Method: HEAD 2023-10-17T18:54:13.861 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.861 DEBUG:boto:Data: 2023-10-17T18:54:13.861 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.862 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.863 DEBUG:boto:Params: {} 2023-10-17T18:54:13.863 DEBUG:boto:Token: None 2023-10-17T18:54:13.864 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OjV1FqEYz5dr0tgiFQlESZkR4/w= 2023-10-17T18:54:13.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OjV1FqEYz5dr0tgiFQlESZkR4/w=', 'Content-Length': '0'} 2023-10-17T18:54:13.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b7a20e650258f17-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-17T18:54:13.867 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.868 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.868 DEBUG:boto:Method: GET 2023-10-17T18:54:13.869 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.869 DEBUG:boto:Data: 2023-10-17T18:54:13.870 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.870 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.871 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.871 DEBUG:boto:Params: {} 2023-10-17T18:54:13.871 DEBUG:boto:Token: None 2023-10-17T18:54:13.872 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.872 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4mACrcFOFzbWduJfSLwDvZBRPOA= 2023-10-17T18:54:13.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4mACrcFOFzbWduJfSLwDvZBRPOA=', 'Content-Length': '0'} 2023-10-17T18:54:13.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058fb1f5339a02a81-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.876 DEBUG:boto:path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.876 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.876 DEBUG:boto:Method: GET 2023-10-17T18:54:13.877 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.877 DEBUG:boto:Data: 2023-10-17T18:54:13.878 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.879 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.879 DEBUG:boto:Params: {} 2023-10-17T18:54:13.880 DEBUG:boto:Token: None 2023-10-17T18:54:13.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000003 2023-10-17T18:54:13.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4mACrcFOFzbWduJfSLwDvZBRPOA= 2023-10-17T18:54:13.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4mACrcFOFzbWduJfSLwDvZBRPOA=', 'Content-Length': '0'} 2023-10-17T18:54:13.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8bc57afc756475-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.884 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.884 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.885 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.885 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.885 DEBUG:boto:Method: GET 2023-10-17T18:54:13.886 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.886 DEBUG:boto:Data: 2023-10-17T18:54:13.887 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.887 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.888 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.888 DEBUG:boto:Params: {} 2023-10-17T18:54:13.889 DEBUG:boto:Token: None 2023-10-17T18:54:13.889 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014652f9dac9db298-00652ed855-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.938 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.939 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:13.940 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:13.940 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.941 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.941 DEBUG:boto:Method: GET 2023-10-17T18:54:13.942 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000003 2023-10-17T18:54:13.943 DEBUG:boto:Data: 2023-10-17T18:54:13.944 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.944 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.945 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.946 DEBUG:boto:Params: {} 2023-10-17T18:54:13.946 DEBUG:boto:Token: None 2023-10-17T18:54:13.947 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/ 2023-10-17T18:54:13.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA= 2023-10-17T18:54:13.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qspgnW/QzI7i5ugcBbcx328idnA=', 'Content-Length': '0'} 2023-10-17T18:54:13.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae2eb3643f6f99c-00652ed855-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.953 DEBUG:boto:b'hbmuog-58obj000000000000000000031000falseobj000000000000000000032023-10-17T18:54:07.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:13.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:54:13.954 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:13.955 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:13.955 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.956 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.956 DEBUG:boto:Method: GET 2023-10-17T18:54:13.957 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.957 DEBUG:boto:Data: 2023-10-17T18:54:13.957 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.958 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:13.958 DEBUG:boto:Port: 8001 2023-10-17T18:54:13.959 DEBUG:boto:Params: {} 2023-10-17T18:54:13.959 DEBUG:boto:Token: None 2023-10-17T18:54:13.960 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.960 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H5NeLUWT3MwsHhC5pJsuioqH0rY= 2023-10-17T18:54:13.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H5NeLUWT3MwsHhC5pJsuioqH0rY=', 'Content-Length': '0'} 2023-10-17T18:54:13.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000db56c37ccd618a-00652ed855-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:13.964 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:13.965 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:13.965 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.966 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.966 DEBUG:boto:Method: GET 2023-10-17T18:54:13.967 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.967 DEBUG:boto:Data: 2023-10-17T18:54:13.967 DEBUG:boto:Headers: {} 2023-10-17T18:54:13.968 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:13.968 DEBUG:boto:Port: 8000 2023-10-17T18:54:13.969 DEBUG:boto:Params: {} 2023-10-17T18:54:13.969 DEBUG:boto:Token: None 2023-10-17T18:54:13.970 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:13 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:13.970 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H5NeLUWT3MwsHhC5pJsuioqH0rY= 2023-10-17T18:54:13.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H5NeLUWT3MwsHhC5pJsuioqH0rY=', 'Content-Length': '0'} 2023-10-17T18:54:13.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0df93bf293ec889-00652ed855-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.014 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.015 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.015 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.016 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.016 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.016 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.017 DEBUG:boto:Data: 2023-10-17T18:54:14.017 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.018 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.019 DEBUG:boto:Params: {} 2023-10-17T18:54:14.019 DEBUG:boto:Token: None 2023-10-17T18:54:14.020 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cUU6vW3SBxKrTG3OFIx3+eYy9mo= 2023-10-17T18:54:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cUU6vW3SBxKrTG3OFIx3+eYy9mo=', 'Content-Length': '0'} 2023-10-17T18:54:14.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000007125ab86fbe82d-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.025 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.025 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.026 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.027 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.027 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.028 DEBUG:boto:Data: 2023-10-17T18:54:14.028 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.028 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.029 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.029 DEBUG:boto:Params: {} 2023-10-17T18:54:14.030 DEBUG:boto:Token: None 2023-10-17T18:54:14.030 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.031 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cUU6vW3SBxKrTG3OFIx3+eYy9mo= 2023-10-17T18:54:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cUU6vW3SBxKrTG3OFIx3+eYy9mo=', 'Content-Length': '0'} 2023-10-17T18:54:14.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c287022c51267cbd-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:54:14.035 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.036 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.036 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.037 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.037 DEBUG:boto:Method: GET 2023-10-17T18:54:14.038 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.039 DEBUG:boto:Data: 2023-10-17T18:54:14.039 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.041 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.041 DEBUG:boto:Params: {} 2023-10-17T18:54:14.042 DEBUG:boto:Token: None 2023-10-17T18:54:14.043 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.043 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PuzPAANR653kvqSsukenYITZDCE= 2023-10-17T18:54:14.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PuzPAANR653kvqSsukenYITZDCE=', 'Content-Length': '0'} 2023-10-17T18:54:14.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee50f94852fbef0b-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.047 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.047 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.047 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.048 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.048 DEBUG:boto:Method: GET 2023-10-17T18:54:14.049 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.049 DEBUG:boto:Data: 2023-10-17T18:54:14.049 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.050 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.050 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.051 DEBUG:boto:Params: {} 2023-10-17T18:54:14.052 DEBUG:boto:Token: None 2023-10-17T18:54:14.052 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004?versionId=null 2023-10-17T18:54:14.053 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PuzPAANR653kvqSsukenYITZDCE= 2023-10-17T18:54:14.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PuzPAANR653kvqSsukenYITZDCE=', 'Content-Length': '0'} 2023-10-17T18:54:14.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5594b0a7d65a7c-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.056 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.056 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.057 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.057 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.058 DEBUG:boto:Method: GET 2023-10-17T18:54:14.058 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.059 DEBUG:boto:Data: 2023-10-17T18:54:14.059 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.059 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.060 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.060 DEBUG:boto:Params: {} 2023-10-17T18:54:14.061 DEBUG:boto:Token: None 2023-10-17T18:54:14.061 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.062 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044fb048be3e4002c-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.065 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.066 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.066 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.067 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.067 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.068 DEBUG:boto:Method: GET 2023-10-17T18:54:14.068 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.069 DEBUG:boto:Data: 2023-10-17T18:54:14.069 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.069 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.070 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.070 DEBUG:boto:Params: {} 2023-10-17T18:54:14.071 DEBUG:boto:Token: None 2023-10-17T18:54:14.071 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009fdaa4db1792ecc-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.078 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.078 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.079 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.079 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.079 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.080 DEBUG:boto:Data: 2023-10-17T18:54:14.080 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.081 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.081 DEBUG:boto:Params: {} 2023-10-17T18:54:14.082 DEBUG:boto:Token: None 2023-10-17T18:54:14.082 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.083 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bPk/EgJuSec1GVY2JG4xF3+t//U= 2023-10-17T18:54:14.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bPk/EgJuSec1GVY2JG4xF3+t//U=', 'Content-Length': '0'} 2023-10-17T18:54:14.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb26f1f787166533-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.086 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.086 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.087 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.087 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.088 DEBUG:boto:Data: 2023-10-17T18:54:14.088 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.089 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.089 DEBUG:boto:Params: {} 2023-10-17T18:54:14.090 DEBUG:boto:Token: None 2023-10-17T18:54:14.090 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.091 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bPk/EgJuSec1GVY2JG4xF3+t//U= 2023-10-17T18:54:14.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bPk/EgJuSec1GVY2JG4xF3+t//U=', 'Content-Length': '0'} 2023-10-17T18:54:14.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ea4fb348865810-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-17T18:54:14.094 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.095 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.095 DEBUG:boto:Method: GET 2023-10-17T18:54:14.096 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.096 DEBUG:boto:Data: 2023-10-17T18:54:14.097 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.097 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.098 DEBUG:boto:Params: {} 2023-10-17T18:54:14.098 DEBUG:boto:Token: None 2023-10-17T18:54:14.099 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L6rsNUXCWqEXCx+mC9yxLUVVoKQ= 2023-10-17T18:54:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L6rsNUXCWqEXCx+mC9yxLUVVoKQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4271110d10a0c04-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.102 DEBUG:boto:path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.103 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.103 DEBUG:boto:Method: GET 2023-10-17T18:54:14.103 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.104 DEBUG:boto:Data: 2023-10-17T18:54:14.104 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.105 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.105 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.106 DEBUG:boto:Params: {} 2023-10-17T18:54:14.106 DEBUG:boto:Token: None 2023-10-17T18:54:14.107 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000004 2023-10-17T18:54:14.107 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L6rsNUXCWqEXCx+mC9yxLUVVoKQ= 2023-10-17T18:54:14.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L6rsNUXCWqEXCx+mC9yxLUVVoKQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009620bcc92613d2a2-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.150 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.151 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.151 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.152 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.152 DEBUG:boto:Method: GET 2023-10-17T18:54:14.153 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.153 DEBUG:boto:Data: 2023-10-17T18:54:14.154 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.154 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.155 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.155 DEBUG:boto:Params: {} 2023-10-17T18:54:14.155 DEBUG:boto:Token: None 2023-10-17T18:54:14.156 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.156 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae9d3bd0ef4aeba-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.160 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.161 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.161 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.162 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.162 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.162 DEBUG:boto:Method: GET 2023-10-17T18:54:14.163 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000004 2023-10-17T18:54:14.163 DEBUG:boto:Data: 2023-10-17T18:54:14.164 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.164 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.165 DEBUG:boto:Params: {} 2023-10-17T18:54:14.165 DEBUG:boto:Token: None 2023-10-17T18:54:14.166 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f70997f0d18a0e32-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.171 DEBUG:boto:b'hbmuog-58obj000000000000000000041000falseobj000000000000000000042023-10-17T18:54:08.544Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:54:14.173 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.173 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.174 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.174 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.174 DEBUG:boto:Method: GET 2023-10-17T18:54:14.175 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.175 DEBUG:boto:Data: 2023-10-17T18:54:14.176 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.176 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.177 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.177 DEBUG:boto:Params: {} 2023-10-17T18:54:14.178 DEBUG:boto:Token: None 2023-10-17T18:54:14.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.179 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU= 2023-10-17T18:54:14.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU=', 'Content-Length': '0'} 2023-10-17T18:54:14.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b50bbf395ae4a7-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.222 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.223 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.223 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.224 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.224 DEBUG:boto:Method: GET 2023-10-17T18:54:14.225 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.225 DEBUG:boto:Data: 2023-10-17T18:54:14.226 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.227 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.227 DEBUG:boto:Params: {} 2023-10-17T18:54:14.228 DEBUG:boto:Token: None 2023-10-17T18:54:14.228 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU= 2023-10-17T18:54:14.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU=', 'Content-Length': '0'} 2023-10-17T18:54:14.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fce1bcb09cf48ba-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.233 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.233 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.233 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.234 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.234 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.235 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.235 DEBUG:boto:Data: 2023-10-17T18:54:14.235 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.236 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.237 DEBUG:boto:Params: {} 2023-10-17T18:54:14.237 DEBUG:boto:Token: None 2023-10-17T18:54:14.238 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xpPnubYOPP1VV1o/t28epCkIklo= 2023-10-17T18:54:14.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xpPnubYOPP1VV1o/t28epCkIklo=', 'Content-Length': '0'} 2023-10-17T18:54:14.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000570a05e349ebc66-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.241 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.241 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.242 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.242 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.243 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.243 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.244 DEBUG:boto:Data: 2023-10-17T18:54:14.244 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.244 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.245 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.245 DEBUG:boto:Params: {} 2023-10-17T18:54:14.246 DEBUG:boto:Token: None 2023-10-17T18:54:14.246 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xpPnubYOPP1VV1o/t28epCkIklo= 2023-10-17T18:54:14.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xpPnubYOPP1VV1o/t28epCkIklo=', 'Content-Length': '0'} 2023-10-17T18:54:14.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b3042054dce163-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:54:14.250 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.250 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.251 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.251 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.252 DEBUG:boto:Method: GET 2023-10-17T18:54:14.252 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.253 DEBUG:boto:Data: 2023-10-17T18:54:14.253 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.254 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.254 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.254 DEBUG:boto:Params: {} 2023-10-17T18:54:14.255 DEBUG:boto:Token: None 2023-10-17T18:54:14.255 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.256 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU= 2023-10-17T18:54:14.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU=', 'Content-Length': '0'} 2023-10-17T18:54:14.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000397d4f9c3df6d701-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.260 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.260 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.262 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.263 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.263 DEBUG:boto:Method: GET 2023-10-17T18:54:14.264 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.264 DEBUG:boto:Data: 2023-10-17T18:54:14.264 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.265 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.266 DEBUG:boto:Params: {} 2023-10-17T18:54:14.266 DEBUG:boto:Token: None 2023-10-17T18:54:14.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005?versionId=null 2023-10-17T18:54:14.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU= 2023-10-17T18:54:14.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:InKmcFKDsb2w+oIT0MoE4Sb2viU=', 'Content-Length': '0'} 2023-10-17T18:54:14.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba8177602988820-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.270 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.271 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.271 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.271 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.272 DEBUG:boto:Method: GET 2023-10-17T18:54:14.272 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.273 DEBUG:boto:Data: 2023-10-17T18:54:14.273 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.274 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.274 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.275 DEBUG:boto:Params: {} 2023-10-17T18:54:14.275 DEBUG:boto:Token: None 2023-10-17T18:54:14.275 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.276 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb2a954564cfb27-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.280 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.281 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.281 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.282 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.282 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.283 DEBUG:boto:Method: GET 2023-10-17T18:54:14.283 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.284 DEBUG:boto:Data: 2023-10-17T18:54:14.284 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.284 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.285 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.285 DEBUG:boto:Params: {} 2023-10-17T18:54:14.286 DEBUG:boto:Token: None 2023-10-17T18:54:14.286 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.287 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4fb9ff4739a1730-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.292 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.292 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.293 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.293 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.294 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.294 DEBUG:boto:Data: 2023-10-17T18:54:14.295 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.295 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.296 DEBUG:boto:Params: {} 2023-10-17T18:54:14.296 DEBUG:boto:Token: None 2023-10-17T18:54:14.297 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ROiIrtsE1UuVSVoqr8UbB2ZZAoA= 2023-10-17T18:54:14.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ROiIrtsE1UuVSVoqr8UbB2ZZAoA=', 'Content-Length': '0'} 2023-10-17T18:54:14.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21543dafbe205df-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.300 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.300 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.301 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.301 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.302 DEBUG:boto:Data: 2023-10-17T18:54:14.302 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.303 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.303 DEBUG:boto:Params: {} 2023-10-17T18:54:14.304 DEBUG:boto:Token: None 2023-10-17T18:54:14.304 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ROiIrtsE1UuVSVoqr8UbB2ZZAoA= 2023-10-17T18:54:14.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ROiIrtsE1UuVSVoqr8UbB2ZZAoA=', 'Content-Length': '0'} 2023-10-17T18:54:14.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a40e66c2cd3eae-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-17T18:54:14.308 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.309 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.309 DEBUG:boto:Method: GET 2023-10-17T18:54:14.309 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.310 DEBUG:boto:Data: 2023-10-17T18:54:14.310 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.311 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.311 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.312 DEBUG:boto:Params: {} 2023-10-17T18:54:14.312 DEBUG:boto:Token: None 2023-10-17T18:54:14.313 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.313 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SOtWcLI6GQkpjjyGk3bnf+gcSjU= 2023-10-17T18:54:14.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SOtWcLI6GQkpjjyGk3bnf+gcSjU=', 'Content-Length': '0'} 2023-10-17T18:54:14.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bf308a871db340-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.358 DEBUG:boto:path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.359 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.360 DEBUG:boto:Method: GET 2023-10-17T18:54:14.360 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.360 DEBUG:boto:Data: 2023-10-17T18:54:14.361 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.361 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.362 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.362 DEBUG:boto:Params: {} 2023-10-17T18:54:14.363 DEBUG:boto:Token: None 2023-10-17T18:54:14.363 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000005 2023-10-17T18:54:14.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SOtWcLI6GQkpjjyGk3bnf+gcSjU= 2023-10-17T18:54:14.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SOtWcLI6GQkpjjyGk3bnf+gcSjU=', 'Content-Length': '0'} 2023-10-17T18:54:14.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605902bbe62267c4-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.367 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.368 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.368 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.368 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.369 DEBUG:boto:Method: GET 2023-10-17T18:54:14.369 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.370 DEBUG:boto:Data: 2023-10-17T18:54:14.370 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.371 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.371 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.371 DEBUG:boto:Params: {} 2023-10-17T18:54:14.372 DEBUG:boto:Token: None 2023-10-17T18:54:14.372 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.373 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022d1bb59637f59bc-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.377 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.377 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.378 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.378 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.379 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.379 DEBUG:boto:Method: GET 2023-10-17T18:54:14.380 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000005 2023-10-17T18:54:14.380 DEBUG:boto:Data: 2023-10-17T18:54:14.380 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.381 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.382 DEBUG:boto:Params: {} 2023-10-17T18:54:14.382 DEBUG:boto:Token: None 2023-10-17T18:54:14.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005140f201389dd13a-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.430 DEBUG:boto:b'hbmuog-58obj000000000000000000051000falseobj000000000000000000052023-10-17T18:54:07.755Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:54:14.432 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.432 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.433 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.433 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.434 DEBUG:boto:Method: GET 2023-10-17T18:54:14.434 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.434 DEBUG:boto:Data: 2023-10-17T18:54:14.435 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.435 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.436 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.436 DEBUG:boto:Params: {} 2023-10-17T18:54:14.437 DEBUG:boto:Token: None 2023-10-17T18:54:14.437 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.438 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo= 2023-10-17T18:54:14.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo=', 'Content-Length': '0'} 2023-10-17T18:54:14.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f041564ee233c6f-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.441 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.441 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.442 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.442 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.443 DEBUG:boto:Method: GET 2023-10-17T18:54:14.443 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.444 DEBUG:boto:Data: 2023-10-17T18:54:14.444 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.445 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.445 DEBUG:boto:Params: {} 2023-10-17T18:54:14.446 DEBUG:boto:Token: None 2023-10-17T18:54:14.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo= 2023-10-17T18:54:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo=', 'Content-Length': '0'} 2023-10-17T18:54:14.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2423accab3f0cc5-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.450 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.451 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.451 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.451 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.452 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.452 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.453 DEBUG:boto:Data: 2023-10-17T18:54:14.453 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.454 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.454 DEBUG:boto:Params: {} 2023-10-17T18:54:14.455 DEBUG:boto:Token: None 2023-10-17T18:54:14.455 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MnsE/GyTxdDsG1fnbhVrOeSRdCQ= 2023-10-17T18:54:14.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MnsE/GyTxdDsG1fnbhVrOeSRdCQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e562822e371ae8b8-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.459 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.459 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.460 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.460 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.461 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.461 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.461 DEBUG:boto:Data: 2023-10-17T18:54:14.462 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.463 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.463 DEBUG:boto:Params: {} 2023-10-17T18:54:14.463 DEBUG:boto:Token: None 2023-10-17T18:54:14.464 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MnsE/GyTxdDsG1fnbhVrOeSRdCQ= 2023-10-17T18:54:14.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MnsE/GyTxdDsG1fnbhVrOeSRdCQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009035656c0257f4ae-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:54:14.468 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.468 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.469 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.469 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.469 DEBUG:boto:Method: GET 2023-10-17T18:54:14.470 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.470 DEBUG:boto:Data: 2023-10-17T18:54:14.471 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.472 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.472 DEBUG:boto:Params: {} 2023-10-17T18:54:14.472 DEBUG:boto:Token: None 2023-10-17T18:54:14.473 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo= 2023-10-17T18:54:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo=', 'Content-Length': '0'} 2023-10-17T18:54:14.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f054691faa354a41-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.477 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.477 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.478 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.478 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.478 DEBUG:boto:Method: GET 2023-10-17T18:54:14.479 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.479 DEBUG:boto:Data: 2023-10-17T18:54:14.480 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.481 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.481 DEBUG:boto:Params: {} 2023-10-17T18:54:14.481 DEBUG:boto:Token: None 2023-10-17T18:54:14.482 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006?versionId=null 2023-10-17T18:54:14.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo= 2023-10-17T18:54:14.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4xcLL3NzggbZtXJkH2l/df15IHo=', 'Content-Length': '0'} 2023-10-17T18:54:14.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005daabfbff37c142c-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.485 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.486 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.486 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.487 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.487 DEBUG:boto:Method: GET 2023-10-17T18:54:14.488 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.488 DEBUG:boto:Data: 2023-10-17T18:54:14.488 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.489 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.490 DEBUG:boto:Params: {} 2023-10-17T18:54:14.490 DEBUG:boto:Token: None 2023-10-17T18:54:14.491 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ded61101f4bedce-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.495 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.495 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.496 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.496 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.497 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.497 DEBUG:boto:Method: GET 2023-10-17T18:54:14.498 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.498 DEBUG:boto:Data: 2023-10-17T18:54:14.498 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.499 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.500 DEBUG:boto:Params: {} 2023-10-17T18:54:14.500 DEBUG:boto:Token: None 2023-10-17T18:54:14.501 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037366909cd7b36d2-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.506 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.507 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.507 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.508 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.508 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.509 DEBUG:boto:Data: 2023-10-17T18:54:14.509 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.509 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.510 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.510 DEBUG:boto:Params: {} 2023-10-17T18:54:14.511 DEBUG:boto:Token: None 2023-10-17T18:54:14.511 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.512 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w/QdKxWzSyo1m8zvG0pM4aVD0+Y= 2023-10-17T18:54:14.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w/QdKxWzSyo1m8zvG0pM4aVD0+Y=', 'Content-Length': '0'} 2023-10-17T18:54:14.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee41cc990abb35c0-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.514 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.515 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.515 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.516 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.516 DEBUG:boto:Data: 2023-10-17T18:54:14.516 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.517 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.518 DEBUG:boto:Params: {} 2023-10-17T18:54:14.518 DEBUG:boto:Token: None 2023-10-17T18:54:14.519 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.519 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w/QdKxWzSyo1m8zvG0pM4aVD0+Y= 2023-10-17T18:54:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w/QdKxWzSyo1m8zvG0pM4aVD0+Y=', 'Content-Length': '0'} 2023-10-17T18:54:14.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006164c956ba2b22a5-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-17T18:54:14.523 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.523 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.523 DEBUG:boto:Method: GET 2023-10-17T18:54:14.524 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.524 DEBUG:boto:Data: 2023-10-17T18:54:14.525 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.525 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.526 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.526 DEBUG:boto:Params: {} 2023-10-17T18:54:14.526 DEBUG:boto:Token: None 2023-10-17T18:54:14.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.527 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c4HXBCwD4jfYPsW7l1jODNtP5BI= 2023-10-17T18:54:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c4HXBCwD4jfYPsW7l1jODNtP5BI=', 'Content-Length': '0'} 2023-10-17T18:54:14.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3cbdfdeacbb83c-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.574 DEBUG:boto:path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.575 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.575 DEBUG:boto:Method: GET 2023-10-17T18:54:14.576 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.576 DEBUG:boto:Data: 2023-10-17T18:54:14.577 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.577 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.578 DEBUG:boto:Params: {} 2023-10-17T18:54:14.578 DEBUG:boto:Token: None 2023-10-17T18:54:14.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000006 2023-10-17T18:54:14.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c4HXBCwD4jfYPsW7l1jODNtP5BI= 2023-10-17T18:54:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c4HXBCwD4jfYPsW7l1jODNtP5BI=', 'Content-Length': '0'} 2023-10-17T18:54:14.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024846ac5d0ea9657-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.582 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.583 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.583 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.584 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.584 DEBUG:boto:Method: GET 2023-10-17T18:54:14.585 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.585 DEBUG:boto:Data: 2023-10-17T18:54:14.585 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.586 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.587 DEBUG:boto:Params: {} 2023-10-17T18:54:14.587 DEBUG:boto:Token: None 2023-10-17T18:54:14.588 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840ec07338c0cd3d-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.592 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.593 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.593 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.594 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.594 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.594 DEBUG:boto:Method: GET 2023-10-17T18:54:14.595 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000006 2023-10-17T18:54:14.595 DEBUG:boto:Data: 2023-10-17T18:54:14.596 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.597 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.597 DEBUG:boto:Params: {} 2023-10-17T18:54:14.597 DEBUG:boto:Token: None 2023-10-17T18:54:14.598 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c62792998a753d-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.604 DEBUG:boto:b'hbmuog-58obj000000000000000000061000falseobj000000000000000000062023-10-17T18:54:08.484Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-17T18:54:14.605 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.605 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.606 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.606 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.607 DEBUG:boto:Method: GET 2023-10-17T18:54:14.607 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.607 DEBUG:boto:Data: 2023-10-17T18:54:14.608 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.608 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.609 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.609 DEBUG:boto:Params: {} 2023-10-17T18:54:14.610 DEBUG:boto:Token: None 2023-10-17T18:54:14.610 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.611 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88= 2023-10-17T18:54:14.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88=', 'Content-Length': '0'} 2023-10-17T18:54:14.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb277be7ce08e82-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.614 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.614 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.615 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.615 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.616 DEBUG:boto:Method: GET 2023-10-17T18:54:14.616 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.616 DEBUG:boto:Data: 2023-10-17T18:54:14.617 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.618 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.618 DEBUG:boto:Params: {} 2023-10-17T18:54:14.619 DEBUG:boto:Token: None 2023-10-17T18:54:14.619 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88= 2023-10-17T18:54:14.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88=', 'Content-Length': '0'} 2023-10-17T18:54:14.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8461e6585d573a2-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.662 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.663 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.663 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.664 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.664 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.665 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.665 DEBUG:boto:Data: 2023-10-17T18:54:14.665 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.666 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.667 DEBUG:boto:Params: {} 2023-10-17T18:54:14.667 DEBUG:boto:Token: None 2023-10-17T18:54:14.668 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.668 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:30lbE2TfG7Tq4QFUwiQLEZX5Cgs= 2023-10-17T18:54:14.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:30lbE2TfG7Tq4QFUwiQLEZX5Cgs=', 'Content-Length': '0'} 2023-10-17T18:54:14.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b082bd516cae304d-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.671 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.672 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.672 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.672 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.673 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.673 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.674 DEBUG:boto:Data: 2023-10-17T18:54:14.674 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.675 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.675 DEBUG:boto:Params: {} 2023-10-17T18:54:14.676 DEBUG:boto:Token: None 2023-10-17T18:54:14.676 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.677 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:30lbE2TfG7Tq4QFUwiQLEZX5Cgs= 2023-10-17T18:54:14.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:30lbE2TfG7Tq4QFUwiQLEZX5Cgs=', 'Content-Length': '0'} 2023-10-17T18:54:14.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6016271f1f6c55-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-17T18:54:14.680 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.681 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.681 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.681 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.682 DEBUG:boto:Method: GET 2023-10-17T18:54:14.682 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.683 DEBUG:boto:Data: 2023-10-17T18:54:14.683 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.684 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.684 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.684 DEBUG:boto:Params: {} 2023-10-17T18:54:14.685 DEBUG:boto:Token: None 2023-10-17T18:54:14.685 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88= 2023-10-17T18:54:14.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88=', 'Content-Length': '0'} 2023-10-17T18:54:14.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe3cadfe8ac065b-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.730 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.731 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.731 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.732 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.732 DEBUG:boto:Method: GET 2023-10-17T18:54:14.732 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.733 DEBUG:boto:Data: 2023-10-17T18:54:14.733 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.734 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.734 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.735 DEBUG:boto:Params: {} 2023-10-17T18:54:14.735 DEBUG:boto:Token: None 2023-10-17T18:54:14.736 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007?versionId=null 2023-10-17T18:54:14.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88= 2023-10-17T18:54:14.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ZV3CPVAhQGRblnoggkvZ61n+88=', 'Content-Length': '0'} 2023-10-17T18:54:14.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4e9a269b9d029c-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.740 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.740 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.740 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.741 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.741 DEBUG:boto:Method: GET 2023-10-17T18:54:14.742 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.742 DEBUG:boto:Data: 2023-10-17T18:54:14.743 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.744 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.744 DEBUG:boto:Params: {} 2023-10-17T18:54:14.744 DEBUG:boto:Token: None 2023-10-17T18:54:14.745 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f47927feae3985-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.749 DEBUG:boto:b'hbmuog-58obj000000000000000000071000falseobj000000000000000000072023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.750 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.750 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.751 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.751 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.751 DEBUG:boto:Method: GET 2023-10-17T18:54:14.752 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.752 DEBUG:boto:Data: 2023-10-17T18:54:14.753 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.753 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.754 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.754 DEBUG:boto:Params: {} 2023-10-17T18:54:14.754 DEBUG:boto:Token: None 2023-10-17T18:54:14.755 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.755 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005edbfbd4fd763277-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.761 DEBUG:boto:b'hbmuog-58obj000000000000000000071000falseobj000000000000000000072023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.761 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.762 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.762 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.763 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.763 DEBUG:boto:Data: 2023-10-17T18:54:14.763 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.764 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.764 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.765 DEBUG:boto:Params: {} 2023-10-17T18:54:14.765 DEBUG:boto:Token: None 2023-10-17T18:54:14.766 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.766 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eFcgPu95TsSC3ogt2lCGWYSetSQ= 2023-10-17T18:54:14.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eFcgPu95TsSC3ogt2lCGWYSetSQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0dbb68bedefdf9a-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.769 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.770 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.770 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.770 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.771 DEBUG:boto:Data: 2023-10-17T18:54:14.771 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.772 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.772 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.772 DEBUG:boto:Params: {} 2023-10-17T18:54:14.773 DEBUG:boto:Token: None 2023-10-17T18:54:14.773 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eFcgPu95TsSC3ogt2lCGWYSetSQ= 2023-10-17T18:54:14.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eFcgPu95TsSC3ogt2lCGWYSetSQ=', 'Content-Length': '0'} 2023-10-17T18:54:14.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccdd678d8400ba07-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-17T18:54:14.777 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.778 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.778 DEBUG:boto:Method: GET 2023-10-17T18:54:14.779 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.779 DEBUG:boto:Data: 2023-10-17T18:54:14.780 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.780 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.780 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.781 DEBUG:boto:Params: {} 2023-10-17T18:54:14.781 DEBUG:boto:Token: None 2023-10-17T18:54:14.782 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.782 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zol1feyPQcdqG+M+AuTsGtljE9g= 2023-10-17T18:54:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zol1feyPQcdqG+M+AuTsGtljE9g=', 'Content-Length': '0'} 2023-10-17T18:54:14.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c33241bb243d734-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.786 DEBUG:boto:path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.786 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.787 DEBUG:boto:Method: GET 2023-10-17T18:54:14.787 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.788 DEBUG:boto:Data: 2023-10-17T18:54:14.788 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.789 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.789 DEBUG:boto:Params: {} 2023-10-17T18:54:14.790 DEBUG:boto:Token: None 2023-10-17T18:54:14.790 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000007 2023-10-17T18:54:14.791 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zol1feyPQcdqG+M+AuTsGtljE9g= 2023-10-17T18:54:14.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zol1feyPQcdqG+M+AuTsGtljE9g=', 'Content-Length': '0'} 2023-10-17T18:54:14.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c5d40128af33f9-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.834 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.835 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.835 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.836 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.836 DEBUG:boto:Method: GET 2023-10-17T18:54:14.837 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.837 DEBUG:boto:Data: 2023-10-17T18:54:14.838 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.838 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.839 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.839 DEBUG:boto:Params: {} 2023-10-17T18:54:14.840 DEBUG:boto:Token: None 2023-10-17T18:54:14.840 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.841 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51b64921e653348-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.845 DEBUG:boto:b'hbmuog-58obj000000000000000000071000falseobj000000000000000000072023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.845 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.846 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.846 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.847 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.847 DEBUG:boto:Method: GET 2023-10-17T18:54:14.848 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000007 2023-10-17T18:54:14.848 DEBUG:boto:Data: 2023-10-17T18:54:14.849 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.850 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.850 DEBUG:boto:Params: {} 2023-10-17T18:54:14.851 DEBUG:boto:Token: None 2023-10-17T18:54:14.851 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de8b8f83bed01a9-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.857 DEBUG:boto:b'hbmuog-58obj000000000000000000071000falseobj000000000000000000072023-10-17T18:54:07.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-17T18:54:14.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.859 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.860 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.860 DEBUG:boto:Method: GET 2023-10-17T18:54:14.861 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.861 DEBUG:boto:Data: 2023-10-17T18:54:14.862 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.863 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.863 DEBUG:boto:Params: {} 2023-10-17T18:54:14.864 DEBUG:boto:Token: None 2023-10-17T18:54:14.864 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g= 2023-10-17T18:54:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g=', 'Content-Length': '0'} 2023-10-17T18:54:14.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b02cc005d33e90f-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.868 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.869 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.869 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.870 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.870 DEBUG:boto:Method: GET 2023-10-17T18:54:14.871 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.871 DEBUG:boto:Data: 2023-10-17T18:54:14.872 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.873 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.873 DEBUG:boto:Params: {} 2023-10-17T18:54:14.874 DEBUG:boto:Token: None 2023-10-17T18:54:14.874 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g= 2023-10-17T18:54:14.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g=', 'Content-Length': '0'} 2023-10-17T18:54:14.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005339e16bad168789-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.878 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.878 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.879 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.879 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.880 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.880 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.881 DEBUG:boto:Data: 2023-10-17T18:54:14.881 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.882 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.882 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.882 DEBUG:boto:Params: {} 2023-10-17T18:54:14.883 DEBUG:boto:Token: None 2023-10-17T18:54:14.883 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.884 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yYsnd9GFKhrDFaj2P4dyNPbcm00= 2023-10-17T18:54:14.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yYsnd9GFKhrDFaj2P4dyNPbcm00=', 'Content-Length': '0'} 2023-10-17T18:54:14.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a707933d2abb0c-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.887 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.888 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.888 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.888 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.889 DEBUG:boto:Method: HEAD 2023-10-17T18:54:14.889 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.890 DEBUG:boto:Data: 2023-10-17T18:54:14.890 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.891 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.891 DEBUG:boto:Params: {} 2023-10-17T18:54:14.892 DEBUG:boto:Token: None 2023-10-17T18:54:14.892 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yYsnd9GFKhrDFaj2P4dyNPbcm00= 2023-10-17T18:54:14.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yYsnd9GFKhrDFaj2P4dyNPbcm00=', 'Content-Length': '0'} 2023-10-17T18:54:14.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc90ed4b4b7b4ef-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-17T18:54:14.897 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.897 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.898 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.898 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.899 DEBUG:boto:Method: GET 2023-10-17T18:54:14.899 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.900 DEBUG:boto:Data: 2023-10-17T18:54:14.900 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.901 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.902 DEBUG:boto:Params: {} 2023-10-17T18:54:14.902 DEBUG:boto:Token: None 2023-10-17T18:54:14.903 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.903 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g= 2023-10-17T18:54:14.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g=', 'Content-Length': '0'} 2023-10-17T18:54:14.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb6dd67c02a524b-00652ed856-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.946 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.947 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:14.947 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.948 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.948 DEBUG:boto:Method: GET 2023-10-17T18:54:14.949 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.949 DEBUG:boto:Data: 2023-10-17T18:54:14.950 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.950 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.951 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.951 DEBUG:boto:Params: {} 2023-10-17T18:54:14.951 DEBUG:boto:Token: None 2023-10-17T18:54:14.952 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/obj00000000000000000008?versionId=null 2023-10-17T18:54:14.952 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g= 2023-10-17T18:54:14.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/U7CLchxk1cWQx9Dxqt+3+0kK3g=', 'Content-Length': '0'} 2023-10-17T18:54:14.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271a46943fb6ce2f-00652ed856-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.956 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.956 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.957 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.957 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.957 DEBUG:boto:Method: GET 2023-10-17T18:54:14.958 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.958 DEBUG:boto:Data: 2023-10-17T18:54:14.959 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.959 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:14.959 DEBUG:boto:Port: 8001 2023-10-17T18:54:14.960 DEBUG:boto:Params: {} 2023-10-17T18:54:14.960 DEBUG:boto:Token: None 2023-10-17T18:54:14.961 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.961 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e560af02c7849ba3-00652ed856-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:14.965 DEBUG:boto:b'hbmuog-58obj000000000000000000081000falseobj000000000000000000082023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:14.966 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:14.966 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:14.967 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.967 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.967 DEBUG:boto:Method: GET 2023-10-17T18:54:14.968 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:14.968 DEBUG:boto:Data: 2023-10-17T18:54:14.969 DEBUG:boto:Headers: {} 2023-10-17T18:54:14.969 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:14.970 DEBUG:boto:Port: 8000 2023-10-17T18:54:14.970 DEBUG:boto:Params: {} 2023-10-17T18:54:14.971 DEBUG:boto:Token: None 2023-10-17T18:54:14.971 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:14 GMT /hbmuog-58/ 2023-10-17T18:54:14.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU= 2023-10-17T18:54:14.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:14 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dQvbWaf7JcHwHwv2aR7qms3RNpU=', 'Content-Length': '0'} 2023-10-17T18:54:14.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e02ef9c8d78b5f-00652ed856-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.018 DEBUG:boto:b'hbmuog-58obj000000000000000000081000falseobj000000000000000000082023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.019 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.019 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.020 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.020 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.021 DEBUG:boto:Data: 2023-10-17T18:54:15.021 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.021 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.022 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.022 DEBUG:boto:Params: {} 2023-10-17T18:54:15.023 DEBUG:boto:Token: None 2023-10-17T18:54:15.023 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jLKDGtI34+84oHI5IPHDjUm1/C4= 2023-10-17T18:54:15.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jLKDGtI34+84oHI5IPHDjUm1/C4=', 'Content-Length': '0'} 2023-10-17T18:54:15.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000111b7e9983bb7811-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.027 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.028 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.028 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.029 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.029 DEBUG:boto:Data: 2023-10-17T18:54:15.030 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.030 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.031 DEBUG:boto:Params: {} 2023-10-17T18:54:15.031 DEBUG:boto:Token: None 2023-10-17T18:54:15.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jLKDGtI34+84oHI5IPHDjUm1/C4= 2023-10-17T18:54:15.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jLKDGtI34+84oHI5IPHDjUm1/C4=', 'Content-Length': '0'} 2023-10-17T18:54:15.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc1cff09b6f335c-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-17T18:54:15.036 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.036 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.037 DEBUG:boto:Method: GET 2023-10-17T18:54:15.037 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.038 DEBUG:boto:Data: 2023-10-17T18:54:15.038 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.039 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.039 DEBUG:boto:Params: {} 2023-10-17T18:54:15.040 DEBUG:boto:Token: None 2023-10-17T18:54:15.040 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+QuuGQ1+UC3V9nfFhEVDFSTfc5I= 2023-10-17T18:54:15.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+QuuGQ1+UC3V9nfFhEVDFSTfc5I=', 'Content-Length': '0'} 2023-10-17T18:54:15.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356e5fafecdf72fa-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.044 DEBUG:boto:path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.045 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.045 DEBUG:boto:Method: GET 2023-10-17T18:54:15.045 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.046 DEBUG:boto:Data: 2023-10-17T18:54:15.046 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.047 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.048 DEBUG:boto:Params: {} 2023-10-17T18:54:15.048 DEBUG:boto:Token: None 2023-10-17T18:54:15.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000008 2023-10-17T18:54:15.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+QuuGQ1+UC3V9nfFhEVDFSTfc5I= 2023-10-17T18:54:15.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+QuuGQ1+UC3V9nfFhEVDFSTfc5I=', 'Content-Length': '0'} 2023-10-17T18:54:15.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044bc0dfa1984455d-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.052 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.053 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.053 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.054 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.054 DEBUG:boto:Method: GET 2023-10-17T18:54:15.055 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.055 DEBUG:boto:Data: 2023-10-17T18:54:15.055 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.056 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.057 DEBUG:boto:Params: {} 2023-10-17T18:54:15.057 DEBUG:boto:Token: None 2023-10-17T18:54:15.058 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db3aaffc670885d9-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.062 DEBUG:boto:b'hbmuog-58obj000000000000000000081000falseobj000000000000000000082023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.063 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.063 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.063 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.064 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.064 DEBUG:boto:Method: GET 2023-10-17T18:54:15.065 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000008 2023-10-17T18:54:15.065 DEBUG:boto:Data: 2023-10-17T18:54:15.066 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.066 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.067 DEBUG:boto:Params: {} 2023-10-17T18:54:15.067 DEBUG:boto:Token: None 2023-10-17T18:54:15.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33e4b7190a09e89-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.074 DEBUG:boto:b'hbmuog-58obj000000000000000000081000falseobj000000000000000000082023-10-17T18:54:08.139Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-17T18:54:15.075 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.075 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.076 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.076 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.077 DEBUG:boto:Method: GET 2023-10-17T18:54:15.077 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.078 DEBUG:boto:Data: 2023-10-17T18:54:15.078 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.079 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.080 DEBUG:boto:Params: {} 2023-10-17T18:54:15.080 DEBUG:boto:Token: None 2023-10-17T18:54:15.081 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY= 2023-10-17T18:54:15.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY=', 'Content-Length': '0'} 2023-10-17T18:54:15.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929118769d7817a2-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.084 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.085 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.085 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.086 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.086 DEBUG:boto:Method: GET 2023-10-17T18:54:15.087 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.087 DEBUG:boto:Data: 2023-10-17T18:54:15.087 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.088 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.089 DEBUG:boto:Params: {} 2023-10-17T18:54:15.089 DEBUG:boto:Token: None 2023-10-17T18:54:15.090 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY= 2023-10-17T18:54:15.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY=', 'Content-Length': '0'} 2023-10-17T18:54:15.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1b2b2ac4f4e05b-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.134 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.135 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.135 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.136 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.136 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.137 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.137 DEBUG:boto:Data: 2023-10-17T18:54:15.137 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.138 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.139 DEBUG:boto:Params: {} 2023-10-17T18:54:15.139 DEBUG:boto:Token: None 2023-10-17T18:54:15.140 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iIc9v2BYh5wkCd0P2rAOD9ybv64= 2023-10-17T18:54:15.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iIc9v2BYh5wkCd0P2rAOD9ybv64=', 'Content-Length': '0'} 2023-10-17T18:54:15.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ff3a5fa8a5ee4fc-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.143 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.144 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.144 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.144 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.145 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.145 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.146 DEBUG:boto:Data: 2023-10-17T18:54:15.146 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.147 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.147 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.147 DEBUG:boto:Params: {} 2023-10-17T18:54:15.148 DEBUG:boto:Token: None 2023-10-17T18:54:15.148 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.149 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iIc9v2BYh5wkCd0P2rAOD9ybv64= 2023-10-17T18:54:15.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iIc9v2BYh5wkCd0P2rAOD9ybv64=', 'Content-Length': '0'} 2023-10-17T18:54:15.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063701dbb9da138f0-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-17T18:54:15.152 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.153 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.153 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.154 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.154 DEBUG:boto:Method: GET 2023-10-17T18:54:15.155 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.155 DEBUG:boto:Data: 2023-10-17T18:54:15.155 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.156 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.157 DEBUG:boto:Params: {} 2023-10-17T18:54:15.157 DEBUG:boto:Token: None 2023-10-17T18:54:15.158 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.158 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY= 2023-10-17T18:54:15.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY=', 'Content-Length': '0'} 2023-10-17T18:54:15.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b6f5cc64ae7374-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.202 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.203 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.203 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.204 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.204 DEBUG:boto:Method: GET 2023-10-17T18:54:15.205 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.205 DEBUG:boto:Data: 2023-10-17T18:54:15.205 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.206 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.207 DEBUG:boto:Params: {} 2023-10-17T18:54:15.207 DEBUG:boto:Token: None 2023-10-17T18:54:15.208 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009?versionId=null 2023-10-17T18:54:15.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY= 2023-10-17T18:54:15.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQ5blg2tn7ws/IdLLFSSKGncNRY=', 'Content-Length': '0'} 2023-10-17T18:54:15.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d47bee52d6f0bb6-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.211 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.212 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.212 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.213 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.213 DEBUG:boto:Method: GET 2023-10-17T18:54:15.214 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.214 DEBUG:boto:Data: 2023-10-17T18:54:15.214 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.215 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.216 DEBUG:boto:Params: {} 2023-10-17T18:54:15.217 DEBUG:boto:Token: None 2023-10-17T18:54:15.217 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.218 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5cbcbc07662474-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.221 DEBUG:boto:b'hbmuog-58obj000000000000000000091000falseobj000000000000000000092023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.222 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.222 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.223 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.223 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.223 DEBUG:boto:Method: GET 2023-10-17T18:54:15.224 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.224 DEBUG:boto:Data: 2023-10-17T18:54:15.225 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.225 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.226 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.226 DEBUG:boto:Params: {} 2023-10-17T18:54:15.226 DEBUG:boto:Token: None 2023-10-17T18:54:15.227 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.227 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7aa1bfc0380c403-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.235 DEBUG:boto:b'hbmuog-58obj000000000000000000091000falseobj000000000000000000092023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.235 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.237 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.238 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.238 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.238 DEBUG:boto:Data: 2023-10-17T18:54:15.239 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.240 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.240 DEBUG:boto:Params: {} 2023-10-17T18:54:15.241 DEBUG:boto:Token: None 2023-10-17T18:54:15.241 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LjH42Ldl/JtbsB+E4vQ6CkHT/G4= 2023-10-17T18:54:15.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LjH42Ldl/JtbsB+E4vQ6CkHT/G4=', 'Content-Length': '0'} 2023-10-17T18:54:15.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899379753bca537a-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.245 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.245 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.246 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.246 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.246 DEBUG:boto:Data: 2023-10-17T18:54:15.247 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.248 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.248 DEBUG:boto:Params: {} 2023-10-17T18:54:15.248 DEBUG:boto:Token: None 2023-10-17T18:54:15.249 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.250 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LjH42Ldl/JtbsB+E4vQ6CkHT/G4= 2023-10-17T18:54:15.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LjH42Ldl/JtbsB+E4vQ6CkHT/G4=', 'Content-Length': '0'} 2023-10-17T18:54:15.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff93252a2c915613-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-17T18:54:15.253 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.254 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.254 DEBUG:boto:Method: GET 2023-10-17T18:54:15.254 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.255 DEBUG:boto:Data: 2023-10-17T18:54:15.255 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.256 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.256 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.257 DEBUG:boto:Params: {} 2023-10-17T18:54:15.257 DEBUG:boto:Token: None 2023-10-17T18:54:15.257 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.258 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:459p/Y5ccb2Xt9VJqEbNpBQj9p0= 2023-10-17T18:54:15.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:459p/Y5ccb2Xt9VJqEbNpBQj9p0=', 'Content-Length': '0'} 2023-10-17T18:54:15.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f576f5c0d4492c-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.261 DEBUG:boto:path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.262 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.262 DEBUG:boto:Method: GET 2023-10-17T18:54:15.263 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.263 DEBUG:boto:Data: 2023-10-17T18:54:15.264 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.264 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.264 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.265 DEBUG:boto:Params: {} 2023-10-17T18:54:15.265 DEBUG:boto:Token: None 2023-10-17T18:54:15.266 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000009 2023-10-17T18:54:15.266 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:459p/Y5ccb2Xt9VJqEbNpBQj9p0= 2023-10-17T18:54:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:459p/Y5ccb2Xt9VJqEbNpBQj9p0=', 'Content-Length': '0'} 2023-10-17T18:54:15.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fcddb08c898b6f0-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.310 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.311 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.311 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.312 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.312 DEBUG:boto:Method: GET 2023-10-17T18:54:15.313 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.313 DEBUG:boto:Data: 2023-10-17T18:54:15.314 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.314 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.314 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.315 DEBUG:boto:Params: {} 2023-10-17T18:54:15.315 DEBUG:boto:Token: None 2023-10-17T18:54:15.316 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c8065f8314c64b0-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.320 DEBUG:boto:b'hbmuog-58obj000000000000000000091000falseobj000000000000000000092023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.321 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.321 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.322 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.322 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.322 DEBUG:boto:Method: GET 2023-10-17T18:54:15.323 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000009 2023-10-17T18:54:15.323 DEBUG:boto:Data: 2023-10-17T18:54:15.324 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.324 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.325 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.325 DEBUG:boto:Params: {} 2023-10-17T18:54:15.325 DEBUG:boto:Token: None 2023-10-17T18:54:15.326 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.326 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003381653e1a36b60c-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.332 DEBUG:boto:b'hbmuog-58obj000000000000000000091000falseobj000000000000000000092023-10-17T18:54:05.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:54:15.333 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.334 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.334 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.335 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.335 DEBUG:boto:Method: GET 2023-10-17T18:54:15.335 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.336 DEBUG:boto:Data: 2023-10-17T18:54:15.336 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.337 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.337 DEBUG:boto:Params: {} 2023-10-17T18:54:15.338 DEBUG:boto:Token: None 2023-10-17T18:54:15.338 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.339 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ= 2023-10-17T18:54:15.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ=', 'Content-Length': '0'} 2023-10-17T18:54:15.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006774be26250e1b58-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.342 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.343 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.343 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.344 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.344 DEBUG:boto:Method: GET 2023-10-17T18:54:15.345 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.345 DEBUG:boto:Data: 2023-10-17T18:54:15.345 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.346 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.347 DEBUG:boto:Params: {} 2023-10-17T18:54:15.347 DEBUG:boto:Token: None 2023-10-17T18:54:15.348 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ= 2023-10-17T18:54:15.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ=', 'Content-Length': '0'} 2023-10-17T18:54:15.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d64d6efb9273a5-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.351 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.352 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.352 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.352 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.353 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.353 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.354 DEBUG:boto:Data: 2023-10-17T18:54:15.354 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.355 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.355 DEBUG:boto:Params: {} 2023-10-17T18:54:15.356 DEBUG:boto:Token: None 2023-10-17T18:54:15.356 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:STfZlZ8+1o2mouPkDOFIgR95dqg= 2023-10-17T18:54:15.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:STfZlZ8+1o2mouPkDOFIgR95dqg=', 'Content-Length': '0'} 2023-10-17T18:54:15.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c0cdb98902b13d-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.360 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.360 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.361 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.361 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.361 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.362 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.362 DEBUG:boto:Data: 2023-10-17T18:54:15.363 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.364 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.364 DEBUG:boto:Params: {} 2023-10-17T18:54:15.364 DEBUG:boto:Token: None 2023-10-17T18:54:15.365 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:STfZlZ8+1o2mouPkDOFIgR95dqg= 2023-10-17T18:54:15.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:STfZlZ8+1o2mouPkDOFIgR95dqg=', 'Content-Length': '0'} 2023-10-17T18:54:15.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31f0803e815c633-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:54:15.369 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.369 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.370 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.370 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.371 DEBUG:boto:Method: GET 2023-10-17T18:54:15.371 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.371 DEBUG:boto:Data: 2023-10-17T18:54:15.372 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.373 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.373 DEBUG:boto:Params: {} 2023-10-17T18:54:15.374 DEBUG:boto:Token: None 2023-10-17T18:54:15.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ= 2023-10-17T18:54:15.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ=', 'Content-Length': '0'} 2023-10-17T18:54:15.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035aab2d05cd91c1d-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.418 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.419 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.419 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.420 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.420 DEBUG:boto:Method: GET 2023-10-17T18:54:15.421 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.421 DEBUG:boto:Data: 2023-10-17T18:54:15.422 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.423 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.423 DEBUG:boto:Params: {} 2023-10-17T18:54:15.424 DEBUG:boto:Token: None 2023-10-17T18:54:15.424 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010?versionId=null 2023-10-17T18:54:15.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ= 2023-10-17T18:54:15.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wJ10uHmdjlZaL++dRvsAuGx9QqQ=', 'Content-Length': '0'} 2023-10-17T18:54:15.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0d7b67f80daa86-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.428 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.428 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.429 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.429 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.430 DEBUG:boto:Method: GET 2023-10-17T18:54:15.430 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.431 DEBUG:boto:Data: 2023-10-17T18:54:15.431 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.431 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.432 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.432 DEBUG:boto:Params: {} 2023-10-17T18:54:15.433 DEBUG:boto:Token: None 2023-10-17T18:54:15.433 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.434 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ba6f3106cb9197-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.438 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.439 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.439 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.439 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.440 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.440 DEBUG:boto:Method: GET 2023-10-17T18:54:15.441 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.441 DEBUG:boto:Data: 2023-10-17T18:54:15.442 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.442 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.443 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.443 DEBUG:boto:Params: {} 2023-10-17T18:54:15.443 DEBUG:boto:Token: None 2023-10-17T18:54:15.444 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.444 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d34b3cd693e8182-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.450 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.451 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.453 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.453 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.454 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.454 DEBUG:boto:Data: 2023-10-17T18:54:15.455 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.456 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.456 DEBUG:boto:Params: {} 2023-10-17T18:54:15.456 DEBUG:boto:Token: None 2023-10-17T18:54:15.457 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kvSwkeAF/em3jTA4oIl+LYoMoOI= 2023-10-17T18:54:15.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kvSwkeAF/em3jTA4oIl+LYoMoOI=', 'Content-Length': '0'} 2023-10-17T18:54:15.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089830f3e50b17a83-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.460 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.461 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.461 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.462 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.462 DEBUG:boto:Data: 2023-10-17T18:54:15.463 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.463 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.464 DEBUG:boto:Params: {} 2023-10-17T18:54:15.464 DEBUG:boto:Token: None 2023-10-17T18:54:15.465 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kvSwkeAF/em3jTA4oIl+LYoMoOI= 2023-10-17T18:54:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kvSwkeAF/em3jTA4oIl+LYoMoOI=', 'Content-Length': '0'} 2023-10-17T18:54:15.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9bbad4c19f0b74-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-17T18:54:15.469 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.469 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.470 DEBUG:boto:Method: GET 2023-10-17T18:54:15.470 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.471 DEBUG:boto:Data: 2023-10-17T18:54:15.471 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.472 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.472 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.472 DEBUG:boto:Params: {} 2023-10-17T18:54:15.473 DEBUG:boto:Token: None 2023-10-17T18:54:15.473 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.474 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MgCmfgZ10x74hbJgISoqeu3eCXY= 2023-10-17T18:54:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MgCmfgZ10x74hbJgISoqeu3eCXY=', 'Content-Length': '0'} 2023-10-17T18:54:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b3eb74eb804f60-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.477 DEBUG:boto:path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.478 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.478 DEBUG:boto:Method: GET 2023-10-17T18:54:15.478 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.479 DEBUG:boto:Data: 2023-10-17T18:54:15.479 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.480 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.481 DEBUG:boto:Params: {} 2023-10-17T18:54:15.481 DEBUG:boto:Token: None 2023-10-17T18:54:15.482 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000010 2023-10-17T18:54:15.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MgCmfgZ10x74hbJgISoqeu3eCXY= 2023-10-17T18:54:15.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MgCmfgZ10x74hbJgISoqeu3eCXY=', 'Content-Length': '0'} 2023-10-17T18:54:15.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f97bc2b075552a-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.485 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.486 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.486 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.487 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.487 DEBUG:boto:Method: GET 2023-10-17T18:54:15.488 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.488 DEBUG:boto:Data: 2023-10-17T18:54:15.488 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.489 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.490 DEBUG:boto:Params: {} 2023-10-17T18:54:15.490 DEBUG:boto:Token: None 2023-10-17T18:54:15.491 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa84d84b4517a2c6-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.495 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.496 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.496 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.497 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.497 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.497 DEBUG:boto:Method: GET 2023-10-17T18:54:15.498 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000010 2023-10-17T18:54:15.498 DEBUG:boto:Data: 2023-10-17T18:54:15.499 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.500 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.500 DEBUG:boto:Params: {} 2023-10-17T18:54:15.501 DEBUG:boto:Token: None 2023-10-17T18:54:15.501 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.502 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc48941810c626e7-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.507 DEBUG:boto:b'hbmuog-58obj000000000000000000101000falseobj000000000000000000102023-10-17T18:54:08.243Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:54:15.508 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.508 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.509 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.509 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.510 DEBUG:boto:Method: GET 2023-10-17T18:54:15.510 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.511 DEBUG:boto:Data: 2023-10-17T18:54:15.511 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.512 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.512 DEBUG:boto:Params: {} 2023-10-17T18:54:15.513 DEBUG:boto:Token: None 2023-10-17T18:54:15.513 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A= 2023-10-17T18:54:15.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A=', 'Content-Length': '0'} 2023-10-17T18:54:15.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f07f0594874dd9-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.517 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.518 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.518 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.519 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.519 DEBUG:boto:Method: GET 2023-10-17T18:54:15.520 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.520 DEBUG:boto:Data: 2023-10-17T18:54:15.521 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.521 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.522 DEBUG:boto:Params: {} 2023-10-17T18:54:15.522 DEBUG:boto:Token: None 2023-10-17T18:54:15.523 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A= 2023-10-17T18:54:15.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A=', 'Content-Length': '0'} 2023-10-17T18:54:15.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043436233ecbc08f9-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.566 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.567 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.567 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.568 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.568 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.569 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.569 DEBUG:boto:Data: 2023-10-17T18:54:15.570 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.570 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.571 DEBUG:boto:Params: {} 2023-10-17T18:54:15.571 DEBUG:boto:Token: None 2023-10-17T18:54:15.572 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uqylXMY1QvSaC3+omlKizhqWejM= 2023-10-17T18:54:15.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uqylXMY1QvSaC3+omlKizhqWejM=', 'Content-Length': '0'} 2023-10-17T18:54:15.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121732db19bb1882-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.576 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.576 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.577 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.577 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.577 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.578 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.578 DEBUG:boto:Data: 2023-10-17T18:54:15.579 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.579 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.580 DEBUG:boto:Params: {} 2023-10-17T18:54:15.580 DEBUG:boto:Token: None 2023-10-17T18:54:15.581 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uqylXMY1QvSaC3+omlKizhqWejM= 2023-10-17T18:54:15.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uqylXMY1QvSaC3+omlKizhqWejM=', 'Content-Length': '0'} 2023-10-17T18:54:15.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139e46c7a9f8e1f0-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:54:15.585 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.585 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.586 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.586 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.587 DEBUG:boto:Method: GET 2023-10-17T18:54:15.587 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.587 DEBUG:boto:Data: 2023-10-17T18:54:15.588 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.589 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.589 DEBUG:boto:Params: {} 2023-10-17T18:54:15.590 DEBUG:boto:Token: None 2023-10-17T18:54:15.590 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A= 2023-10-17T18:54:15.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A=', 'Content-Length': '0'} 2023-10-17T18:54:15.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22af79e5d242133-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.638 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.639 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.639 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.640 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.640 DEBUG:boto:Method: GET 2023-10-17T18:54:15.641 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.641 DEBUG:boto:Data: 2023-10-17T18:54:15.642 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.642 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.643 DEBUG:boto:Params: {} 2023-10-17T18:54:15.643 DEBUG:boto:Token: None 2023-10-17T18:54:15.644 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011?versionId=null 2023-10-17T18:54:15.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A= 2023-10-17T18:54:15.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b5fZLgFdpQUfgt8Fa5PCgWIPo+A=', 'Content-Length': '0'} 2023-10-17T18:54:15.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa88c1516678692d-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.648 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.648 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.648 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.649 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.649 DEBUG:boto:Method: GET 2023-10-17T18:54:15.650 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.650 DEBUG:boto:Data: 2023-10-17T18:54:15.651 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.652 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.652 DEBUG:boto:Params: {} 2023-10-17T18:54:15.652 DEBUG:boto:Token: None 2023-10-17T18:54:15.653 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2476661320a92ce-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.657 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.658 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.658 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.659 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.659 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.660 DEBUG:boto:Method: GET 2023-10-17T18:54:15.660 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.661 DEBUG:boto:Data: 2023-10-17T18:54:15.661 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.662 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.662 DEBUG:boto:Params: {} 2023-10-17T18:54:15.663 DEBUG:boto:Token: None 2023-10-17T18:54:15.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f39c49f3651ed4a-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.670 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.671 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.672 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.672 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.672 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.673 DEBUG:boto:Data: 2023-10-17T18:54:15.673 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.674 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.675 DEBUG:boto:Params: {} 2023-10-17T18:54:15.675 DEBUG:boto:Token: None 2023-10-17T18:54:15.675 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kawZDcrFqyfRGP5kynUHqcNjB+s= 2023-10-17T18:54:15.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kawZDcrFqyfRGP5kynUHqcNjB+s=', 'Content-Length': '0'} 2023-10-17T18:54:15.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a75750980d97719-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.679 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.679 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.680 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.680 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.681 DEBUG:boto:Data: 2023-10-17T18:54:15.681 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.682 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.682 DEBUG:boto:Params: {} 2023-10-17T18:54:15.683 DEBUG:boto:Token: None 2023-10-17T18:54:15.684 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kawZDcrFqyfRGP5kynUHqcNjB+s= 2023-10-17T18:54:15.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kawZDcrFqyfRGP5kynUHqcNjB+s=', 'Content-Length': '0'} 2023-10-17T18:54:15.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000c4ab3cdc7b415-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-17T18:54:15.687 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.688 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.688 DEBUG:boto:Method: GET 2023-10-17T18:54:15.689 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.689 DEBUG:boto:Data: 2023-10-17T18:54:15.690 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.691 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.691 DEBUG:boto:Params: {} 2023-10-17T18:54:15.691 DEBUG:boto:Token: None 2023-10-17T18:54:15.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:63x5A2NT1F/oZoz5fkHfE4dH23o= 2023-10-17T18:54:15.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:63x5A2NT1F/oZoz5fkHfE4dH23o=', 'Content-Length': '0'} 2023-10-17T18:54:15.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebfc674f5fd74fad-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.695 DEBUG:boto:path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.696 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.696 DEBUG:boto:Method: GET 2023-10-17T18:54:15.697 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.697 DEBUG:boto:Data: 2023-10-17T18:54:15.698 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.698 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.699 DEBUG:boto:Params: {} 2023-10-17T18:54:15.699 DEBUG:boto:Token: None 2023-10-17T18:54:15.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000011 2023-10-17T18:54:15.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:63x5A2NT1F/oZoz5fkHfE4dH23o= 2023-10-17T18:54:15.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:63x5A2NT1F/oZoz5fkHfE4dH23o=', 'Content-Length': '0'} 2023-10-17T18:54:15.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7b9066f49d26c6-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.746 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.747 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.747 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.748 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.748 DEBUG:boto:Method: GET 2023-10-17T18:54:15.749 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.749 DEBUG:boto:Data: 2023-10-17T18:54:15.750 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.751 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.751 DEBUG:boto:Params: {} 2023-10-17T18:54:15.752 DEBUG:boto:Token: None 2023-10-17T18:54:15.752 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ebdcc86028bf7d-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.757 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.757 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.758 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.758 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.759 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.759 DEBUG:boto:Method: GET 2023-10-17T18:54:15.760 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000011 2023-10-17T18:54:15.760 DEBUG:boto:Data: 2023-10-17T18:54:15.761 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.761 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.761 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.762 DEBUG:boto:Params: {} 2023-10-17T18:54:15.762 DEBUG:boto:Token: None 2023-10-17T18:54:15.763 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.763 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000879b88bbc9f052a-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.769 DEBUG:boto:b'hbmuog-58obj000000000000000000111000falseobj000000000000000000112023-10-17T18:54:06.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-17T18:54:15.770 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.771 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.771 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.771 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.772 DEBUG:boto:Method: GET 2023-10-17T18:54:15.772 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.773 DEBUG:boto:Data: 2023-10-17T18:54:15.773 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.774 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.774 DEBUG:boto:Params: {} 2023-10-17T18:54:15.775 DEBUG:boto:Token: None 2023-10-17T18:54:15.775 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho= 2023-10-17T18:54:15.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho=', 'Content-Length': '0'} 2023-10-17T18:54:15.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f406c0cefea17547-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.779 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.780 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.780 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.781 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.781 DEBUG:boto:Method: GET 2023-10-17T18:54:15.781 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.782 DEBUG:boto:Data: 2023-10-17T18:54:15.782 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.783 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.784 DEBUG:boto:Params: {} 2023-10-17T18:54:15.784 DEBUG:boto:Token: None 2023-10-17T18:54:15.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho= 2023-10-17T18:54:15.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho=', 'Content-Length': '0'} 2023-10-17T18:54:15.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019acd558dd2292b6-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.788 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.789 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.790 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.790 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.791 DEBUG:boto:Data: 2023-10-17T18:54:15.791 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.792 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.792 DEBUG:boto:Params: {} 2023-10-17T18:54:15.793 DEBUG:boto:Token: None 2023-10-17T18:54:15.793 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bAajLMh1ygCuhwggloXumSLoElk= 2023-10-17T18:54:15.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bAajLMh1ygCuhwggloXumSLoElk=', 'Content-Length': '0'} 2023-10-17T18:54:15.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d902529eee9e2b-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.797 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.797 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.798 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.798 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.798 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.799 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.799 DEBUG:boto:Data: 2023-10-17T18:54:15.800 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.800 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.800 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.801 DEBUG:boto:Params: {} 2023-10-17T18:54:15.801 DEBUG:boto:Token: None 2023-10-17T18:54:15.802 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bAajLMh1ygCuhwggloXumSLoElk= 2023-10-17T18:54:15.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bAajLMh1ygCuhwggloXumSLoElk=', 'Content-Length': '0'} 2023-10-17T18:54:15.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378246dab5eb488a-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-17T18:54:15.805 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.806 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.806 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.807 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.807 DEBUG:boto:Method: GET 2023-10-17T18:54:15.808 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.808 DEBUG:boto:Data: 2023-10-17T18:54:15.808 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.809 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.810 DEBUG:boto:Params: {} 2023-10-17T18:54:15.810 DEBUG:boto:Token: None 2023-10-17T18:54:15.811 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho= 2023-10-17T18:54:15.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho=', 'Content-Length': '0'} 2023-10-17T18:54:15.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74dc22186d4af84-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.854 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.855 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.855 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.855 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.856 DEBUG:boto:Method: GET 2023-10-17T18:54:15.856 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.857 DEBUG:boto:Data: 2023-10-17T18:54:15.857 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.858 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.859 DEBUG:boto:Params: {} 2023-10-17T18:54:15.859 DEBUG:boto:Token: None 2023-10-17T18:54:15.860 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012?versionId=null 2023-10-17T18:54:15.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho= 2023-10-17T18:54:15.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EzzIpWazprD02sQQh90OTP3UIho=', 'Content-Length': '0'} 2023-10-17T18:54:15.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8c14a70d13872e-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.863 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.864 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.864 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.865 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.865 DEBUG:boto:Method: GET 2023-10-17T18:54:15.866 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.866 DEBUG:boto:Data: 2023-10-17T18:54:15.867 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.868 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.868 DEBUG:boto:Params: {} 2023-10-17T18:54:15.869 DEBUG:boto:Token: None 2023-10-17T18:54:15.869 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.870 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e7e9a9a99be55d-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.873 DEBUG:boto:b'hbmuog-58obj000000000000000000121000falseobj000000000000000000122023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.874 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.875 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.875 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.876 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.876 DEBUG:boto:Method: GET 2023-10-17T18:54:15.877 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.877 DEBUG:boto:Data: 2023-10-17T18:54:15.877 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.878 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.879 DEBUG:boto:Params: {} 2023-10-17T18:54:15.879 DEBUG:boto:Token: None 2023-10-17T18:54:15.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.880 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcea5e8a8a240918-00652ed857-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.926 DEBUG:boto:b'hbmuog-58obj000000000000000000121000falseobj000000000000000000122023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:15.927 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.928 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.928 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.928 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.929 DEBUG:boto:Data: 2023-10-17T18:54:15.929 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.930 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.931 DEBUG:boto:Params: {} 2023-10-17T18:54:15.931 DEBUG:boto:Token: None 2023-10-17T18:54:15.932 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6n0aD1lIj6PQ0wZgEA/HvEowiAg= 2023-10-17T18:54:15.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6n0aD1lIj6PQ0wZgEA/HvEowiAg=', 'Content-Length': '0'} 2023-10-17T18:54:15.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000819b2af75dd920e1-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.935 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.936 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.936 DEBUG:boto:Method: HEAD 2023-10-17T18:54:15.936 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.937 DEBUG:boto:Data: 2023-10-17T18:54:15.937 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.938 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.939 DEBUG:boto:Params: {} 2023-10-17T18:54:15.939 DEBUG:boto:Token: None 2023-10-17T18:54:15.940 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.940 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6n0aD1lIj6PQ0wZgEA/HvEowiAg= 2023-10-17T18:54:15.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6n0aD1lIj6PQ0wZgEA/HvEowiAg=', 'Content-Length': '0'} 2023-10-17T18:54:15.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258d65ea3606dd74-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-17T18:54:15.943 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.944 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.944 DEBUG:boto:Method: GET 2023-10-17T18:54:15.945 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.945 DEBUG:boto:Data: 2023-10-17T18:54:15.946 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.947 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.947 DEBUG:boto:Params: {} 2023-10-17T18:54:15.948 DEBUG:boto:Token: None 2023-10-17T18:54:15.948 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaKhxTzmIluOrIhmnXOLXG24094= 2023-10-17T18:54:15.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaKhxTzmIluOrIhmnXOLXG24094=', 'Content-Length': '0'} 2023-10-17T18:54:15.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c467f4a1f83ef6c7-00652ed857-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.952 DEBUG:boto:path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.952 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.953 DEBUG:boto:Method: GET 2023-10-17T18:54:15.953 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.953 DEBUG:boto:Data: 2023-10-17T18:54:15.954 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:15.955 DEBUG:boto:Port: 8000 2023-10-17T18:54:15.955 DEBUG:boto:Params: {} 2023-10-17T18:54:15.956 DEBUG:boto:Token: None 2023-10-17T18:54:15.956 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/obj00000000000000000012 2023-10-17T18:54:15.957 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VaKhxTzmIluOrIhmnXOLXG24094= 2023-10-17T18:54:15.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VaKhxTzmIluOrIhmnXOLXG24094=', 'Content-Length': '0'} 2023-10-17T18:54:15.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c941d06adab01d35-00652ed857-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:15.960 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:15.960 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:15.961 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.961 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.962 DEBUG:boto:Method: GET 2023-10-17T18:54:15.962 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:15.963 DEBUG:boto:Data: 2023-10-17T18:54:15.963 DEBUG:boto:Headers: {} 2023-10-17T18:54:15.963 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:15.964 DEBUG:boto:Port: 8001 2023-10-17T18:54:15.964 DEBUG:boto:Params: {} 2023-10-17T18:54:15.965 DEBUG:boto:Token: None 2023-10-17T18:54:15.965 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:15 GMT /hbmuog-58/ 2023-10-17T18:54:15.966 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY= 2023-10-17T18:54:15.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dyU4UikYA+A9AXy927t5xsAVNjY=', 'Content-Length': '0'} 2023-10-17T18:54:15.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa97d535a20216ca-00652ed857-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.010 DEBUG:boto:b'hbmuog-58obj000000000000000000121000falseobj000000000000000000122023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.011 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.012 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.012 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:16.013 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:16.013 DEBUG:boto:Method: GET 2023-10-17T18:54:16.014 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000012 2023-10-17T18:54:16.014 DEBUG:boto:Data: 2023-10-17T18:54:16.014 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.015 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.015 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.016 DEBUG:boto:Params: {} 2023-10-17T18:54:16.016 DEBUG:boto:Token: None 2023-10-17T18:54:16.017 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001035d50fbb96d741-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.022 DEBUG:boto:b'hbmuog-58obj000000000000000000121000falseobj000000000000000000122023-10-17T18:54:08.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:54:16.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.024 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.025 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.025 DEBUG:boto:Method: GET 2023-10-17T18:54:16.026 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.026 DEBUG:boto:Data: 2023-10-17T18:54:16.027 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.027 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.028 DEBUG:boto:Params: {} 2023-10-17T18:54:16.028 DEBUG:boto:Token: None 2023-10-17T18:54:16.029 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw= 2023-10-17T18:54:16.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw=', 'Content-Length': '0'} 2023-10-17T18:54:16.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b350e0d038f076-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.033 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.033 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.034 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.034 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.034 DEBUG:boto:Method: GET 2023-10-17T18:54:16.035 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.035 DEBUG:boto:Data: 2023-10-17T18:54:16.036 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.036 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.037 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.037 DEBUG:boto:Params: {} 2023-10-17T18:54:16.037 DEBUG:boto:Token: None 2023-10-17T18:54:16.038 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.038 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw= 2023-10-17T18:54:16.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw=', 'Content-Length': '0'} 2023-10-17T18:54:16.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ce80d985a4472d-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.042 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.042 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.042 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.043 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.043 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.044 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.044 DEBUG:boto:Data: 2023-10-17T18:54:16.044 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.045 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.046 DEBUG:boto:Params: {} 2023-10-17T18:54:16.046 DEBUG:boto:Token: None 2023-10-17T18:54:16.047 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.047 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ig0PaySHtoNwsWh2aBXs/CKLXNM= 2023-10-17T18:54:16.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ig0PaySHtoNwsWh2aBXs/CKLXNM=', 'Content-Length': '0'} 2023-10-17T18:54:16.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e6abefeaa9e9d9-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.050 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.050 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.051 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.051 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.052 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.052 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.053 DEBUG:boto:Data: 2023-10-17T18:54:16.053 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.054 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.054 DEBUG:boto:Params: {} 2023-10-17T18:54:16.055 DEBUG:boto:Token: None 2023-10-17T18:54:16.055 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ig0PaySHtoNwsWh2aBXs/CKLXNM= 2023-10-17T18:54:16.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ig0PaySHtoNwsWh2aBXs/CKLXNM=', 'Content-Length': '0'} 2023-10-17T18:54:16.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88004a27763dc9f-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:54:16.059 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.060 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.060 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.061 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.061 DEBUG:boto:Method: GET 2023-10-17T18:54:16.062 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.062 DEBUG:boto:Data: 2023-10-17T18:54:16.063 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.063 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.063 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.064 DEBUG:boto:Params: {} 2023-10-17T18:54:16.064 DEBUG:boto:Token: None 2023-10-17T18:54:16.065 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw= 2023-10-17T18:54:16.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw=', 'Content-Length': '0'} 2023-10-17T18:54:16.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8ff231be8f6fb4-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.068 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.069 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.069 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.070 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.070 DEBUG:boto:Method: GET 2023-10-17T18:54:16.071 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.071 DEBUG:boto:Data: 2023-10-17T18:54:16.072 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.072 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.072 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.073 DEBUG:boto:Params: {} 2023-10-17T18:54:16.073 DEBUG:boto:Token: None 2023-10-17T18:54:16.074 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013?versionId=null 2023-10-17T18:54:16.074 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw= 2023-10-17T18:54:16.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zZFsCshdOtMtJd+0YqE5/fAwqNw=', 'Content-Length': '0'} 2023-10-17T18:54:16.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17d2c3d9bd5a9a5-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.078 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.078 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.079 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.079 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.079 DEBUG:boto:Method: GET 2023-10-17T18:54:16.080 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.080 DEBUG:boto:Data: 2023-10-17T18:54:16.081 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.082 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.082 DEBUG:boto:Params: {} 2023-10-17T18:54:16.083 DEBUG:boto:Token: None 2023-10-17T18:54:16.083 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024ed236805aeb45-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.087 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.088 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.088 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.089 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.089 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.090 DEBUG:boto:Method: GET 2023-10-17T18:54:16.090 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.091 DEBUG:boto:Data: 2023-10-17T18:54:16.091 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.092 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.092 DEBUG:boto:Params: {} 2023-10-17T18:54:16.093 DEBUG:boto:Token: None 2023-10-17T18:54:16.093 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009de473dc8b560262-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.099 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.100 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.100 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.101 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.101 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.101 DEBUG:boto:Data: 2023-10-17T18:54:16.102 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.103 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.103 DEBUG:boto:Params: {} 2023-10-17T18:54:16.103 DEBUG:boto:Token: None 2023-10-17T18:54:16.104 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vZaAvze53NjjcU50y802EYwHY5w= 2023-10-17T18:54:16.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vZaAvze53NjjcU50y802EYwHY5w=', 'Content-Length': '0'} 2023-10-17T18:54:16.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56f1d9b0e484b38-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.108 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.108 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.108 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.109 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.109 DEBUG:boto:Data: 2023-10-17T18:54:16.110 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.111 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.111 DEBUG:boto:Params: {} 2023-10-17T18:54:16.112 DEBUG:boto:Token: None 2023-10-17T18:54:16.112 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vZaAvze53NjjcU50y802EYwHY5w= 2023-10-17T18:54:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vZaAvze53NjjcU50y802EYwHY5w=', 'Content-Length': '0'} 2023-10-17T18:54:16.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003218ad791be7c6a8-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-17T18:54:16.116 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.116 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.117 DEBUG:boto:Method: GET 2023-10-17T18:54:16.117 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.118 DEBUG:boto:Data: 2023-10-17T18:54:16.118 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.119 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.119 DEBUG:boto:Params: {} 2023-10-17T18:54:16.120 DEBUG:boto:Token: None 2023-10-17T18:54:16.120 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oioF4nF+O0yONStvqSEbJ1bgpT8= 2023-10-17T18:54:16.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oioF4nF+O0yONStvqSEbJ1bgpT8=', 'Content-Length': '0'} 2023-10-17T18:54:16.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082470566804d544d-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.166 DEBUG:boto:path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.167 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.167 DEBUG:boto:Method: GET 2023-10-17T18:54:16.168 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.168 DEBUG:boto:Data: 2023-10-17T18:54:16.169 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.169 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.170 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.170 DEBUG:boto:Params: {} 2023-10-17T18:54:16.171 DEBUG:boto:Token: None 2023-10-17T18:54:16.171 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000013 2023-10-17T18:54:16.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oioF4nF+O0yONStvqSEbJ1bgpT8= 2023-10-17T18:54:16.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oioF4nF+O0yONStvqSEbJ1bgpT8=', 'Content-Length': '0'} 2023-10-17T18:54:16.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda1061716ad4d87-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.175 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.175 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.176 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.176 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.177 DEBUG:boto:Method: GET 2023-10-17T18:54:16.177 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.177 DEBUG:boto:Data: 2023-10-17T18:54:16.178 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.179 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.179 DEBUG:boto:Params: {} 2023-10-17T18:54:16.180 DEBUG:boto:Token: None 2023-10-17T18:54:16.180 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.181 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009571851fc9f24015-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.184 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.185 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.185 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.186 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.186 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.187 DEBUG:boto:Method: GET 2023-10-17T18:54:16.187 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000013 2023-10-17T18:54:16.188 DEBUG:boto:Data: 2023-10-17T18:54:16.188 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.189 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.189 DEBUG:boto:Params: {} 2023-10-17T18:54:16.190 DEBUG:boto:Token: None 2023-10-17T18:54:16.190 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.191 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3eebc430d98b3a-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.195 DEBUG:boto:b'hbmuog-58obj000000000000000000131000falseobj000000000000000000132023-10-17T18:54:07.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-17T18:54:16.196 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.197 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.197 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.198 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.198 DEBUG:boto:Method: GET 2023-10-17T18:54:16.199 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.199 DEBUG:boto:Data: 2023-10-17T18:54:16.199 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.200 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.200 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.201 DEBUG:boto:Params: {} 2023-10-17T18:54:16.201 DEBUG:boto:Token: None 2023-10-17T18:54:16.201 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.202 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk= 2023-10-17T18:54:16.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk=', 'Content-Length': '0'} 2023-10-17T18:54:16.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f259ad31f1b185d-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.206 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.206 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.206 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.207 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.207 DEBUG:boto:Method: GET 2023-10-17T18:54:16.208 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.208 DEBUG:boto:Data: 2023-10-17T18:54:16.208 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.209 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.210 DEBUG:boto:Params: {} 2023-10-17T18:54:16.210 DEBUG:boto:Token: None 2023-10-17T18:54:16.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk= 2023-10-17T18:54:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk=', 'Content-Length': '0'} 2023-10-17T18:54:16.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c84916134cc1f8-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.214 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.215 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.215 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.215 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.216 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.216 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.217 DEBUG:boto:Data: 2023-10-17T18:54:16.217 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.218 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.218 DEBUG:boto:Params: {} 2023-10-17T18:54:16.219 DEBUG:boto:Token: None 2023-10-17T18:54:16.219 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HbuJ+LN2bcmuNIQTG4dWmtJs030= 2023-10-17T18:54:16.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbuJ+LN2bcmuNIQTG4dWmtJs030=', 'Content-Length': '0'} 2023-10-17T18:54:16.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014562e32fa8dca40-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.223 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.223 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.223 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.224 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.224 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.225 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.225 DEBUG:boto:Data: 2023-10-17T18:54:16.225 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.226 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.227 DEBUG:boto:Params: {} 2023-10-17T18:54:16.227 DEBUG:boto:Token: None 2023-10-17T18:54:16.227 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HbuJ+LN2bcmuNIQTG4dWmtJs030= 2023-10-17T18:54:16.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbuJ+LN2bcmuNIQTG4dWmtJs030=', 'Content-Length': '0'} 2023-10-17T18:54:16.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003431e4631e4b6673-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-17T18:54:16.231 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.232 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.232 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.233 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.233 DEBUG:boto:Method: GET 2023-10-17T18:54:16.233 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.234 DEBUG:boto:Data: 2023-10-17T18:54:16.234 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.235 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.235 DEBUG:boto:Params: {} 2023-10-17T18:54:16.236 DEBUG:boto:Token: None 2023-10-17T18:54:16.236 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk= 2023-10-17T18:54:16.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk=', 'Content-Length': '0'} 2023-10-17T18:54:16.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087adf1b6bafa98f7-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.240 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.240 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.241 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.241 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.242 DEBUG:boto:Method: GET 2023-10-17T18:54:16.242 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.242 DEBUG:boto:Data: 2023-10-17T18:54:16.243 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.243 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.244 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.244 DEBUG:boto:Params: {} 2023-10-17T18:54:16.245 DEBUG:boto:Token: None 2023-10-17T18:54:16.245 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014?versionId=null 2023-10-17T18:54:16.246 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk= 2023-10-17T18:54:16.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wT8jUgZhDMFY1bt/smLr+RGUlhk=', 'Content-Length': '0'} 2023-10-17T18:54:16.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963ecc071090f4ad-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.290 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.291 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.291 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.292 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.292 DEBUG:boto:Method: GET 2023-10-17T18:54:16.293 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.293 DEBUG:boto:Data: 2023-10-17T18:54:16.294 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.294 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.295 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.295 DEBUG:boto:Params: {} 2023-10-17T18:54:16.295 DEBUG:boto:Token: None 2023-10-17T18:54:16.296 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e47a216c8a8fd22-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.300 DEBUG:boto:b'hbmuog-58obj000000000000000000141000falseobj000000000000000000142023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.301 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.301 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.302 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.302 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.303 DEBUG:boto:Method: GET 2023-10-17T18:54:16.303 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.303 DEBUG:boto:Data: 2023-10-17T18:54:16.304 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.304 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.305 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.305 DEBUG:boto:Params: {} 2023-10-17T18:54:16.305 DEBUG:boto:Token: None 2023-10-17T18:54:16.306 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.306 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002006df6cc651bf02-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.312 DEBUG:boto:b'hbmuog-58obj000000000000000000141000falseobj000000000000000000142023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.313 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.313 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.314 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.314 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.314 DEBUG:boto:Data: 2023-10-17T18:54:16.315 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.315 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.316 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.316 DEBUG:boto:Params: {} 2023-10-17T18:54:16.316 DEBUG:boto:Token: None 2023-10-17T18:54:16.317 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.317 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IsmScecBH/DSFMVISe+BViWWUQs= 2023-10-17T18:54:16.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IsmScecBH/DSFMVISe+BViWWUQs=', 'Content-Length': '0'} 2023-10-17T18:54:16.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d00c326462cd20-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.320 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.321 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.321 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.321 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.322 DEBUG:boto:Data: 2023-10-17T18:54:16.322 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.323 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.323 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.323 DEBUG:boto:Params: {} 2023-10-17T18:54:16.324 DEBUG:boto:Token: None 2023-10-17T18:54:16.324 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IsmScecBH/DSFMVISe+BViWWUQs= 2023-10-17T18:54:16.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IsmScecBH/DSFMVISe+BViWWUQs=', 'Content-Length': '0'} 2023-10-17T18:54:16.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b3f251f906cc59-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-17T18:54:16.328 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.328 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.329 DEBUG:boto:Method: GET 2023-10-17T18:54:16.329 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.330 DEBUG:boto:Data: 2023-10-17T18:54:16.330 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.331 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.331 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.331 DEBUG:boto:Params: {} 2023-10-17T18:54:16.332 DEBUG:boto:Token: None 2023-10-17T18:54:16.332 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.333 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g2quvJoA89gH9MF9M6cq+FWDLQc= 2023-10-17T18:54:16.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g2quvJoA89gH9MF9M6cq+FWDLQc=', 'Content-Length': '0'} 2023-10-17T18:54:16.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2aef8fbafff6fab-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.378 DEBUG:boto:path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.379 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.379 DEBUG:boto:Method: GET 2023-10-17T18:54:16.380 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.380 DEBUG:boto:Data: 2023-10-17T18:54:16.381 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.381 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.382 DEBUG:boto:Params: {} 2023-10-17T18:54:16.382 DEBUG:boto:Token: None 2023-10-17T18:54:16.383 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000014 2023-10-17T18:54:16.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g2quvJoA89gH9MF9M6cq+FWDLQc= 2023-10-17T18:54:16.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g2quvJoA89gH9MF9M6cq+FWDLQc=', 'Content-Length': '0'} 2023-10-17T18:54:16.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536f3356056b3509-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.387 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.387 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.388 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.388 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.388 DEBUG:boto:Method: GET 2023-10-17T18:54:16.389 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.389 DEBUG:boto:Data: 2023-10-17T18:54:16.390 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.391 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.391 DEBUG:boto:Params: {} 2023-10-17T18:54:16.391 DEBUG:boto:Token: None 2023-10-17T18:54:16.392 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005536a42c17cf56b8-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.396 DEBUG:boto:b'hbmuog-58obj000000000000000000141000falseobj000000000000000000142023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.397 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.397 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.398 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.398 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.398 DEBUG:boto:Method: GET 2023-10-17T18:54:16.399 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000014 2023-10-17T18:54:16.399 DEBUG:boto:Data: 2023-10-17T18:54:16.400 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.401 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.401 DEBUG:boto:Params: {} 2023-10-17T18:54:16.401 DEBUG:boto:Token: None 2023-10-17T18:54:16.402 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc21b9bba3cdb28-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.450 DEBUG:boto:b'hbmuog-58obj000000000000000000141000falseobj000000000000000000142023-10-17T18:54:08.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-17T18:54:16.451 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.452 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.452 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.453 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.453 DEBUG:boto:Method: GET 2023-10-17T18:54:16.453 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.454 DEBUG:boto:Data: 2023-10-17T18:54:16.454 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.455 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.455 DEBUG:boto:Params: {} 2023-10-17T18:54:16.456 DEBUG:boto:Token: None 2023-10-17T18:54:16.456 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g= 2023-10-17T18:54:16.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g=', 'Content-Length': '0'} 2023-10-17T18:54:16.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005614adbcf6e34e86-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.460 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.461 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.461 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.462 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.462 DEBUG:boto:Method: GET 2023-10-17T18:54:16.463 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.463 DEBUG:boto:Data: 2023-10-17T18:54:16.463 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.464 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.464 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.465 DEBUG:boto:Params: {} 2023-10-17T18:54:16.465 DEBUG:boto:Token: None 2023-10-17T18:54:16.466 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.466 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g= 2023-10-17T18:54:16.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g=', 'Content-Length': '0'} 2023-10-17T18:54:16.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eeb18ddddaf7d98-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.469 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.470 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.470 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.471 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.471 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.472 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.472 DEBUG:boto:Data: 2023-10-17T18:54:16.472 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.473 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.473 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.474 DEBUG:boto:Params: {} 2023-10-17T18:54:16.474 DEBUG:boto:Token: None 2023-10-17T18:54:16.474 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.475 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b2KCsTgX5Ih/4AUWaaZr86pmomI= 2023-10-17T18:54:16.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b2KCsTgX5Ih/4AUWaaZr86pmomI=', 'Content-Length': '0'} 2023-10-17T18:54:16.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001015dd03566affa9-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.478 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.478 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.479 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.479 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.480 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.480 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.480 DEBUG:boto:Data: 2023-10-17T18:54:16.481 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.481 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.482 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.482 DEBUG:boto:Params: {} 2023-10-17T18:54:16.483 DEBUG:boto:Token: None 2023-10-17T18:54:16.483 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.483 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:b2KCsTgX5Ih/4AUWaaZr86pmomI= 2023-10-17T18:54:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:b2KCsTgX5Ih/4AUWaaZr86pmomI=', 'Content-Length': '0'} 2023-10-17T18:54:16.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48e593e28d3e006-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-17T18:54:16.487 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.487 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.488 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.488 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.489 DEBUG:boto:Method: GET 2023-10-17T18:54:16.489 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.489 DEBUG:boto:Data: 2023-10-17T18:54:16.490 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.490 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.491 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.491 DEBUG:boto:Params: {} 2023-10-17T18:54:16.492 DEBUG:boto:Token: None 2023-10-17T18:54:16.492 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.492 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g= 2023-10-17T18:54:16.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g=', 'Content-Length': '0'} 2023-10-17T18:54:16.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033237bd470a040ad-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.495 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.496 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.496 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.497 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.497 DEBUG:boto:Method: GET 2023-10-17T18:54:16.497 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.498 DEBUG:boto:Data: 2023-10-17T18:54:16.498 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.499 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.499 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.499 DEBUG:boto:Params: {} 2023-10-17T18:54:16.500 DEBUG:boto:Token: None 2023-10-17T18:54:16.500 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015?versionId=null 2023-10-17T18:54:16.501 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g= 2023-10-17T18:54:16.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4pYTIDpRSxsVwEMA8Z2YZoxOm5g=', 'Content-Length': '0'} 2023-10-17T18:54:16.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b75d2083aa51fc-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.504 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.504 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.505 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.505 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.505 DEBUG:boto:Method: GET 2023-10-17T18:54:16.506 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.506 DEBUG:boto:Data: 2023-10-17T18:54:16.507 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.507 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.508 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.508 DEBUG:boto:Params: {} 2023-10-17T18:54:16.508 DEBUG:boto:Token: None 2023-10-17T18:54:16.509 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de3fef86f618fb3-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.554 DEBUG:boto:b'hbmuog-58obj000000000000000000151000falseobj000000000000000000152023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.555 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.555 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.556 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.556 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.557 DEBUG:boto:Method: GET 2023-10-17T18:54:16.557 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.557 DEBUG:boto:Data: 2023-10-17T18:54:16.558 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.559 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.559 DEBUG:boto:Params: {} 2023-10-17T18:54:16.559 DEBUG:boto:Token: None 2023-10-17T18:54:16.560 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074e1558117e7c94-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.565 DEBUG:boto:b'hbmuog-58obj000000000000000000151000falseobj000000000000000000152023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.566 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.566 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.567 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.567 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.568 DEBUG:boto:Data: 2023-10-17T18:54:16.568 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.569 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.569 DEBUG:boto:Params: {} 2023-10-17T18:54:16.570 DEBUG:boto:Token: None 2023-10-17T18:54:16.570 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6Zrwuxy31r1ubdGh1QVOW/FRA4= 2023-10-17T18:54:16.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6Zrwuxy31r1ubdGh1QVOW/FRA4=', 'Content-Length': '0'} 2023-10-17T18:54:16.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002764db217f4a27e3-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.574 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.574 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.575 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.575 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.575 DEBUG:boto:Data: 2023-10-17T18:54:16.576 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.576 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.577 DEBUG:boto:Params: {} 2023-10-17T18:54:16.577 DEBUG:boto:Token: None 2023-10-17T18:54:16.578 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6Zrwuxy31r1ubdGh1QVOW/FRA4= 2023-10-17T18:54:16.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6Zrwuxy31r1ubdGh1QVOW/FRA4=', 'Content-Length': '0'} 2023-10-17T18:54:16.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c01dd1eccee4f7-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-17T18:54:16.582 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.582 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.583 DEBUG:boto:Method: GET 2023-10-17T18:54:16.583 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.584 DEBUG:boto:Data: 2023-10-17T18:54:16.584 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.584 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.585 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.585 DEBUG:boto:Params: {} 2023-10-17T18:54:16.586 DEBUG:boto:Token: None 2023-10-17T18:54:16.586 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.586 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jjaATidR7J1Ks4QRZyZiMqYPXDo= 2023-10-17T18:54:16.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jjaATidR7J1Ks4QRZyZiMqYPXDo=', 'Content-Length': '0'} 2023-10-17T18:54:16.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057952b1aa62ea73d-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.589 DEBUG:boto:path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.590 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.590 DEBUG:boto:Method: GET 2023-10-17T18:54:16.591 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.591 DEBUG:boto:Data: 2023-10-17T18:54:16.592 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.592 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.592 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.593 DEBUG:boto:Params: {} 2023-10-17T18:54:16.593 DEBUG:boto:Token: None 2023-10-17T18:54:16.594 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000015 2023-10-17T18:54:16.594 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jjaATidR7J1Ks4QRZyZiMqYPXDo= 2023-10-17T18:54:16.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jjaATidR7J1Ks4QRZyZiMqYPXDo=', 'Content-Length': '0'} 2023-10-17T18:54:16.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e836afb51d4bf3-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.597 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.598 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.598 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.599 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.599 DEBUG:boto:Method: GET 2023-10-17T18:54:16.600 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.600 DEBUG:boto:Data: 2023-10-17T18:54:16.600 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.601 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.601 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.602 DEBUG:boto:Params: {} 2023-10-17T18:54:16.602 DEBUG:boto:Token: None 2023-10-17T18:54:16.603 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.603 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004768154366f8ffda-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.607 DEBUG:boto:b'hbmuog-58obj000000000000000000151000falseobj000000000000000000152023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.607 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.608 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.608 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.609 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.609 DEBUG:boto:Method: GET 2023-10-17T18:54:16.609 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000015 2023-10-17T18:54:16.610 DEBUG:boto:Data: 2023-10-17T18:54:16.610 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.611 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.612 DEBUG:boto:Params: {} 2023-10-17T18:54:16.612 DEBUG:boto:Token: None 2023-10-17T18:54:16.613 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dadd3b7a796651e5-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.618 DEBUG:boto:b'hbmuog-58obj000000000000000000151000falseobj000000000000000000152023-10-17T18:54:08.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-17T18:54:16.619 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.620 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.620 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.620 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.621 DEBUG:boto:Method: GET 2023-10-17T18:54:16.621 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.622 DEBUG:boto:Data: 2023-10-17T18:54:16.622 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.623 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.624 DEBUG:boto:Params: {} 2023-10-17T18:54:16.624 DEBUG:boto:Token: None 2023-10-17T18:54:16.624 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo= 2023-10-17T18:54:16.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo=', 'Content-Length': '0'} 2023-10-17T18:54:16.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35ec71c794c7ca0-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.628 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.629 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.629 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.630 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.630 DEBUG:boto:Method: GET 2023-10-17T18:54:16.631 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.631 DEBUG:boto:Data: 2023-10-17T18:54:16.631 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.632 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.633 DEBUG:boto:Params: {} 2023-10-17T18:54:16.633 DEBUG:boto:Token: None 2023-10-17T18:54:16.634 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo= 2023-10-17T18:54:16.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo=', 'Content-Length': '0'} 2023-10-17T18:54:16.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e17a7c1adb0dfb9-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.637 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.638 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.638 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.639 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.639 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.639 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.640 DEBUG:boto:Data: 2023-10-17T18:54:16.640 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.641 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.641 DEBUG:boto:Params: {} 2023-10-17T18:54:16.642 DEBUG:boto:Token: None 2023-10-17T18:54:16.642 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mZ6dWOyLpombPduYR6ee0tARBV4= 2023-10-17T18:54:16.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mZ6dWOyLpombPduYR6ee0tARBV4=', 'Content-Length': '0'} 2023-10-17T18:54:16.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06bb0369a509bff-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.646 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.646 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.647 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.647 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.647 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.648 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.648 DEBUG:boto:Data: 2023-10-17T18:54:16.649 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.649 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.650 DEBUG:boto:Params: {} 2023-10-17T18:54:16.650 DEBUG:boto:Token: None 2023-10-17T18:54:16.651 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mZ6dWOyLpombPduYR6ee0tARBV4= 2023-10-17T18:54:16.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mZ6dWOyLpombPduYR6ee0tARBV4=', 'Content-Length': '0'} 2023-10-17T18:54:16.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8aee645c266dacd-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-17T18:54:16.654 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.655 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.655 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.656 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.656 DEBUG:boto:Method: GET 2023-10-17T18:54:16.657 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.657 DEBUG:boto:Data: 2023-10-17T18:54:16.657 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.658 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.659 DEBUG:boto:Params: {} 2023-10-17T18:54:16.659 DEBUG:boto:Token: None 2023-10-17T18:54:16.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.660 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo= 2023-10-17T18:54:16.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo=', 'Content-Length': '0'} 2023-10-17T18:54:16.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321047774d4785b1-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.663 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.664 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.664 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.665 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.665 DEBUG:boto:Method: GET 2023-10-17T18:54:16.665 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.666 DEBUG:boto:Data: 2023-10-17T18:54:16.666 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.667 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.667 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.668 DEBUG:boto:Params: {} 2023-10-17T18:54:16.668 DEBUG:boto:Token: None 2023-10-17T18:54:16.668 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016?versionId=null 2023-10-17T18:54:16.669 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo= 2023-10-17T18:54:16.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uz7qpkQKsrANpF5g1LLJoYqNwzo=', 'Content-Length': '0'} 2023-10-17T18:54:16.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089a9a82a1c1a72e3-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.673 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.673 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.674 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.674 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.674 DEBUG:boto:Method: GET 2023-10-17T18:54:16.675 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.675 DEBUG:boto:Data: 2023-10-17T18:54:16.676 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.676 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.677 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.677 DEBUG:boto:Params: {} 2023-10-17T18:54:16.677 DEBUG:boto:Token: None 2023-10-17T18:54:16.678 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.678 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ecc7bccca984d9-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.726 DEBUG:boto:b'hbmuog-58obj000000000000000000161000falseobj000000000000000000162023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.727 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.728 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.728 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.729 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.729 DEBUG:boto:Method: GET 2023-10-17T18:54:16.729 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.730 DEBUG:boto:Data: 2023-10-17T18:54:16.730 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.731 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.731 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.732 DEBUG:boto:Params: {} 2023-10-17T18:54:16.732 DEBUG:boto:Token: None 2023-10-17T18:54:16.733 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.733 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b62be8532096690a-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.738 DEBUG:boto:b'hbmuog-58obj000000000000000000161000falseobj000000000000000000162023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.739 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.739 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.740 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.740 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.741 DEBUG:boto:Data: 2023-10-17T18:54:16.741 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.742 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.742 DEBUG:boto:Params: {} 2023-10-17T18:54:16.743 DEBUG:boto:Token: None 2023-10-17T18:54:16.743 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bx+1K3tNEMJc3l/xjWYY5ruz9dw= 2023-10-17T18:54:16.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bx+1K3tNEMJc3l/xjWYY5ruz9dw=', 'Content-Length': '0'} 2023-10-17T18:54:16.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000048e9dac4be8de0a-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.747 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.747 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.748 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.748 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.749 DEBUG:boto:Data: 2023-10-17T18:54:16.749 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.750 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.750 DEBUG:boto:Params: {} 2023-10-17T18:54:16.751 DEBUG:boto:Token: None 2023-10-17T18:54:16.751 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bx+1K3tNEMJc3l/xjWYY5ruz9dw= 2023-10-17T18:54:16.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bx+1K3tNEMJc3l/xjWYY5ruz9dw=', 'Content-Length': '0'} 2023-10-17T18:54:16.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d6c368421462b6-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-17T18:54:16.755 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.755 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.756 DEBUG:boto:Method: GET 2023-10-17T18:54:16.756 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.757 DEBUG:boto:Data: 2023-10-17T18:54:16.757 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.758 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.759 DEBUG:boto:Params: {} 2023-10-17T18:54:16.759 DEBUG:boto:Token: None 2023-10-17T18:54:16.760 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pwfm/bmNUp71uu8nN+XX3LLt184= 2023-10-17T18:54:16.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pwfm/bmNUp71uu8nN+XX3LLt184=', 'Content-Length': '0'} 2023-10-17T18:54:16.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb1e99dabfbdd54-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.764 DEBUG:boto:path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.764 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.765 DEBUG:boto:Method: GET 2023-10-17T18:54:16.765 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.766 DEBUG:boto:Data: 2023-10-17T18:54:16.766 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.767 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.767 DEBUG:boto:Params: {} 2023-10-17T18:54:16.768 DEBUG:boto:Token: None 2023-10-17T18:54:16.768 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000016 2023-10-17T18:54:16.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pwfm/bmNUp71uu8nN+XX3LLt184= 2023-10-17T18:54:16.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pwfm/bmNUp71uu8nN+XX3LLt184=', 'Content-Length': '0'} 2023-10-17T18:54:16.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25c31867c018ca4-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.772 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.773 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.773 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.774 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.774 DEBUG:boto:Method: GET 2023-10-17T18:54:16.775 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.775 DEBUG:boto:Data: 2023-10-17T18:54:16.776 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.776 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.776 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.777 DEBUG:boto:Params: {} 2023-10-17T18:54:16.777 DEBUG:boto:Token: None 2023-10-17T18:54:16.778 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d12959ba9eed509-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.822 DEBUG:boto:b'hbmuog-58obj000000000000000000161000falseobj000000000000000000162023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.823 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.824 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.824 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.825 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.825 DEBUG:boto:Method: GET 2023-10-17T18:54:16.826 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000016 2023-10-17T18:54:16.826 DEBUG:boto:Data: 2023-10-17T18:54:16.827 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.827 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.828 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.828 DEBUG:boto:Params: {} 2023-10-17T18:54:16.829 DEBUG:boto:Token: None 2023-10-17T18:54:16.829 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf420245b0419dab-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.835 DEBUG:boto:b'hbmuog-58obj000000000000000000161000falseobj000000000000000000162023-10-17T18:54:07.996Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-17T18:54:16.836 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.837 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.837 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.838 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.838 DEBUG:boto:Method: GET 2023-10-17T18:54:16.839 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.839 DEBUG:boto:Data: 2023-10-17T18:54:16.840 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.840 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.841 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.841 DEBUG:boto:Params: {} 2023-10-17T18:54:16.842 DEBUG:boto:Token: None 2023-10-17T18:54:16.842 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U= 2023-10-17T18:54:16.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U=', 'Content-Length': '0'} 2023-10-17T18:54:16.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d766a42bcd0029-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.846 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.847 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.847 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.848 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.848 DEBUG:boto:Method: GET 2023-10-17T18:54:16.849 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.849 DEBUG:boto:Data: 2023-10-17T18:54:16.850 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.851 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.851 DEBUG:boto:Params: {} 2023-10-17T18:54:16.852 DEBUG:boto:Token: None 2023-10-17T18:54:16.852 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.853 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U= 2023-10-17T18:54:16.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U=', 'Content-Length': '0'} 2023-10-17T18:54:16.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9dada8737194d7f-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.898 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.899 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.899 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.900 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.900 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.900 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.901 DEBUG:boto:Data: 2023-10-17T18:54:16.901 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.902 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.902 DEBUG:boto:Params: {} 2023-10-17T18:54:16.903 DEBUG:boto:Token: None 2023-10-17T18:54:16.903 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SZb0DCsw6UQq90WGIin6elgYhD4= 2023-10-17T18:54:16.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SZb0DCsw6UQq90WGIin6elgYhD4=', 'Content-Length': '0'} 2023-10-17T18:54:16.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b549762afebc94d-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.907 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.907 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.908 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.908 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.908 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.909 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.909 DEBUG:boto:Data: 2023-10-17T18:54:16.910 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.911 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.911 DEBUG:boto:Params: {} 2023-10-17T18:54:16.911 DEBUG:boto:Token: None 2023-10-17T18:54:16.912 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SZb0DCsw6UQq90WGIin6elgYhD4= 2023-10-17T18:54:16.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SZb0DCsw6UQq90WGIin6elgYhD4=', 'Content-Length': '0'} 2023-10-17T18:54:16.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40b08b610a85c10-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-17T18:54:16.916 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.916 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.916 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.917 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.917 DEBUG:boto:Method: GET 2023-10-17T18:54:16.918 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.918 DEBUG:boto:Data: 2023-10-17T18:54:16.919 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.919 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.919 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.920 DEBUG:boto:Params: {} 2023-10-17T18:54:16.920 DEBUG:boto:Token: None 2023-10-17T18:54:16.921 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U= 2023-10-17T18:54:16.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U=', 'Content-Length': '0'} 2023-10-17T18:54:16.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b99fe0427b9799-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.924 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.925 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.925 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.926 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.926 DEBUG:boto:Method: GET 2023-10-17T18:54:16.926 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.927 DEBUG:boto:Data: 2023-10-17T18:54:16.927 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.928 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.928 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.928 DEBUG:boto:Params: {} 2023-10-17T18:54:16.929 DEBUG:boto:Token: None 2023-10-17T18:54:16.929 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017?versionId=null 2023-10-17T18:54:16.930 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U= 2023-10-17T18:54:16.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oXi9NWnsdIHpgK2B9qt4fEfBI/U=', 'Content-Length': '0'} 2023-10-17T18:54:16.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f19651cdc16691d-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.933 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.933 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.934 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.934 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.935 DEBUG:boto:Method: GET 2023-10-17T18:54:16.935 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.936 DEBUG:boto:Data: 2023-10-17T18:54:16.936 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.936 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.937 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.937 DEBUG:boto:Params: {} 2023-10-17T18:54:16.938 DEBUG:boto:Token: None 2023-10-17T18:54:16.938 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b30b5dc4925cb5f-00652ed858-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.942 DEBUG:boto:b'hbmuog-58obj000000000000000000171000falseobj000000000000000000172023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.943 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:16.943 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:16.944 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.944 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.944 DEBUG:boto:Method: GET 2023-10-17T18:54:16.945 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:16.945 DEBUG:boto:Data: 2023-10-17T18:54:16.946 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.946 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.947 DEBUG:boto:Params: {} 2023-10-17T18:54:16.947 DEBUG:boto:Token: None 2023-10-17T18:54:16.948 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/ 2023-10-17T18:54:16.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs= 2023-10-17T18:54:16.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gB7tMhcVjxNkzbnyiohHaIch5Xs=', 'Content-Length': '0'} 2023-10-17T18:54:16.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa624e8f8ecb2fe-00652ed858-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.954 DEBUG:boto:b'hbmuog-58obj000000000000000000171000falseobj000000000000000000172023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:16.954 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.955 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.955 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.956 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.956 DEBUG:boto:Data: 2023-10-17T18:54:16.957 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.957 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.957 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.958 DEBUG:boto:Params: {} 2023-10-17T18:54:16.958 DEBUG:boto:Token: None 2023-10-17T18:54:16.959 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tpJH4i16XllvDxIRh4TgOhMVgJ0= 2023-10-17T18:54:16.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tpJH4i16XllvDxIRh4TgOhMVgJ0=', 'Content-Length': '0'} 2023-10-17T18:54:16.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0fcfa3b5d302e8e-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.962 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.963 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.963 DEBUG:boto:Method: HEAD 2023-10-17T18:54:16.964 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.964 DEBUG:boto:Data: 2023-10-17T18:54:16.964 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.965 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.965 DEBUG:boto:Params: {} 2023-10-17T18:54:16.966 DEBUG:boto:Token: None 2023-10-17T18:54:16.966 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tpJH4i16XllvDxIRh4TgOhMVgJ0= 2023-10-17T18:54:16.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tpJH4i16XllvDxIRh4TgOhMVgJ0=', 'Content-Length': '0'} 2023-10-17T18:54:16.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfa50d697bbe01f-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-17T18:54:16.970 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.970 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.971 DEBUG:boto:Method: GET 2023-10-17T18:54:16.971 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.971 DEBUG:boto:Data: 2023-10-17T18:54:16.972 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:16.973 DEBUG:boto:Port: 8001 2023-10-17T18:54:16.973 DEBUG:boto:Params: {} 2023-10-17T18:54:16.973 DEBUG:boto:Token: None 2023-10-17T18:54:16.974 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:41K9LjYPSnRX8eAOmh6/eQz6y7Y= 2023-10-17T18:54:16.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:41K9LjYPSnRX8eAOmh6/eQz6y7Y=', 'Content-Length': '0'} 2023-10-17T18:54:16.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008a1f9cbe1a5e379-00652ed858-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:16.977 DEBUG:boto:path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.978 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.978 DEBUG:boto:Method: GET 2023-10-17T18:54:16.979 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.979 DEBUG:boto:Data: 2023-10-17T18:54:16.979 DEBUG:boto:Headers: {} 2023-10-17T18:54:16.980 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:16.980 DEBUG:boto:Port: 8000 2023-10-17T18:54:16.981 DEBUG:boto:Params: {} 2023-10-17T18:54:16.981 DEBUG:boto:Token: None 2023-10-17T18:54:16.981 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:16 GMT /hbmuog-58/obj00000000000000000017 2023-10-17T18:54:16.982 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:41K9LjYPSnRX8eAOmh6/eQz6y7Y= 2023-10-17T18:54:16.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:41K9LjYPSnRX8eAOmh6/eQz6y7Y=', 'Content-Length': '0'} 2023-10-17T18:54:16.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df53bcb867ca6c65-00652ed858-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.027 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.027 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.028 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.028 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.028 DEBUG:boto:Method: GET 2023-10-17T18:54:17.029 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.029 DEBUG:boto:Data: 2023-10-17T18:54:17.030 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.031 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.031 DEBUG:boto:Params: {} 2023-10-17T18:54:17.032 DEBUG:boto:Token: None 2023-10-17T18:54:17.032 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf06770786dcc82-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.036 DEBUG:boto:b'hbmuog-58obj000000000000000000171000falseobj000000000000000000172023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.037 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.037 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.038 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.038 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.039 DEBUG:boto:Method: GET 2023-10-17T18:54:17.039 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000017 2023-10-17T18:54:17.040 DEBUG:boto:Data: 2023-10-17T18:54:17.040 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.041 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.041 DEBUG:boto:Params: {} 2023-10-17T18:54:17.042 DEBUG:boto:Token: None 2023-10-17T18:54:17.042 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.043 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96ef3b2a0ca7a43-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.048 DEBUG:boto:b'hbmuog-58obj000000000000000000171000falseobj000000000000000000172023-10-17T18:54:07.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-17T18:54:17.049 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.050 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.050 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.050 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.051 DEBUG:boto:Method: GET 2023-10-17T18:54:17.051 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.052 DEBUG:boto:Data: 2023-10-17T18:54:17.052 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.053 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.053 DEBUG:boto:Params: {} 2023-10-17T18:54:17.054 DEBUG:boto:Token: None 2023-10-17T18:54:17.054 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.055 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk= 2023-10-17T18:54:17.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk=', 'Content-Length': '0'} 2023-10-17T18:54:17.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49dc70c727abc29-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.098 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.099 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.099 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.100 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.100 DEBUG:boto:Method: GET 2023-10-17T18:54:17.101 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.101 DEBUG:boto:Data: 2023-10-17T18:54:17.101 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.102 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.103 DEBUG:boto:Params: {} 2023-10-17T18:54:17.103 DEBUG:boto:Token: None 2023-10-17T18:54:17.104 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk= 2023-10-17T18:54:17.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk=', 'Content-Length': '0'} 2023-10-17T18:54:17.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef5980297c781024-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.107 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.108 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.108 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.109 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.109 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.109 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.110 DEBUG:boto:Data: 2023-10-17T18:54:17.110 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.111 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.111 DEBUG:boto:Params: {} 2023-10-17T18:54:17.112 DEBUG:boto:Token: None 2023-10-17T18:54:17.112 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYYpHpFkzLDoBiMPQnjRSJJIy14= 2023-10-17T18:54:17.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYYpHpFkzLDoBiMPQnjRSJJIy14=', 'Content-Length': '0'} 2023-10-17T18:54:17.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011468a0c170a2a6-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.116 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.116 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.117 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.117 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.117 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.118 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.118 DEBUG:boto:Data: 2023-10-17T18:54:17.119 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.119 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.120 DEBUG:boto:Params: {} 2023-10-17T18:54:17.120 DEBUG:boto:Token: None 2023-10-17T18:54:17.121 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYYpHpFkzLDoBiMPQnjRSJJIy14= 2023-10-17T18:54:17.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYYpHpFkzLDoBiMPQnjRSJJIy14=', 'Content-Length': '0'} 2023-10-17T18:54:17.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b2ff2b4c5105cb-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-17T18:54:17.125 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.125 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.126 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.126 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.127 DEBUG:boto:Method: GET 2023-10-17T18:54:17.127 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.127 DEBUG:boto:Data: 2023-10-17T18:54:17.128 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.128 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.129 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.129 DEBUG:boto:Params: {} 2023-10-17T18:54:17.130 DEBUG:boto:Token: None 2023-10-17T18:54:17.130 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk= 2023-10-17T18:54:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk=', 'Content-Length': '0'} 2023-10-17T18:54:17.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e63b9f250ff112-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.134 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.134 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.135 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.135 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.135 DEBUG:boto:Method: GET 2023-10-17T18:54:17.136 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.136 DEBUG:boto:Data: 2023-10-17T18:54:17.137 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.137 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.138 DEBUG:boto:Params: {} 2023-10-17T18:54:17.138 DEBUG:boto:Token: None 2023-10-17T18:54:17.139 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018?versionId=null 2023-10-17T18:54:17.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk= 2023-10-17T18:54:17.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ue8ju+x9GuAQfs6pkwKMyIPKlkk=', 'Content-Length': '0'} 2023-10-17T18:54:17.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008823ceacad1e5a8c-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.142 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.143 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.143 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.144 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.144 DEBUG:boto:Method: GET 2023-10-17T18:54:17.144 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.145 DEBUG:boto:Data: 2023-10-17T18:54:17.145 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.146 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.147 DEBUG:boto:Params: {} 2023-10-17T18:54:17.147 DEBUG:boto:Token: None 2023-10-17T18:54:17.147 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.148 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea7ecf59acc67f09-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.151 DEBUG:boto:b'hbmuog-58obj000000000000000000181000falseobj000000000000000000182023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.152 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.153 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.153 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.153 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.154 DEBUG:boto:Method: GET 2023-10-17T18:54:17.154 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.155 DEBUG:boto:Data: 2023-10-17T18:54:17.155 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.156 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.156 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.156 DEBUG:boto:Params: {} 2023-10-17T18:54:17.157 DEBUG:boto:Token: None 2023-10-17T18:54:17.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.158 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb7c7534e201930-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.163 DEBUG:boto:b'hbmuog-58obj000000000000000000181000falseobj000000000000000000182023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.164 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.164 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.164 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.165 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.165 DEBUG:boto:Data: 2023-10-17T18:54:17.166 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.166 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.167 DEBUG:boto:Params: {} 2023-10-17T18:54:17.167 DEBUG:boto:Token: None 2023-10-17T18:54:17.168 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.168 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bRU7vZQPqcssygEibp0ZtHdBBis= 2023-10-17T18:54:17.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bRU7vZQPqcssygEibp0ZtHdBBis=', 'Content-Length': '0'} 2023-10-17T18:54:17.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023aa18fd59f55802-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.171 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.172 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.172 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.172 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.173 DEBUG:boto:Data: 2023-10-17T18:54:17.173 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.174 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.174 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.174 DEBUG:boto:Params: {} 2023-10-17T18:54:17.175 DEBUG:boto:Token: None 2023-10-17T18:54:17.175 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.176 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bRU7vZQPqcssygEibp0ZtHdBBis= 2023-10-17T18:54:17.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bRU7vZQPqcssygEibp0ZtHdBBis=', 'Content-Length': '0'} 2023-10-17T18:54:17.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb909bd717c2dcf-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-17T18:54:17.179 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.180 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.180 DEBUG:boto:Method: GET 2023-10-17T18:54:17.181 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.181 DEBUG:boto:Data: 2023-10-17T18:54:17.181 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.182 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.182 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.183 DEBUG:boto:Params: {} 2023-10-17T18:54:17.183 DEBUG:boto:Token: None 2023-10-17T18:54:17.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.184 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tyWbxddPpmVEZb3xoetQmlLiYZs= 2023-10-17T18:54:17.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tyWbxddPpmVEZb3xoetQmlLiYZs=', 'Content-Length': '0'} 2023-10-17T18:54:17.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd662ef8899dc18-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.230 DEBUG:boto:path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.231 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.231 DEBUG:boto:Method: GET 2023-10-17T18:54:17.232 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.232 DEBUG:boto:Data: 2023-10-17T18:54:17.232 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.233 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.233 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.234 DEBUG:boto:Params: {} 2023-10-17T18:54:17.234 DEBUG:boto:Token: None 2023-10-17T18:54:17.235 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000018 2023-10-17T18:54:17.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tyWbxddPpmVEZb3xoetQmlLiYZs= 2023-10-17T18:54:17.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tyWbxddPpmVEZb3xoetQmlLiYZs=', 'Content-Length': '0'} 2023-10-17T18:54:17.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b61c7a6036dd8962-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.239 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.239 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.240 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.240 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.240 DEBUG:boto:Method: GET 2023-10-17T18:54:17.241 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.241 DEBUG:boto:Data: 2023-10-17T18:54:17.241 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.242 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.242 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.243 DEBUG:boto:Params: {} 2023-10-17T18:54:17.243 DEBUG:boto:Token: None 2023-10-17T18:54:17.244 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76ce969c0ef7ffa-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.248 DEBUG:boto:b'hbmuog-58obj000000000000000000181000falseobj000000000000000000182023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.248 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.249 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.249 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.250 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.250 DEBUG:boto:Method: GET 2023-10-17T18:54:17.250 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000018 2023-10-17T18:54:17.251 DEBUG:boto:Data: 2023-10-17T18:54:17.251 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.252 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.252 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.252 DEBUG:boto:Params: {} 2023-10-17T18:54:17.253 DEBUG:boto:Token: None 2023-10-17T18:54:17.253 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.254 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066bf7e3230efc216-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.259 DEBUG:boto:b'hbmuog-58obj000000000000000000181000falseobj000000000000000000182023-10-17T18:54:07.816Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-17T18:54:17.260 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.260 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.261 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.261 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.262 DEBUG:boto:Method: GET 2023-10-17T18:54:17.262 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.263 DEBUG:boto:Data: 2023-10-17T18:54:17.263 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.264 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.264 DEBUG:boto:Params: {} 2023-10-17T18:54:17.265 DEBUG:boto:Token: None 2023-10-17T18:54:17.265 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.266 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk= 2023-10-17T18:54:17.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk=', 'Content-Length': '0'} 2023-10-17T18:54:17.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ca365be6474031-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.269 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.269 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.270 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.270 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.271 DEBUG:boto:Method: GET 2023-10-17T18:54:17.271 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.272 DEBUG:boto:Data: 2023-10-17T18:54:17.272 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.272 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.273 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.273 DEBUG:boto:Params: {} 2023-10-17T18:54:17.274 DEBUG:boto:Token: None 2023-10-17T18:54:17.274 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk= 2023-10-17T18:54:17.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk=', 'Content-Length': '0'} 2023-10-17T18:54:17.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4a83e32f0aaaa7-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.278 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.278 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.279 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.279 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.281 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.281 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.282 DEBUG:boto:Data: 2023-10-17T18:54:17.282 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.282 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.283 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.283 DEBUG:boto:Params: {} 2023-10-17T18:54:17.284 DEBUG:boto:Token: None 2023-10-17T18:54:17.284 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.285 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uAmys0gYKs0wlrI3ThCp/yqulvo= 2023-10-17T18:54:17.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uAmys0gYKs0wlrI3ThCp/yqulvo=', 'Content-Length': '0'} 2023-10-17T18:54:17.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006867976bb06f7ae8-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.288 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.288 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.288 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.289 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.289 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.290 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.290 DEBUG:boto:Data: 2023-10-17T18:54:17.290 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.291 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.292 DEBUG:boto:Params: {} 2023-10-17T18:54:17.292 DEBUG:boto:Token: None 2023-10-17T18:54:17.293 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.293 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uAmys0gYKs0wlrI3ThCp/yqulvo= 2023-10-17T18:54:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uAmys0gYKs0wlrI3ThCp/yqulvo=', 'Content-Length': '0'} 2023-10-17T18:54:17.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bab0b254627306f-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-17T18:54:17.296 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.297 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.297 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.298 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.298 DEBUG:boto:Method: GET 2023-10-17T18:54:17.298 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.299 DEBUG:boto:Data: 2023-10-17T18:54:17.299 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.300 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.301 DEBUG:boto:Params: {} 2023-10-17T18:54:17.301 DEBUG:boto:Token: None 2023-10-17T18:54:17.301 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.302 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk= 2023-10-17T18:54:17.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk=', 'Content-Length': '0'} 2023-10-17T18:54:17.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9bbe408b63b69e-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.346 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.347 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.347 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.348 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.348 DEBUG:boto:Method: GET 2023-10-17T18:54:17.349 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.349 DEBUG:boto:Data: 2023-10-17T18:54:17.349 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.350 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.350 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.351 DEBUG:boto:Params: {} 2023-10-17T18:54:17.351 DEBUG:boto:Token: None 2023-10-17T18:54:17.352 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019?versionId=null 2023-10-17T18:54:17.352 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk= 2023-10-17T18:54:17.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6qyZESlwMR0ZPjksvU9YPLhIuVk=', 'Content-Length': '0'} 2023-10-17T18:54:17.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958bc0f7be2714a0-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.355 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.356 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.356 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.357 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.357 DEBUG:boto:Method: GET 2023-10-17T18:54:17.357 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.358 DEBUG:boto:Data: 2023-10-17T18:54:17.358 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.359 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.359 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.360 DEBUG:boto:Params: {} 2023-10-17T18:54:17.360 DEBUG:boto:Token: None 2023-10-17T18:54:17.361 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.361 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198db39865dd4f71-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.365 DEBUG:boto:b'hbmuog-58obj000000000000000000191000falseobj000000000000000000192023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.365 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.366 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.366 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.367 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.367 DEBUG:boto:Method: GET 2023-10-17T18:54:17.368 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.368 DEBUG:boto:Data: 2023-10-17T18:54:17.368 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.369 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.370 DEBUG:boto:Params: {} 2023-10-17T18:54:17.370 DEBUG:boto:Token: None 2023-10-17T18:54:17.371 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.371 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e4dfaefe9fddb3-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.418 DEBUG:boto:b'hbmuog-58obj000000000000000000191000falseobj000000000000000000192023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.419 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.420 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.420 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.420 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.421 DEBUG:boto:Data: 2023-10-17T18:54:17.421 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.422 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.422 DEBUG:boto:Params: {} 2023-10-17T18:54:17.423 DEBUG:boto:Token: None 2023-10-17T18:54:17.423 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.424 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QlbcOq+1Pl/KhBpqwqwprq74h/s= 2023-10-17T18:54:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QlbcOq+1Pl/KhBpqwqwprq74h/s=', 'Content-Length': '0'} 2023-10-17T18:54:17.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076818c280c2cbd5c-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.427 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.427 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.428 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.428 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.429 DEBUG:boto:Data: 2023-10-17T18:54:17.429 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.429 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.430 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.430 DEBUG:boto:Params: {} 2023-10-17T18:54:17.431 DEBUG:boto:Token: None 2023-10-17T18:54:17.431 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.431 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QlbcOq+1Pl/KhBpqwqwprq74h/s= 2023-10-17T18:54:17.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QlbcOq+1Pl/KhBpqwqwprq74h/s=', 'Content-Length': '0'} 2023-10-17T18:54:17.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd725fd74a1dca4e-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-17T18:54:17.435 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.435 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.435 DEBUG:boto:Method: GET 2023-10-17T18:54:17.436 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.436 DEBUG:boto:Data: 2023-10-17T18:54:17.437 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.437 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.437 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.438 DEBUG:boto:Params: {} 2023-10-17T18:54:17.438 DEBUG:boto:Token: None 2023-10-17T18:54:17.439 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wnV9C2eVnPiO7URq2SRh23aeffM= 2023-10-17T18:54:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wnV9C2eVnPiO7URq2SRh23aeffM=', 'Content-Length': '0'} 2023-10-17T18:54:17.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6669e829cd03d9-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.442 DEBUG:boto:path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.443 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.443 DEBUG:boto:Method: GET 2023-10-17T18:54:17.444 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.444 DEBUG:boto:Data: 2023-10-17T18:54:17.444 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.445 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.446 DEBUG:boto:Params: {} 2023-10-17T18:54:17.446 DEBUG:boto:Token: None 2023-10-17T18:54:17.446 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000019 2023-10-17T18:54:17.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wnV9C2eVnPiO7URq2SRh23aeffM= 2023-10-17T18:54:17.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wnV9C2eVnPiO7URq2SRh23aeffM=', 'Content-Length': '0'} 2023-10-17T18:54:17.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4d0b4adb0d65d4-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.450 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.451 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.451 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.451 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.452 DEBUG:boto:Method: GET 2023-10-17T18:54:17.452 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.452 DEBUG:boto:Data: 2023-10-17T18:54:17.453 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.454 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.454 DEBUG:boto:Params: {} 2023-10-17T18:54:17.455 DEBUG:boto:Token: None 2023-10-17T18:54:17.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.455 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7fac8e5e34f8492-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.459 DEBUG:boto:b'hbmuog-58obj000000000000000000191000falseobj000000000000000000192023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.459 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.460 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.460 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.461 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.461 DEBUG:boto:Method: GET 2023-10-17T18:54:17.461 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000019 2023-10-17T18:54:17.462 DEBUG:boto:Data: 2023-10-17T18:54:17.462 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.463 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.464 DEBUG:boto:Params: {} 2023-10-17T18:54:17.464 DEBUG:boto:Token: None 2023-10-17T18:54:17.465 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd06579762c3c8e-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.473 DEBUG:boto:b'hbmuog-58obj000000000000000000191000falseobj000000000000000000192023-10-17T18:54:07.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-17T18:54:17.474 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.475 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.475 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.476 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.476 DEBUG:boto:Method: GET 2023-10-17T18:54:17.477 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.477 DEBUG:boto:Data: 2023-10-17T18:54:17.477 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.478 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.479 DEBUG:boto:Params: {} 2023-10-17T18:54:17.479 DEBUG:boto:Token: None 2023-10-17T18:54:17.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I= 2023-10-17T18:54:17.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I=', 'Content-Length': '0'} 2023-10-17T18:54:17.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11c92101103cf9b-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.526 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.527 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.527 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.528 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.528 DEBUG:boto:Method: GET 2023-10-17T18:54:17.528 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.529 DEBUG:boto:Data: 2023-10-17T18:54:17.529 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.530 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.531 DEBUG:boto:Params: {} 2023-10-17T18:54:17.531 DEBUG:boto:Token: None 2023-10-17T18:54:17.531 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I= 2023-10-17T18:54:17.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I=', 'Content-Length': '0'} 2023-10-17T18:54:17.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f74fb04f451946-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.535 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.535 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.536 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.536 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.536 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.537 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.537 DEBUG:boto:Data: 2023-10-17T18:54:17.537 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.538 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.539 DEBUG:boto:Params: {} 2023-10-17T18:54:17.539 DEBUG:boto:Token: None 2023-10-17T18:54:17.539 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XQUAcAayZvc4mx6Db9VIGPxgKxM= 2023-10-17T18:54:17.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XQUAcAayZvc4mx6Db9VIGPxgKxM=', 'Content-Length': '0'} 2023-10-17T18:54:17.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24b689b205651e0-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.543 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.543 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.543 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.544 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.544 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.544 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.545 DEBUG:boto:Data: 2023-10-17T18:54:17.545 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.545 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.546 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.546 DEBUG:boto:Params: {} 2023-10-17T18:54:17.547 DEBUG:boto:Token: None 2023-10-17T18:54:17.547 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.547 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XQUAcAayZvc4mx6Db9VIGPxgKxM= 2023-10-17T18:54:17.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XQUAcAayZvc4mx6Db9VIGPxgKxM=', 'Content-Length': '0'} 2023-10-17T18:54:17.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baa0d7014086b78e-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-17T18:54:17.551 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.551 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.552 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.552 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.552 DEBUG:boto:Method: GET 2023-10-17T18:54:17.553 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.553 DEBUG:boto:Data: 2023-10-17T18:54:17.553 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.554 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.554 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.555 DEBUG:boto:Params: {} 2023-10-17T18:54:17.555 DEBUG:boto:Token: None 2023-10-17T18:54:17.556 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I= 2023-10-17T18:54:17.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I=', 'Content-Length': '0'} 2023-10-17T18:54:17.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ecfa60faa04ac4-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.559 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.560 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.560 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.560 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.561 DEBUG:boto:Method: GET 2023-10-17T18:54:17.561 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.562 DEBUG:boto:Data: 2023-10-17T18:54:17.562 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.562 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.563 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.563 DEBUG:boto:Params: {} 2023-10-17T18:54:17.564 DEBUG:boto:Token: None 2023-10-17T18:54:17.564 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020?versionId=null 2023-10-17T18:54:17.564 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I= 2023-10-17T18:54:17.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7jOqfJlrqBGsFds1KwL54vtCT6I=', 'Content-Length': '0'} 2023-10-17T18:54:17.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c233d20e277b39bc-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.567 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.568 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.568 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.568 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.569 DEBUG:boto:Method: GET 2023-10-17T18:54:17.569 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.570 DEBUG:boto:Data: 2023-10-17T18:54:17.570 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.571 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.571 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.571 DEBUG:boto:Params: {} 2023-10-17T18:54:17.572 DEBUG:boto:Token: None 2023-10-17T18:54:17.572 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004759f99faac36bd-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.576 DEBUG:boto:b'hbmuog-58obj000000000000000000201000falseobj000000000000000000202023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.577 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.577 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.578 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.578 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.579 DEBUG:boto:Method: GET 2023-10-17T18:54:17.579 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.579 DEBUG:boto:Data: 2023-10-17T18:54:17.580 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.580 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.581 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.581 DEBUG:boto:Params: {} 2023-10-17T18:54:17.581 DEBUG:boto:Token: None 2023-10-17T18:54:17.582 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.582 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddecd109d56fff04-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.630 DEBUG:boto:b'hbmuog-58obj000000000000000000201000falseobj000000000000000000202023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.631 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.631 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.632 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.632 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.632 DEBUG:boto:Data: 2023-10-17T18:54:17.633 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.633 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.633 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.634 DEBUG:boto:Params: {} 2023-10-17T18:54:17.634 DEBUG:boto:Token: None 2023-10-17T18:54:17.635 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.635 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dwxqLpI7ZSRkf/p7Pbqnfnwwk7o= 2023-10-17T18:54:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dwxqLpI7ZSRkf/p7Pbqnfnwwk7o=', 'Content-Length': '0'} 2023-10-17T18:54:17.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6677afd8ddd043f-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.638 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.638 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.639 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.639 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.640 DEBUG:boto:Data: 2023-10-17T18:54:17.640 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.641 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.641 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.641 DEBUG:boto:Params: {} 2023-10-17T18:54:17.642 DEBUG:boto:Token: None 2023-10-17T18:54:17.642 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.643 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dwxqLpI7ZSRkf/p7Pbqnfnwwk7o= 2023-10-17T18:54:17.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dwxqLpI7ZSRkf/p7Pbqnfnwwk7o=', 'Content-Length': '0'} 2023-10-17T18:54:17.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d1427f0f85f4c0-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-17T18:54:17.646 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.647 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.647 DEBUG:boto:Method: GET 2023-10-17T18:54:17.647 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.648 DEBUG:boto:Data: 2023-10-17T18:54:17.648 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.649 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.649 DEBUG:boto:Params: {} 2023-10-17T18:54:17.650 DEBUG:boto:Token: None 2023-10-17T18:54:17.650 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KsBxMuF6xMe1RXHB/tdS81WPcy0= 2023-10-17T18:54:17.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KsBxMuF6xMe1RXHB/tdS81WPcy0=', 'Content-Length': '0'} 2023-10-17T18:54:17.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6a17c54cd600de-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.654 DEBUG:boto:path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.654 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.655 DEBUG:boto:Method: GET 2023-10-17T18:54:17.655 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.656 DEBUG:boto:Data: 2023-10-17T18:54:17.656 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.656 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.657 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.657 DEBUG:boto:Params: {} 2023-10-17T18:54:17.657 DEBUG:boto:Token: None 2023-10-17T18:54:17.658 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000020 2023-10-17T18:54:17.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KsBxMuF6xMe1RXHB/tdS81WPcy0= 2023-10-17T18:54:17.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KsBxMuF6xMe1RXHB/tdS81WPcy0=', 'Content-Length': '0'} 2023-10-17T18:54:17.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02d5699206f1bb8-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.662 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.662 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.662 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.663 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.663 DEBUG:boto:Method: GET 2023-10-17T18:54:17.664 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.664 DEBUG:boto:Data: 2023-10-17T18:54:17.664 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.665 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.665 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.666 DEBUG:boto:Params: {} 2023-10-17T18:54:17.666 DEBUG:boto:Token: None 2023-10-17T18:54:17.667 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.667 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c02405a3f95480a-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.671 DEBUG:boto:b'hbmuog-58obj000000000000000000201000falseobj000000000000000000202023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.671 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.672 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.672 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.673 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.673 DEBUG:boto:Method: GET 2023-10-17T18:54:17.673 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000020 2023-10-17T18:54:17.674 DEBUG:boto:Data: 2023-10-17T18:54:17.674 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.675 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.675 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.676 DEBUG:boto:Params: {} 2023-10-17T18:54:17.676 DEBUG:boto:Token: None 2023-10-17T18:54:17.677 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.677 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017cd56aaf4b4e87e-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.683 DEBUG:boto:b'hbmuog-58obj000000000000000000201000falseobj000000000000000000202023-10-17T18:54:07.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-17T18:54:17.684 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.684 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.684 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.685 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.685 DEBUG:boto:Method: GET 2023-10-17T18:54:17.686 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.686 DEBUG:boto:Data: 2023-10-17T18:54:17.687 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.687 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.688 DEBUG:boto:Params: {} 2023-10-17T18:54:17.689 DEBUG:boto:Token: None 2023-10-17T18:54:17.690 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.691 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo= 2023-10-17T18:54:17.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo=', 'Content-Length': '0'} 2023-10-17T18:54:17.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350b3e1c160901cc-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.734 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.735 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.735 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.736 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.736 DEBUG:boto:Method: GET 2023-10-17T18:54:17.737 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.737 DEBUG:boto:Data: 2023-10-17T18:54:17.737 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.738 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.739 DEBUG:boto:Params: {} 2023-10-17T18:54:17.739 DEBUG:boto:Token: None 2023-10-17T18:54:17.740 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo= 2023-10-17T18:54:17.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo=', 'Content-Length': '0'} 2023-10-17T18:54:17.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087aef143be099eba-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.743 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.744 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.744 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.745 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.745 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.745 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.746 DEBUG:boto:Data: 2023-10-17T18:54:17.746 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.747 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.747 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.747 DEBUG:boto:Params: {} 2023-10-17T18:54:17.748 DEBUG:boto:Token: None 2023-10-17T18:54:17.748 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.749 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1zIwrzYYhivFvtxnuotVdjcgPTs= 2023-10-17T18:54:17.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1zIwrzYYhivFvtxnuotVdjcgPTs=', 'Content-Length': '0'} 2023-10-17T18:54:17.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ac48c2d5098fd2-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.751 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.752 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.752 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.753 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.753 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.753 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.754 DEBUG:boto:Data: 2023-10-17T18:54:17.754 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.755 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.755 DEBUG:boto:Params: {} 2023-10-17T18:54:17.756 DEBUG:boto:Token: None 2023-10-17T18:54:17.756 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.757 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1zIwrzYYhivFvtxnuotVdjcgPTs= 2023-10-17T18:54:17.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1zIwrzYYhivFvtxnuotVdjcgPTs=', 'Content-Length': '0'} 2023-10-17T18:54:17.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe64334f4f3c5a8e-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-17T18:54:17.760 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.760 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.761 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.761 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.761 DEBUG:boto:Method: GET 2023-10-17T18:54:17.762 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.762 DEBUG:boto:Data: 2023-10-17T18:54:17.763 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.763 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.764 DEBUG:boto:Params: {} 2023-10-17T18:54:17.764 DEBUG:boto:Token: None 2023-10-17T18:54:17.765 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo= 2023-10-17T18:54:17.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo=', 'Content-Length': '0'} 2023-10-17T18:54:17.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1167973b0a7f8b8-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.768 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.769 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.769 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.770 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.770 DEBUG:boto:Method: GET 2023-10-17T18:54:17.771 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.771 DEBUG:boto:Data: 2023-10-17T18:54:17.771 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.772 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.772 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.773 DEBUG:boto:Params: {} 2023-10-17T18:54:17.773 DEBUG:boto:Token: None 2023-10-17T18:54:17.773 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021?versionId=null 2023-10-17T18:54:17.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo= 2023-10-17T18:54:17.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKGgnOV7FLjfpPsPsqtx0HPMTqo=', 'Content-Length': '0'} 2023-10-17T18:54:17.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48ea5ea9026b1dc-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.818 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.819 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.819 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.820 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.820 DEBUG:boto:Method: GET 2023-10-17T18:54:17.821 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.821 DEBUG:boto:Data: 2023-10-17T18:54:17.822 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.822 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.823 DEBUG:boto:Params: {} 2023-10-17T18:54:17.823 DEBUG:boto:Token: None 2023-10-17T18:54:17.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000660f3e5f4d57aad-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.828 DEBUG:boto:b'hbmuog-58obj000000000000000000211000falseobj000000000000000000212023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.829 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.829 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.830 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.830 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.831 DEBUG:boto:Method: GET 2023-10-17T18:54:17.831 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.832 DEBUG:boto:Data: 2023-10-17T18:54:17.832 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.832 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.833 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.833 DEBUG:boto:Params: {} 2023-10-17T18:54:17.834 DEBUG:boto:Token: None 2023-10-17T18:54:17.834 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067248ce484ab5997-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.839 DEBUG:boto:b'hbmuog-58obj000000000000000000211000falseobj000000000000000000212023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.840 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.840 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.841 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.841 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.841 DEBUG:boto:Data: 2023-10-17T18:54:17.842 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.842 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.843 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.843 DEBUG:boto:Params: {} 2023-10-17T18:54:17.843 DEBUG:boto:Token: None 2023-10-17T18:54:17.844 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.844 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:39GIDV1CKcbvZSfxGKhr92ltR+I= 2023-10-17T18:54:17.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:39GIDV1CKcbvZSfxGKhr92ltR+I=', 'Content-Length': '0'} 2023-10-17T18:54:17.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b106137fc7bee5c5-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.847 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.848 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.848 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.848 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.849 DEBUG:boto:Data: 2023-10-17T18:54:17.849 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.850 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.851 DEBUG:boto:Params: {} 2023-10-17T18:54:17.851 DEBUG:boto:Token: None 2023-10-17T18:54:17.851 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:39GIDV1CKcbvZSfxGKhr92ltR+I= 2023-10-17T18:54:17.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:39GIDV1CKcbvZSfxGKhr92ltR+I=', 'Content-Length': '0'} 2023-10-17T18:54:17.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023aefe065c9327a1-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-17T18:54:17.855 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.856 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.856 DEBUG:boto:Method: GET 2023-10-17T18:54:17.857 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.857 DEBUG:boto:Data: 2023-10-17T18:54:17.857 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.858 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.859 DEBUG:boto:Params: {} 2023-10-17T18:54:17.859 DEBUG:boto:Token: None 2023-10-17T18:54:17.860 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ftHXxJfbW2cCCFI+c9g3clsNwWI= 2023-10-17T18:54:17.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ftHXxJfbW2cCCFI+c9g3clsNwWI=', 'Content-Length': '0'} 2023-10-17T18:54:17.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8baf9af13d48ada-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.906 DEBUG:boto:path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.907 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.907 DEBUG:boto:Method: GET 2023-10-17T18:54:17.908 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.908 DEBUG:boto:Data: 2023-10-17T18:54:17.909 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.909 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.909 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.910 DEBUG:boto:Params: {} 2023-10-17T18:54:17.910 DEBUG:boto:Token: None 2023-10-17T18:54:17.911 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000021 2023-10-17T18:54:17.911 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ftHXxJfbW2cCCFI+c9g3clsNwWI= 2023-10-17T18:54:17.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ftHXxJfbW2cCCFI+c9g3clsNwWI=', 'Content-Length': '0'} 2023-10-17T18:54:17.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000050849ec32396907-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.915 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.915 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.916 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.916 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.916 DEBUG:boto:Method: GET 2023-10-17T18:54:17.917 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.917 DEBUG:boto:Data: 2023-10-17T18:54:17.918 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.919 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.919 DEBUG:boto:Params: {} 2023-10-17T18:54:17.919 DEBUG:boto:Token: None 2023-10-17T18:54:17.920 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f440353d75d1a8f3-00652ed859-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.924 DEBUG:boto:b'hbmuog-58obj000000000000000000211000falseobj000000000000000000212023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.925 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:17.925 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:17.925 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.926 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.926 DEBUG:boto:Method: GET 2023-10-17T18:54:17.927 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000021 2023-10-17T18:54:17.927 DEBUG:boto:Data: 2023-10-17T18:54:17.928 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.928 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.928 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.929 DEBUG:boto:Params: {} 2023-10-17T18:54:17.929 DEBUG:boto:Token: None 2023-10-17T18:54:17.930 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/ 2023-10-17T18:54:17.930 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4= 2023-10-17T18:54:17.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i1AixI9auECq6lNb81CqsEP7KX4=', 'Content-Length': '0'} 2023-10-17T18:54:17.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6099981e8569d14-00652ed859-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.936 DEBUG:boto:b'hbmuog-58obj000000000000000000211000falseobj000000000000000000212023-10-17T18:54:07.331Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:17.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-17T18:54:17.937 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.937 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.937 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.938 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.938 DEBUG:boto:Method: GET 2023-10-17T18:54:17.939 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.939 DEBUG:boto:Data: 2023-10-17T18:54:17.940 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.940 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.941 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.941 DEBUG:boto:Params: {} 2023-10-17T18:54:17.941 DEBUG:boto:Token: None 2023-10-17T18:54:17.942 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KdIfb4RUxrenPXd89TrKRMiT47s= 2023-10-17T18:54:17.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KdIfb4RUxrenPXd89TrKRMiT47s=', 'Content-Length': '0'} 2023-10-17T18:54:17.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c390c2db1b91bd64-00652ed859-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.946 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.946 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.947 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.947 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.948 DEBUG:boto:Method: GET 2023-10-17T18:54:17.948 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.948 DEBUG:boto:Data: 2023-10-17T18:54:17.949 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.949 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:17.950 DEBUG:boto:Port: 8000 2023-10-17T18:54:17.950 DEBUG:boto:Params: {} 2023-10-17T18:54:17.951 DEBUG:boto:Token: None 2023-10-17T18:54:17.951 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.952 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KdIfb4RUxrenPXd89TrKRMiT47s= 2023-10-17T18:54:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KdIfb4RUxrenPXd89TrKRMiT47s=', 'Content-Length': '0'} 2023-10-17T18:54:17.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0d26044a419add-00652ed859-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:17.994 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.995 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:17.995 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.995 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.996 DEBUG:boto:Method: HEAD 2023-10-17T18:54:17.996 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:17.997 DEBUG:boto:Data: 2023-10-17T18:54:17.997 DEBUG:boto:Headers: {} 2023-10-17T18:54:17.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:17.998 DEBUG:boto:Port: 8001 2023-10-17T18:54:17.998 DEBUG:boto:Params: {} 2023-10-17T18:54:17.999 DEBUG:boto:Token: None 2023-10-17T18:54:17.999 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:17 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.000 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ezTDP+6VaVOsVJsfw3UZYrSYctQ= 2023-10-17T18:54:18.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ezTDP+6VaVOsVJsfw3UZYrSYctQ=', 'Content-Length': '0'} 2023-10-17T18:54:18.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c10925917e4115-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.003 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.004 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.004 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.004 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.005 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.005 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.005 DEBUG:boto:Data: 2023-10-17T18:54:18.006 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.006 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.007 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.007 DEBUG:boto:Params: {} 2023-10-17T18:54:18.007 DEBUG:boto:Token: None 2023-10-17T18:54:18.008 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.008 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2IQQHbEPE+xqn4YqlqFqIY1C+uA= 2023-10-17T18:54:18.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2IQQHbEPE+xqn4YqlqFqIY1C+uA=', 'Content-Length': '0'} 2023-10-17T18:54:18.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008067b82e313d168c-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-17T18:54:18.012 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.013 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.013 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.013 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.014 DEBUG:boto:Method: GET 2023-10-17T18:54:18.014 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.015 DEBUG:boto:Data: 2023-10-17T18:54:18.015 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.015 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.016 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.016 DEBUG:boto:Params: {} 2023-10-17T18:54:18.017 DEBUG:boto:Token: None 2023-10-17T18:54:18.017 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.018 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JLQOhdV+yG2u289+rqd8jZXp6+g= 2023-10-17T18:54:18.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JLQOhdV+yG2u289+rqd8jZXp6+g=', 'Content-Length': '0'} 2023-10-17T18:54:18.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001868034ec7c2bc85-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.062 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.063 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.063 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.064 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.064 DEBUG:boto:Method: GET 2023-10-17T18:54:18.064 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.065 DEBUG:boto:Data: 2023-10-17T18:54:18.065 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.066 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.066 DEBUG:boto:Params: {} 2023-10-17T18:54:18.067 DEBUG:boto:Token: None 2023-10-17T18:54:18.067 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022?versionId=null 2023-10-17T18:54:18.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JLQOhdV+yG2u289+rqd8jZXp6+g= 2023-10-17T18:54:18.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JLQOhdV+yG2u289+rqd8jZXp6+g=', 'Content-Length': '0'} 2023-10-17T18:54:18.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe2654d514ac2ea-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.071 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.072 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.072 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.073 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.073 DEBUG:boto:Method: GET 2023-10-17T18:54:18.073 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.074 DEBUG:boto:Data: 2023-10-17T18:54:18.074 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.075 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.075 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.076 DEBUG:boto:Params: {} 2023-10-17T18:54:18.076 DEBUG:boto:Token: None 2023-10-17T18:54:18.077 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.077 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724fb3b4d4014ed1-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.081 DEBUG:boto:b'hbmuog-58obj000000000000000000221000falseobj000000000000000000222023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.081 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.082 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.082 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.083 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.083 DEBUG:boto:Method: GET 2023-10-17T18:54:18.083 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.084 DEBUG:boto:Data: 2023-10-17T18:54:18.084 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.084 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.085 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.085 DEBUG:boto:Params: {} 2023-10-17T18:54:18.086 DEBUG:boto:Token: None 2023-10-17T18:54:18.086 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.087 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa17fa26eff2ffef-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.092 DEBUG:boto:b'hbmuog-58obj000000000000000000221000falseobj000000000000000000222023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.093 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.093 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.093 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.094 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.094 DEBUG:boto:Data: 2023-10-17T18:54:18.094 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.095 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.095 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.096 DEBUG:boto:Params: {} 2023-10-17T18:54:18.096 DEBUG:boto:Token: None 2023-10-17T18:54:18.096 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.097 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q6eDVHlMCHWVhRQCaGO7YtrkAsQ= 2023-10-17T18:54:18.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q6eDVHlMCHWVhRQCaGO7YtrkAsQ=', 'Content-Length': '0'} 2023-10-17T18:54:18.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d121f6c6d74b9b-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.100 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.100 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.100 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.101 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.101 DEBUG:boto:Data: 2023-10-17T18:54:18.101 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.102 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.103 DEBUG:boto:Params: {} 2023-10-17T18:54:18.103 DEBUG:boto:Token: None 2023-10-17T18:54:18.103 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q6eDVHlMCHWVhRQCaGO7YtrkAsQ= 2023-10-17T18:54:18.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q6eDVHlMCHWVhRQCaGO7YtrkAsQ=', 'Content-Length': '0'} 2023-10-17T18:54:18.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a55b040d610688-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-17T18:54:18.107 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.107 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.108 DEBUG:boto:Method: GET 2023-10-17T18:54:18.108 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.109 DEBUG:boto:Data: 2023-10-17T18:54:18.109 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.110 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.110 DEBUG:boto:Params: {} 2023-10-17T18:54:18.111 DEBUG:boto:Token: None 2023-10-17T18:54:18.111 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rfQDafqib6SeSWjiY+UuYXpqVig= 2023-10-17T18:54:18.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rfQDafqib6SeSWjiY+UuYXpqVig=', 'Content-Length': '0'} 2023-10-17T18:54:18.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d485580f2a9e8bd0-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.115 DEBUG:boto:path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.115 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.115 DEBUG:boto:Method: GET 2023-10-17T18:54:18.116 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.116 DEBUG:boto:Data: 2023-10-17T18:54:18.117 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.117 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.118 DEBUG:boto:Params: {} 2023-10-17T18:54:18.118 DEBUG:boto:Token: None 2023-10-17T18:54:18.119 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000022 2023-10-17T18:54:18.119 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rfQDafqib6SeSWjiY+UuYXpqVig= 2023-10-17T18:54:18.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rfQDafqib6SeSWjiY+UuYXpqVig=', 'Content-Length': '0'} 2023-10-17T18:54:18.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcdeefe5ed6b4a8-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.162 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.163 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.163 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.164 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.164 DEBUG:boto:Method: GET 2023-10-17T18:54:18.165 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.165 DEBUG:boto:Data: 2023-10-17T18:54:18.166 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.166 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.167 DEBUG:boto:Params: {} 2023-10-17T18:54:18.167 DEBUG:boto:Token: None 2023-10-17T18:54:18.168 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.168 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d00cd110b2bbbc8-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.173 DEBUG:boto:b'hbmuog-58obj000000000000000000221000falseobj000000000000000000222023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.174 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.175 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.175 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.176 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.176 DEBUG:boto:Method: GET 2023-10-17T18:54:18.176 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000022 2023-10-17T18:54:18.177 DEBUG:boto:Data: 2023-10-17T18:54:18.177 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.178 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.178 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.178 DEBUG:boto:Params: {} 2023-10-17T18:54:18.179 DEBUG:boto:Token: None 2023-10-17T18:54:18.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076790f6431e0a251-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.185 DEBUG:boto:b'hbmuog-58obj000000000000000000221000falseobj000000000000000000222023-10-17T18:54:06.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-17T18:54:18.186 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.186 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.187 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.187 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.187 DEBUG:boto:Method: GET 2023-10-17T18:54:18.188 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.188 DEBUG:boto:Data: 2023-10-17T18:54:18.189 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.189 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.189 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.190 DEBUG:boto:Params: {} 2023-10-17T18:54:18.190 DEBUG:boto:Token: None 2023-10-17T18:54:18.191 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.191 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs= 2023-10-17T18:54:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs=', 'Content-Length': '0'} 2023-10-17T18:54:18.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001833fdac99d6f3d6-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.234 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.235 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.235 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.236 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.236 DEBUG:boto:Method: GET 2023-10-17T18:54:18.236 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.237 DEBUG:boto:Data: 2023-10-17T18:54:18.237 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.238 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.238 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.238 DEBUG:boto:Params: {} 2023-10-17T18:54:18.239 DEBUG:boto:Token: None 2023-10-17T18:54:18.239 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.240 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs= 2023-10-17T18:54:18.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs=', 'Content-Length': '0'} 2023-10-17T18:54:18.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bdaa6a0707ab74d-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.243 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.243 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.244 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.244 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.244 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.245 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.245 DEBUG:boto:Data: 2023-10-17T18:54:18.246 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.246 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.247 DEBUG:boto:Params: {} 2023-10-17T18:54:18.247 DEBUG:boto:Token: None 2023-10-17T18:54:18.248 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cBjyz8ng8R4LqTjXEhLScw/O4gs= 2023-10-17T18:54:18.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBjyz8ng8R4LqTjXEhLScw/O4gs=', 'Content-Length': '0'} 2023-10-17T18:54:18.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a6e151298be5f8-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.251 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.251 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.252 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.252 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.252 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.253 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.253 DEBUG:boto:Data: 2023-10-17T18:54:18.253 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.254 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.254 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.255 DEBUG:boto:Params: {} 2023-10-17T18:54:18.255 DEBUG:boto:Token: None 2023-10-17T18:54:18.255 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.256 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cBjyz8ng8R4LqTjXEhLScw/O4gs= 2023-10-17T18:54:18.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBjyz8ng8R4LqTjXEhLScw/O4gs=', 'Content-Length': '0'} 2023-10-17T18:54:18.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800d66c65f79561f-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-17T18:54:18.259 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.259 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.260 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.260 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.261 DEBUG:boto:Method: GET 2023-10-17T18:54:18.261 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.261 DEBUG:boto:Data: 2023-10-17T18:54:18.262 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.262 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.263 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.263 DEBUG:boto:Params: {} 2023-10-17T18:54:18.263 DEBUG:boto:Token: None 2023-10-17T18:54:18.264 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.264 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs= 2023-10-17T18:54:18.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs=', 'Content-Length': '0'} 2023-10-17T18:54:18.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f5db4e924e4f3b-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.268 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.268 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.269 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.269 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.269 DEBUG:boto:Method: GET 2023-10-17T18:54:18.270 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.270 DEBUG:boto:Data: 2023-10-17T18:54:18.271 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.271 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.272 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.272 DEBUG:boto:Params: {} 2023-10-17T18:54:18.272 DEBUG:boto:Token: None 2023-10-17T18:54:18.273 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023?versionId=null 2023-10-17T18:54:18.273 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs= 2023-10-17T18:54:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1R/2aG2RyUJ+gtgZVF6H4Tnttvs=', 'Content-Length': '0'} 2023-10-17T18:54:18.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7776249bfdd704-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.276 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.277 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.277 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.278 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.278 DEBUG:boto:Method: GET 2023-10-17T18:54:18.279 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.279 DEBUG:boto:Data: 2023-10-17T18:54:18.280 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.280 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.280 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.281 DEBUG:boto:Params: {} 2023-10-17T18:54:18.281 DEBUG:boto:Token: None 2023-10-17T18:54:18.282 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.282 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e120f04aee9b001-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.326 DEBUG:boto:b'hbmuog-58obj000000000000000000231000falseobj000000000000000000232023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.327 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.328 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.328 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.328 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.329 DEBUG:boto:Method: GET 2023-10-17T18:54:18.329 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.330 DEBUG:boto:Data: 2023-10-17T18:54:18.330 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.330 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.331 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.331 DEBUG:boto:Params: {} 2023-10-17T18:54:18.332 DEBUG:boto:Token: None 2023-10-17T18:54:18.332 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.333 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4244a7a84b37d8-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.338 DEBUG:boto:b'hbmuog-58obj000000000000000000231000falseobj000000000000000000232023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.338 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.339 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.339 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.340 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.340 DEBUG:boto:Data: 2023-10-17T18:54:18.340 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.341 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.341 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.342 DEBUG:boto:Params: {} 2023-10-17T18:54:18.342 DEBUG:boto:Token: None 2023-10-17T18:54:18.343 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.343 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A/jfCDm+hhyqmGupv0+WPA0hSwA= 2023-10-17T18:54:18.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A/jfCDm+hhyqmGupv0+WPA0hSwA=', 'Content-Length': '0'} 2023-10-17T18:54:18.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ca5c0fc39d4340-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.346 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.347 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.347 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.347 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.348 DEBUG:boto:Data: 2023-10-17T18:54:18.348 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.349 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.349 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.349 DEBUG:boto:Params: {} 2023-10-17T18:54:18.350 DEBUG:boto:Token: None 2023-10-17T18:54:18.350 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.351 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A/jfCDm+hhyqmGupv0+WPA0hSwA= 2023-10-17T18:54:18.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A/jfCDm+hhyqmGupv0+WPA0hSwA=', 'Content-Length': '0'} 2023-10-17T18:54:18.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108ea9ba4a17d20b-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-17T18:54:18.354 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.355 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.355 DEBUG:boto:Method: GET 2023-10-17T18:54:18.355 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.356 DEBUG:boto:Data: 2023-10-17T18:54:18.356 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.357 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.357 DEBUG:boto:Params: {} 2023-10-17T18:54:18.358 DEBUG:boto:Token: None 2023-10-17T18:54:18.359 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UGI72ClVcuwyDtwAjBjOXachRWk= 2023-10-17T18:54:18.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UGI72ClVcuwyDtwAjBjOXachRWk=', 'Content-Length': '0'} 2023-10-17T18:54:18.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdff06f2ec492b51-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.362 DEBUG:boto:path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.362 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.363 DEBUG:boto:Method: GET 2023-10-17T18:54:18.363 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.364 DEBUG:boto:Data: 2023-10-17T18:54:18.364 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.365 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.365 DEBUG:boto:Params: {} 2023-10-17T18:54:18.366 DEBUG:boto:Token: None 2023-10-17T18:54:18.366 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000023 2023-10-17T18:54:18.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UGI72ClVcuwyDtwAjBjOXachRWk= 2023-10-17T18:54:18.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UGI72ClVcuwyDtwAjBjOXachRWk=', 'Content-Length': '0'} 2023-10-17T18:54:18.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001840afee1cb9ff52-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.410 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.411 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.411 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.411 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.412 DEBUG:boto:Method: GET 2023-10-17T18:54:18.412 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.413 DEBUG:boto:Data: 2023-10-17T18:54:18.413 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.414 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.414 DEBUG:boto:Params: {} 2023-10-17T18:54:18.415 DEBUG:boto:Token: None 2023-10-17T18:54:18.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.415 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9ffecafba33e51-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.419 DEBUG:boto:b'hbmuog-58obj000000000000000000231000falseobj000000000000000000232023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.419 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.420 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.420 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.421 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.421 DEBUG:boto:Method: GET 2023-10-17T18:54:18.421 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000023 2023-10-17T18:54:18.422 DEBUG:boto:Data: 2023-10-17T18:54:18.422 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.423 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.423 DEBUG:boto:Params: {} 2023-10-17T18:54:18.424 DEBUG:boto:Token: None 2023-10-17T18:54:18.424 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a74afe9fbc14d14-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.430 DEBUG:boto:b'hbmuog-58obj000000000000000000231000falseobj000000000000000000232023-10-17T18:54:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-17T18:54:18.431 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.431 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.432 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.432 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.432 DEBUG:boto:Method: GET 2023-10-17T18:54:18.433 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.433 DEBUG:boto:Data: 2023-10-17T18:54:18.433 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.434 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.435 DEBUG:boto:Params: {} 2023-10-17T18:54:18.435 DEBUG:boto:Token: None 2023-10-17T18:54:18.435 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.436 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE= 2023-10-17T18:54:18.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE=', 'Content-Length': '0'} 2023-10-17T18:54:18.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ddce409eaaf2f2-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.439 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.439 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.440 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.440 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.441 DEBUG:boto:Method: GET 2023-10-17T18:54:18.441 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.441 DEBUG:boto:Data: 2023-10-17T18:54:18.442 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.442 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.442 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.443 DEBUG:boto:Params: {} 2023-10-17T18:54:18.443 DEBUG:boto:Token: None 2023-10-17T18:54:18.444 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.444 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE= 2023-10-17T18:54:18.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE=', 'Content-Length': '0'} 2023-10-17T18:54:18.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078116fef2153ad22-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.447 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.448 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.448 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.449 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.449 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.449 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.450 DEBUG:boto:Data: 2023-10-17T18:54:18.450 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.450 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.451 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.451 DEBUG:boto:Params: {} 2023-10-17T18:54:18.452 DEBUG:boto:Token: None 2023-10-17T18:54:18.452 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.452 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tbs05Eb+ML8WhtoH20x3Vm6mOs8= 2023-10-17T18:54:18.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tbs05Eb+ML8WhtoH20x3Vm6mOs8=', 'Content-Length': '0'} 2023-10-17T18:54:18.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007195c2a277e52739-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.455 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.455 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.456 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.456 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.457 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.457 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.457 DEBUG:boto:Data: 2023-10-17T18:54:18.458 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.458 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.459 DEBUG:boto:Params: {} 2023-10-17T18:54:18.459 DEBUG:boto:Token: None 2023-10-17T18:54:18.460 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tbs05Eb+ML8WhtoH20x3Vm6mOs8= 2023-10-17T18:54:18.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tbs05Eb+ML8WhtoH20x3Vm6mOs8=', 'Content-Length': '0'} 2023-10-17T18:54:18.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c0fb205e4e0929-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-17T18:54:18.463 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.464 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.464 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.464 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.465 DEBUG:boto:Method: GET 2023-10-17T18:54:18.465 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.465 DEBUG:boto:Data: 2023-10-17T18:54:18.466 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.467 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.467 DEBUG:boto:Params: {} 2023-10-17T18:54:18.467 DEBUG:boto:Token: None 2023-10-17T18:54:18.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE= 2023-10-17T18:54:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE=', 'Content-Length': '0'} 2023-10-17T18:54:18.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808363e06218ce1c-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.514 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.515 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.515 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.516 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.516 DEBUG:boto:Method: GET 2023-10-17T18:54:18.516 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.517 DEBUG:boto:Data: 2023-10-17T18:54:18.517 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.518 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.518 DEBUG:boto:Params: {} 2023-10-17T18:54:18.519 DEBUG:boto:Token: None 2023-10-17T18:54:18.519 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024?versionId=null 2023-10-17T18:54:18.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE= 2023-10-17T18:54:18.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2wPLSPh7k/vRoV/wuZOzJldjpE=', 'Content-Length': '0'} 2023-10-17T18:54:18.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0fcf71f3bc1b0b-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.523 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.524 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.524 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.525 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.525 DEBUG:boto:Method: GET 2023-10-17T18:54:18.525 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.526 DEBUG:boto:Data: 2023-10-17T18:54:18.526 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.527 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.527 DEBUG:boto:Params: {} 2023-10-17T18:54:18.528 DEBUG:boto:Token: None 2023-10-17T18:54:18.528 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a09a4e07664e87-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.532 DEBUG:boto:b'hbmuog-58obj000000000000000000241000falseobj000000000000000000242023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.533 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.533 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.534 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.534 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.535 DEBUG:boto:Method: GET 2023-10-17T18:54:18.535 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.535 DEBUG:boto:Data: 2023-10-17T18:54:18.536 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.536 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.536 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.537 DEBUG:boto:Params: {} 2023-10-17T18:54:18.537 DEBUG:boto:Token: None 2023-10-17T18:54:18.538 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.538 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d0973d93b052f66-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.543 DEBUG:boto:b'hbmuog-58obj000000000000000000241000falseobj000000000000000000242023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.544 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.544 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.545 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.545 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.545 DEBUG:boto:Data: 2023-10-17T18:54:18.546 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.546 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.547 DEBUG:boto:Params: {} 2023-10-17T18:54:18.547 DEBUG:boto:Token: None 2023-10-17T18:54:18.548 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5IGTEoR1RZkcHanFDWPLb+t7Ke8= 2023-10-17T18:54:18.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5IGTEoR1RZkcHanFDWPLb+t7Ke8=', 'Content-Length': '0'} 2023-10-17T18:54:18.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32c41a1f063a3cd-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.551 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.551 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.552 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.552 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.552 DEBUG:boto:Data: 2023-10-17T18:54:18.553 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.554 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.554 DEBUG:boto:Params: {} 2023-10-17T18:54:18.554 DEBUG:boto:Token: None 2023-10-17T18:54:18.555 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5IGTEoR1RZkcHanFDWPLb+t7Ke8= 2023-10-17T18:54:18.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5IGTEoR1RZkcHanFDWPLb+t7Ke8=', 'Content-Length': '0'} 2023-10-17T18:54:18.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000311a6286b20c45b-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-17T18:54:18.559 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.559 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.559 DEBUG:boto:Method: GET 2023-10-17T18:54:18.560 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.560 DEBUG:boto:Data: 2023-10-17T18:54:18.561 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.561 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.562 DEBUG:boto:Params: {} 2023-10-17T18:54:18.562 DEBUG:boto:Token: None 2023-10-17T18:54:18.562 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HkNWfud1fbVmD3Gw7Fl/GftGP7s= 2023-10-17T18:54:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HkNWfud1fbVmD3Gw7Fl/GftGP7s=', 'Content-Length': '0'} 2023-10-17T18:54:18.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb3157a6fc34a97-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.606 DEBUG:boto:path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.607 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.607 DEBUG:boto:Method: GET 2023-10-17T18:54:18.607 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.608 DEBUG:boto:Data: 2023-10-17T18:54:18.608 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.609 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.609 DEBUG:boto:Params: {} 2023-10-17T18:54:18.610 DEBUG:boto:Token: None 2023-10-17T18:54:18.610 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000024 2023-10-17T18:54:18.611 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HkNWfud1fbVmD3Gw7Fl/GftGP7s= 2023-10-17T18:54:18.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HkNWfud1fbVmD3Gw7Fl/GftGP7s=', 'Content-Length': '0'} 2023-10-17T18:54:18.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a730aefa3eeea7-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.614 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.614 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.615 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.615 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.616 DEBUG:boto:Method: GET 2023-10-17T18:54:18.616 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.616 DEBUG:boto:Data: 2023-10-17T18:54:18.617 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.617 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.618 DEBUG:boto:Params: {} 2023-10-17T18:54:18.618 DEBUG:boto:Token: None 2023-10-17T18:54:18.619 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.619 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d473411863a3c4fd-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.623 DEBUG:boto:b'hbmuog-58obj000000000000000000241000falseobj000000000000000000242023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.623 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.624 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.624 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.624 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.625 DEBUG:boto:Method: GET 2023-10-17T18:54:18.625 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000024 2023-10-17T18:54:18.625 DEBUG:boto:Data: 2023-10-17T18:54:18.626 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.627 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.627 DEBUG:boto:Params: {} 2023-10-17T18:54:18.627 DEBUG:boto:Token: None 2023-10-17T18:54:18.628 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000744a804dbd32c013-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.633 DEBUG:boto:b'hbmuog-58obj000000000000000000241000falseobj000000000000000000242023-10-17T18:54:07.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-17T18:54:18.634 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.634 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.635 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.635 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.636 DEBUG:boto:Method: GET 2023-10-17T18:54:18.636 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.636 DEBUG:boto:Data: 2023-10-17T18:54:18.637 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.637 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.638 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.638 DEBUG:boto:Params: {} 2023-10-17T18:54:18.638 DEBUG:boto:Token: None 2023-10-17T18:54:18.639 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo= 2023-10-17T18:54:18.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo=', 'Content-Length': '0'} 2023-10-17T18:54:18.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590094bf064a1e19-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.642 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.643 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.643 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.644 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.644 DEBUG:boto:Method: GET 2023-10-17T18:54:18.644 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.645 DEBUG:boto:Data: 2023-10-17T18:54:18.645 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.646 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.646 DEBUG:boto:Params: {} 2023-10-17T18:54:18.647 DEBUG:boto:Token: None 2023-10-17T18:54:18.647 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo= 2023-10-17T18:54:18.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo=', 'Content-Length': '0'} 2023-10-17T18:54:18.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c52b3f5c21bdd9-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.690 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.691 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.691 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.692 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.692 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.692 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.693 DEBUG:boto:Data: 2023-10-17T18:54:18.693 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.694 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.694 DEBUG:boto:Params: {} 2023-10-17T18:54:18.695 DEBUG:boto:Token: None 2023-10-17T18:54:18.695 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6yo6kMj393qy+AN1uYoAEol476U= 2023-10-17T18:54:18.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6yo6kMj393qy+AN1uYoAEol476U=', 'Content-Length': '0'} 2023-10-17T18:54:18.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dded9658baa074db-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.699 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.699 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.700 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.700 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.701 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.701 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.701 DEBUG:boto:Data: 2023-10-17T18:54:18.702 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.703 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.703 DEBUG:boto:Params: {} 2023-10-17T18:54:18.703 DEBUG:boto:Token: None 2023-10-17T18:54:18.704 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6yo6kMj393qy+AN1uYoAEol476U= 2023-10-17T18:54:18.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6yo6kMj393qy+AN1uYoAEol476U=', 'Content-Length': '0'} 2023-10-17T18:54:18.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b5b2aae2f37b45-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-17T18:54:18.708 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.708 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.709 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.709 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.709 DEBUG:boto:Method: GET 2023-10-17T18:54:18.710 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.710 DEBUG:boto:Data: 2023-10-17T18:54:18.711 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.711 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.712 DEBUG:boto:Params: {} 2023-10-17T18:54:18.712 DEBUG:boto:Token: None 2023-10-17T18:54:18.713 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo= 2023-10-17T18:54:18.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo=', 'Content-Length': '0'} 2023-10-17T18:54:18.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2b8225b42d1b6b-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.762 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.763 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.763 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.764 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.764 DEBUG:boto:Method: GET 2023-10-17T18:54:18.764 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.765 DEBUG:boto:Data: 2023-10-17T18:54:18.765 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.766 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.766 DEBUG:boto:Params: {} 2023-10-17T18:54:18.767 DEBUG:boto:Token: None 2023-10-17T18:54:18.767 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025?versionId=null 2023-10-17T18:54:18.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo= 2023-10-17T18:54:18.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KKTb2lPms3Tf5H9yFBbZDuOhlBo=', 'Content-Length': '0'} 2023-10-17T18:54:18.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f65299d778b5a88-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.771 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.771 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.771 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.772 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.772 DEBUG:boto:Method: GET 2023-10-17T18:54:18.773 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.773 DEBUG:boto:Data: 2023-10-17T18:54:18.773 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.774 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.775 DEBUG:boto:Params: {} 2023-10-17T18:54:18.775 DEBUG:boto:Token: None 2023-10-17T18:54:18.776 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089765cd14db81ff4-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.780 DEBUG:boto:b'hbmuog-58obj000000000000000000251000falseobj000000000000000000252023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.781 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.781 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.782 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.782 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.782 DEBUG:boto:Method: GET 2023-10-17T18:54:18.783 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.783 DEBUG:boto:Data: 2023-10-17T18:54:18.783 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.784 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.785 DEBUG:boto:Params: {} 2023-10-17T18:54:18.785 DEBUG:boto:Token: None 2023-10-17T18:54:18.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.786 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b62e2d4e4a9e47e-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.791 DEBUG:boto:b'hbmuog-58obj000000000000000000251000falseobj000000000000000000252023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.792 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.792 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.793 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.793 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.793 DEBUG:boto:Data: 2023-10-17T18:54:18.794 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.794 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.795 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.795 DEBUG:boto:Params: {} 2023-10-17T18:54:18.795 DEBUG:boto:Token: None 2023-10-17T18:54:18.796 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lFOvXCWlGYIgH9rVR7l9WwjEHMo= 2023-10-17T18:54:18.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lFOvXCWlGYIgH9rVR7l9WwjEHMo=', 'Content-Length': '0'} 2023-10-17T18:54:18.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288dad4a38a1a248-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.800 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.800 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.800 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.801 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.801 DEBUG:boto:Data: 2023-10-17T18:54:18.801 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.802 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.803 DEBUG:boto:Params: {} 2023-10-17T18:54:18.803 DEBUG:boto:Token: None 2023-10-17T18:54:18.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lFOvXCWlGYIgH9rVR7l9WwjEHMo= 2023-10-17T18:54:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lFOvXCWlGYIgH9rVR7l9WwjEHMo=', 'Content-Length': '0'} 2023-10-17T18:54:18.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abfef513af1e8fbf-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-17T18:54:18.808 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.808 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.809 DEBUG:boto:Method: GET 2023-10-17T18:54:18.809 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.810 DEBUG:boto:Data: 2023-10-17T18:54:18.810 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.811 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.811 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.811 DEBUG:boto:Params: {} 2023-10-17T18:54:18.812 DEBUG:boto:Token: None 2023-10-17T18:54:18.812 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.813 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RNDuIaPqyuzSLsG1xNFuE5Ytny0= 2023-10-17T18:54:18.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RNDuIaPqyuzSLsG1xNFuE5Ytny0=', 'Content-Length': '0'} 2023-10-17T18:54:18.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebc8e3cbf4a63c0-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.859 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.860 DEBUG:boto:Method: GET 2023-10-17T18:54:18.861 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.861 DEBUG:boto:Data: 2023-10-17T18:54:18.862 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.863 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.863 DEBUG:boto:Params: {} 2023-10-17T18:54:18.864 DEBUG:boto:Token: None 2023-10-17T18:54:18.865 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000025 2023-10-17T18:54:18.866 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RNDuIaPqyuzSLsG1xNFuE5Ytny0= 2023-10-17T18:54:18.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RNDuIaPqyuzSLsG1xNFuE5Ytny0=', 'Content-Length': '0'} 2023-10-17T18:54:18.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1560a458a59050-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.870 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.870 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.871 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.871 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.872 DEBUG:boto:Method: GET 2023-10-17T18:54:18.872 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.873 DEBUG:boto:Data: 2023-10-17T18:54:18.873 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.874 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.874 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.875 DEBUG:boto:Params: {} 2023-10-17T18:54:18.875 DEBUG:boto:Token: None 2023-10-17T18:54:18.876 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.876 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835406ca75c5e849-00652ed85a-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.880 DEBUG:boto:b'hbmuog-58obj000000000000000000251000falseobj000000000000000000252023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.881 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:18.882 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:18.882 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.883 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.883 DEBUG:boto:Method: GET 2023-10-17T18:54:18.884 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000025 2023-10-17T18:54:18.884 DEBUG:boto:Data: 2023-10-17T18:54:18.885 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.886 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.887 DEBUG:boto:Params: {} 2023-10-17T18:54:18.887 DEBUG:boto:Token: None 2023-10-17T18:54:18.888 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/ 2023-10-17T18:54:18.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y= 2023-10-17T18:54:18.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p/qiVqFMjbzhh+tl+zNYBSIfL8Y=', 'Content-Length': '0'} 2023-10-17T18:54:18.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f5f1492990d3b8-00652ed85a-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.894 DEBUG:boto:b'hbmuog-58obj000000000000000000251000falseobj000000000000000000252023-10-17T18:54:04.715Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:18.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-17T18:54:18.895 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.896 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.896 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.897 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.897 DEBUG:boto:Method: GET 2023-10-17T18:54:18.898 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.898 DEBUG:boto:Data: 2023-10-17T18:54:18.899 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.900 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.900 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.901 DEBUG:boto:Params: {} 2023-10-17T18:54:18.901 DEBUG:boto:Token: None 2023-10-17T18:54:18.902 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.902 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s= 2023-10-17T18:54:18.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s=', 'Content-Length': '0'} 2023-10-17T18:54:18.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e0e3556958875b-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.906 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.907 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.907 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.907 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.908 DEBUG:boto:Method: GET 2023-10-17T18:54:18.909 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.909 DEBUG:boto:Data: 2023-10-17T18:54:18.910 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.911 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.911 DEBUG:boto:Params: {} 2023-10-17T18:54:18.912 DEBUG:boto:Token: None 2023-10-17T18:54:18.912 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.913 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s= 2023-10-17T18:54:18.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s=', 'Content-Length': '0'} 2023-10-17T18:54:18.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9a34740b6c2fd6-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.958 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.959 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.959 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.960 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.960 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.961 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.961 DEBUG:boto:Data: 2023-10-17T18:54:18.961 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.962 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.963 DEBUG:boto:Params: {} 2023-10-17T18:54:18.963 DEBUG:boto:Token: None 2023-10-17T18:54:18.964 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PE87D8sdsUwrYy1rfC6fWzN5FS4= 2023-10-17T18:54:18.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE87D8sdsUwrYy1rfC6fWzN5FS4=', 'Content-Length': '0'} 2023-10-17T18:54:18.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981ee0dcc6007e76-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.967 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.968 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.968 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.969 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.969 DEBUG:boto:Method: HEAD 2023-10-17T18:54:18.970 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.970 DEBUG:boto:Data: 2023-10-17T18:54:18.970 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:18.971 DEBUG:boto:Port: 8000 2023-10-17T18:54:18.972 DEBUG:boto:Params: {} 2023-10-17T18:54:18.972 DEBUG:boto:Token: None 2023-10-17T18:54:18.973 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PE87D8sdsUwrYy1rfC6fWzN5FS4= 2023-10-17T18:54:18.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE87D8sdsUwrYy1rfC6fWzN5FS4=', 'Content-Length': '0'} 2023-10-17T18:54:18.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e01929a3a7986bc3-00652ed85a-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:18.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-17T18:54:18.977 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.977 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:18.978 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.978 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.979 DEBUG:boto:Method: GET 2023-10-17T18:54:18.979 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.980 DEBUG:boto:Data: 2023-10-17T18:54:18.980 DEBUG:boto:Headers: {} 2023-10-17T18:54:18.980 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:18.981 DEBUG:boto:Port: 8001 2023-10-17T18:54:18.981 DEBUG:boto:Params: {} 2023-10-17T18:54:18.982 DEBUG:boto:Token: None 2023-10-17T18:54:18.982 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:18 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:18.983 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s= 2023-10-17T18:54:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3zcACXQk2V8duuITX2wceWjpU+s=', 'Content-Length': '0'} 2023-10-17T18:54:18.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604dd2cb295e7ab4-00652ed85a-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.034 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.035 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.035 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:19.036 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:19.036 DEBUG:boto:Method: GET 2023-10-17T18:54:19.037 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:19.037 DEBUG:boto:Data: 2023-10-17T18:54:19.038 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.038 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.039 DEBUG:boto:Params: {} 2023-10-17T18:54:19.039 DEBUG:boto:Token: None 2023-10-17T18:54:19.040 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000026?versionId=null 2023-10-17T18:54:19.040 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lqZoVR1gDMFWXceLogzSuCtCHIQ= 2023-10-17T18:54:19.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lqZoVR1gDMFWXceLogzSuCtCHIQ=', 'Content-Length': '0'} 2023-10-17T18:54:19.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3f006efd6f3a45-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.044 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.044 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.045 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.045 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.045 DEBUG:boto:Method: GET 2023-10-17T18:54:19.046 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.046 DEBUG:boto:Data: 2023-10-17T18:54:19.047 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.048 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.048 DEBUG:boto:Params: {} 2023-10-17T18:54:19.049 DEBUG:boto:Token: None 2023-10-17T18:54:19.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10fdaf5eba441fb-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.053 DEBUG:boto:b'hbmuog-58obj000000000000000000261000falseobj000000000000000000262023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.054 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.054 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.055 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.055 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.056 DEBUG:boto:Method: GET 2023-10-17T18:54:19.056 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.056 DEBUG:boto:Data: 2023-10-17T18:54:19.057 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.057 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.058 DEBUG:boto:Params: {} 2023-10-17T18:54:19.058 DEBUG:boto:Token: None 2023-10-17T18:54:19.059 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c9f3721bf76eb0-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.065 DEBUG:boto:b'hbmuog-58obj000000000000000000261000falseobj000000000000000000262023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.065 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.066 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.066 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.067 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.067 DEBUG:boto:Data: 2023-10-17T18:54:19.068 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.068 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.068 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.069 DEBUG:boto:Params: {} 2023-10-17T18:54:19.069 DEBUG:boto:Token: None 2023-10-17T18:54:19.070 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.070 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g6rBcoCwAJBRY2gLbRHKQoD+zX8= 2023-10-17T18:54:19.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g6rBcoCwAJBRY2gLbRHKQoD+zX8=', 'Content-Length': '0'} 2023-10-17T18:54:19.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d82471a120f456-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.073 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.074 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.074 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.075 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.075 DEBUG:boto:Data: 2023-10-17T18:54:19.075 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.076 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.077 DEBUG:boto:Params: {} 2023-10-17T18:54:19.077 DEBUG:boto:Token: None 2023-10-17T18:54:19.077 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g6rBcoCwAJBRY2gLbRHKQoD+zX8= 2023-10-17T18:54:19.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g6rBcoCwAJBRY2gLbRHKQoD+zX8=', 'Content-Length': '0'} 2023-10-17T18:54:19.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003181dfda45183abd-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-17T18:54:19.081 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.082 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.082 DEBUG:boto:Method: GET 2023-10-17T18:54:19.083 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.083 DEBUG:boto:Data: 2023-10-17T18:54:19.084 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.084 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.084 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.085 DEBUG:boto:Params: {} 2023-10-17T18:54:19.085 DEBUG:boto:Token: None 2023-10-17T18:54:19.086 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.086 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pshH7YKwA00QsEj2nrifXItmVQ8= 2023-10-17T18:54:19.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pshH7YKwA00QsEj2nrifXItmVQ8=', 'Content-Length': '0'} 2023-10-17T18:54:19.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0f454e00e36037-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.089 DEBUG:boto:path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.090 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.090 DEBUG:boto:Method: GET 2023-10-17T18:54:19.091 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.091 DEBUG:boto:Data: 2023-10-17T18:54:19.092 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.092 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.092 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.093 DEBUG:boto:Params: {} 2023-10-17T18:54:19.093 DEBUG:boto:Token: None 2023-10-17T18:54:19.094 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000026 2023-10-17T18:54:19.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pshH7YKwA00QsEj2nrifXItmVQ8= 2023-10-17T18:54:19.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pshH7YKwA00QsEj2nrifXItmVQ8=', 'Content-Length': '0'} 2023-10-17T18:54:19.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b54adf3db7fdade-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.138 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.139 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.140 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.140 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.140 DEBUG:boto:Method: GET 2023-10-17T18:54:19.141 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.141 DEBUG:boto:Data: 2023-10-17T18:54:19.142 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.142 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.143 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.143 DEBUG:boto:Params: {} 2023-10-17T18:54:19.144 DEBUG:boto:Token: None 2023-10-17T18:54:19.144 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202d27fb6883a3da-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.148 DEBUG:boto:b'hbmuog-58obj000000000000000000261000falseobj000000000000000000262023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.149 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.150 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.150 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.150 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.151 DEBUG:boto:Method: GET 2023-10-17T18:54:19.151 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000026 2023-10-17T18:54:19.152 DEBUG:boto:Data: 2023-10-17T18:54:19.152 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.153 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.153 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.154 DEBUG:boto:Params: {} 2023-10-17T18:54:19.154 DEBUG:boto:Token: None 2023-10-17T18:54:19.155 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.155 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b7916f5c7c19779-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.160 DEBUG:boto:b'hbmuog-58obj000000000000000000261000falseobj000000000000000000262023-10-17T18:54:07.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-17T18:54:19.161 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.162 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.162 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.163 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.163 DEBUG:boto:Method: GET 2023-10-17T18:54:19.164 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.164 DEBUG:boto:Data: 2023-10-17T18:54:19.165 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.165 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.165 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.166 DEBUG:boto:Params: {} 2023-10-17T18:54:19.166 DEBUG:boto:Token: None 2023-10-17T18:54:19.167 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.167 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU= 2023-10-17T18:54:19.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU=', 'Content-Length': '0'} 2023-10-17T18:54:19.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdf8d4b1ded8e20-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.171 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.171 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.172 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.172 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.173 DEBUG:boto:Method: GET 2023-10-17T18:54:19.173 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.174 DEBUG:boto:Data: 2023-10-17T18:54:19.174 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.175 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.175 DEBUG:boto:Params: {} 2023-10-17T18:54:19.176 DEBUG:boto:Token: None 2023-10-17T18:54:19.176 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.177 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU= 2023-10-17T18:54:19.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU=', 'Content-Length': '0'} 2023-10-17T18:54:19.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658739210b5416ec-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.180 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.181 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.181 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.181 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.182 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.182 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.183 DEBUG:boto:Data: 2023-10-17T18:54:19.183 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.184 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.184 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.184 DEBUG:boto:Params: {} 2023-10-17T18:54:19.185 DEBUG:boto:Token: None 2023-10-17T18:54:19.185 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.186 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qj/1wSbC45sEj9Gcb1a92/VcWfw= 2023-10-17T18:54:19.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qj/1wSbC45sEj9Gcb1a92/VcWfw=', 'Content-Length': '0'} 2023-10-17T18:54:19.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793b85668079c7b9-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.189 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.189 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.190 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.190 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.191 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.191 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.191 DEBUG:boto:Data: 2023-10-17T18:54:19.192 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.192 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.193 DEBUG:boto:Params: {} 2023-10-17T18:54:19.193 DEBUG:boto:Token: None 2023-10-17T18:54:19.194 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qj/1wSbC45sEj9Gcb1a92/VcWfw= 2023-10-17T18:54:19.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qj/1wSbC45sEj9Gcb1a92/VcWfw=', 'Content-Length': '0'} 2023-10-17T18:54:19.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49d22ee9e4eeae4-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-17T18:54:19.197 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.198 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.198 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.199 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.199 DEBUG:boto:Method: GET 2023-10-17T18:54:19.200 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.200 DEBUG:boto:Data: 2023-10-17T18:54:19.201 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.201 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.201 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.202 DEBUG:boto:Params: {} 2023-10-17T18:54:19.202 DEBUG:boto:Token: None 2023-10-17T18:54:19.203 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.203 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU= 2023-10-17T18:54:19.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU=', 'Content-Length': '0'} 2023-10-17T18:54:19.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a4f51fe11e9213-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.246 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.247 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.247 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.248 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.248 DEBUG:boto:Method: GET 2023-10-17T18:54:19.249 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.249 DEBUG:boto:Data: 2023-10-17T18:54:19.249 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.250 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.250 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.251 DEBUG:boto:Params: {} 2023-10-17T18:54:19.251 DEBUG:boto:Token: None 2023-10-17T18:54:19.252 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027?versionId=null 2023-10-17T18:54:19.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU= 2023-10-17T18:54:19.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VPWPtvaQwBetvFXjCr3I8vbm/IU=', 'Content-Length': '0'} 2023-10-17T18:54:19.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8327d2774a7243a-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.256 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.256 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.257 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.257 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.257 DEBUG:boto:Method: GET 2023-10-17T18:54:19.258 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.258 DEBUG:boto:Data: 2023-10-17T18:54:19.259 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.259 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.260 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.260 DEBUG:boto:Params: {} 2023-10-17T18:54:19.261 DEBUG:boto:Token: None 2023-10-17T18:54:19.261 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d70bfd294c079c8-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.266 DEBUG:boto:b'hbmuog-58obj000000000000000000271000falseobj000000000000000000272023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.267 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.267 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.268 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.268 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.269 DEBUG:boto:Method: GET 2023-10-17T18:54:19.269 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.269 DEBUG:boto:Data: 2023-10-17T18:54:19.270 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.271 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.271 DEBUG:boto:Params: {} 2023-10-17T18:54:19.272 DEBUG:boto:Token: None 2023-10-17T18:54:19.272 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.272 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042bd6467b5214778-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.278 DEBUG:boto:b'hbmuog-58obj000000000000000000271000falseobj000000000000000000272023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.278 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.279 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.279 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.280 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.280 DEBUG:boto:Data: 2023-10-17T18:54:19.280 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.281 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.282 DEBUG:boto:Params: {} 2023-10-17T18:54:19.282 DEBUG:boto:Token: None 2023-10-17T18:54:19.282 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.283 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WZzik64+Wj6+QuFz/B+ZK6hcObA= 2023-10-17T18:54:19.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WZzik64+Wj6+QuFz/B+ZK6hcObA=', 'Content-Length': '0'} 2023-10-17T18:54:19.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e60a86f3e5f27d-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.286 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.287 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.287 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.287 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.288 DEBUG:boto:Data: 2023-10-17T18:54:19.288 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.289 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.290 DEBUG:boto:Params: {} 2023-10-17T18:54:19.290 DEBUG:boto:Token: None 2023-10-17T18:54:19.290 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.291 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WZzik64+Wj6+QuFz/B+ZK6hcObA= 2023-10-17T18:54:19.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WZzik64+Wj6+QuFz/B+ZK6hcObA=', 'Content-Length': '0'} 2023-10-17T18:54:19.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326c1a29e4a3c528-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-17T18:54:19.295 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.295 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.295 DEBUG:boto:Method: GET 2023-10-17T18:54:19.296 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.296 DEBUG:boto:Data: 2023-10-17T18:54:19.297 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.297 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.298 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.298 DEBUG:boto:Params: {} 2023-10-17T18:54:19.298 DEBUG:boto:Token: None 2023-10-17T18:54:19.299 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.299 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bgnDutP7hCGQCQFhdV06DomgwuM= 2023-10-17T18:54:19.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bgnDutP7hCGQCQFhdV06DomgwuM=', 'Content-Length': '0'} 2023-10-17T18:54:19.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009736cb22bfcc07c4-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.302 DEBUG:boto:path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.303 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.303 DEBUG:boto:Method: GET 2023-10-17T18:54:19.304 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.304 DEBUG:boto:Data: 2023-10-17T18:54:19.305 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.305 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.305 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.306 DEBUG:boto:Params: {} 2023-10-17T18:54:19.306 DEBUG:boto:Token: None 2023-10-17T18:54:19.307 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000027 2023-10-17T18:54:19.307 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bgnDutP7hCGQCQFhdV06DomgwuM= 2023-10-17T18:54:19.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bgnDutP7hCGQCQFhdV06DomgwuM=', 'Content-Length': '0'} 2023-10-17T18:54:19.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235c6e84054625b9-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.311 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.311 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.312 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.312 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.312 DEBUG:boto:Method: GET 2023-10-17T18:54:19.313 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.313 DEBUG:boto:Data: 2023-10-17T18:54:19.314 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.314 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.315 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.315 DEBUG:boto:Params: {} 2023-10-17T18:54:19.315 DEBUG:boto:Token: None 2023-10-17T18:54:19.316 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b08e555bdee5ec6-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.320 DEBUG:boto:b'hbmuog-58obj000000000000000000271000falseobj000000000000000000272023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.320 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.321 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.321 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.322 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.322 DEBUG:boto:Method: GET 2023-10-17T18:54:19.323 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000027 2023-10-17T18:54:19.323 DEBUG:boto:Data: 2023-10-17T18:54:19.323 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.324 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.324 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.325 DEBUG:boto:Params: {} 2023-10-17T18:54:19.325 DEBUG:boto:Token: None 2023-10-17T18:54:19.325 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.326 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096f708990cd12d9e-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.331 DEBUG:boto:b'hbmuog-58obj000000000000000000271000falseobj000000000000000000272023-10-17T18:54:07.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-17T18:54:19.332 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.333 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.333 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.334 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.334 DEBUG:boto:Method: GET 2023-10-17T18:54:19.335 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.335 DEBUG:boto:Data: 2023-10-17T18:54:19.335 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.336 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.337 DEBUG:boto:Params: {} 2023-10-17T18:54:19.337 DEBUG:boto:Token: None 2023-10-17T18:54:19.338 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.338 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c= 2023-10-17T18:54:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c=', 'Content-Length': '0'} 2023-10-17T18:54:19.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b94d39a2150abe-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.342 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.342 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.343 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.343 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.344 DEBUG:boto:Method: GET 2023-10-17T18:54:19.344 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.344 DEBUG:boto:Data: 2023-10-17T18:54:19.345 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.346 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.346 DEBUG:boto:Params: {} 2023-10-17T18:54:19.347 DEBUG:boto:Token: None 2023-10-17T18:54:19.347 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c= 2023-10-17T18:54:19.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c=', 'Content-Length': '0'} 2023-10-17T18:54:19.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4da8f213e6e9cf2-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.394 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.395 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.395 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.396 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.396 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.397 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.397 DEBUG:boto:Data: 2023-10-17T18:54:19.397 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.398 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.398 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.399 DEBUG:boto:Params: {} 2023-10-17T18:54:19.399 DEBUG:boto:Token: None 2023-10-17T18:54:19.400 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.400 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g02RNUJ7nINZaSJ/15AZem3MAOA= 2023-10-17T18:54:19.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g02RNUJ7nINZaSJ/15AZem3MAOA=', 'Content-Length': '0'} 2023-10-17T18:54:19.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebbc89269eb21c91-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.403 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.404 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.404 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.405 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.405 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.405 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.406 DEBUG:boto:Data: 2023-10-17T18:54:19.406 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.407 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.407 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.408 DEBUG:boto:Params: {} 2023-10-17T18:54:19.408 DEBUG:boto:Token: None 2023-10-17T18:54:19.409 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.409 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g02RNUJ7nINZaSJ/15AZem3MAOA= 2023-10-17T18:54:19.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g02RNUJ7nINZaSJ/15AZem3MAOA=', 'Content-Length': '0'} 2023-10-17T18:54:19.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900a1db4502252f7-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-17T18:54:19.413 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.413 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.413 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.414 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.414 DEBUG:boto:Method: GET 2023-10-17T18:54:19.415 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.415 DEBUG:boto:Data: 2023-10-17T18:54:19.416 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.416 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.416 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.417 DEBUG:boto:Params: {} 2023-10-17T18:54:19.417 DEBUG:boto:Token: None 2023-10-17T18:54:19.418 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.418 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c= 2023-10-17T18:54:19.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c=', 'Content-Length': '0'} 2023-10-17T18:54:19.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3d9969af50fbda-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.462 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.463 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.463 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.464 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.464 DEBUG:boto:Method: GET 2023-10-17T18:54:19.465 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.465 DEBUG:boto:Data: 2023-10-17T18:54:19.466 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.466 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.467 DEBUG:boto:Params: {} 2023-10-17T18:54:19.467 DEBUG:boto:Token: None 2023-10-17T18:54:19.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028?versionId=null 2023-10-17T18:54:19.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c= 2023-10-17T18:54:19.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FTZw+S/M/+j4UiMjyEGIiT63O9c=', 'Content-Length': '0'} 2023-10-17T18:54:19.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdc877391109752-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.472 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.472 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.473 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.473 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.474 DEBUG:boto:Method: GET 2023-10-17T18:54:19.474 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.475 DEBUG:boto:Data: 2023-10-17T18:54:19.475 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.475 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.476 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.476 DEBUG:boto:Params: {} 2023-10-17T18:54:19.477 DEBUG:boto:Token: None 2023-10-17T18:54:19.477 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.478 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009326d2ece72aa1fd-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.482 DEBUG:boto:b'hbmuog-58obj000000000000000000281000falseobj000000000000000000282023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.482 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.483 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.483 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.484 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.484 DEBUG:boto:Method: GET 2023-10-17T18:54:19.485 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.485 DEBUG:boto:Data: 2023-10-17T18:54:19.485 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.486 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.487 DEBUG:boto:Params: {} 2023-10-17T18:54:19.487 DEBUG:boto:Token: None 2023-10-17T18:54:19.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070b2bbbb89bcf28a-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.495 DEBUG:boto:b'hbmuog-58obj000000000000000000281000falseobj000000000000000000282023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.495 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.496 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.496 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.497 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.497 DEBUG:boto:Data: 2023-10-17T18:54:19.497 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.498 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.498 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.499 DEBUG:boto:Params: {} 2023-10-17T18:54:19.499 DEBUG:boto:Token: None 2023-10-17T18:54:19.500 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rGsBxxq/BiSFQCmC+uG1ST4k0J0= 2023-10-17T18:54:19.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rGsBxxq/BiSFQCmC+uG1ST4k0J0=', 'Content-Length': '0'} 2023-10-17T18:54:19.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198be09e7660652b-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.503 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.503 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.504 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.504 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.505 DEBUG:boto:Data: 2023-10-17T18:54:19.505 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.506 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.506 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.506 DEBUG:boto:Params: {} 2023-10-17T18:54:19.507 DEBUG:boto:Token: None 2023-10-17T18:54:19.507 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.508 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rGsBxxq/BiSFQCmC+uG1ST4k0J0= 2023-10-17T18:54:19.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rGsBxxq/BiSFQCmC+uG1ST4k0J0=', 'Content-Length': '0'} 2023-10-17T18:54:19.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e7cfe04862b509-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-17T18:54:19.511 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.511 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.512 DEBUG:boto:Method: GET 2023-10-17T18:54:19.512 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.513 DEBUG:boto:Data: 2023-10-17T18:54:19.513 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.514 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.515 DEBUG:boto:Params: {} 2023-10-17T18:54:19.515 DEBUG:boto:Token: None 2023-10-17T18:54:19.516 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.516 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BrpyeXGQRX35zzLOUuJqucBCtAY= 2023-10-17T18:54:19.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BrpyeXGQRX35zzLOUuJqucBCtAY=', 'Content-Length': '0'} 2023-10-17T18:54:19.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d6e60033d4276d-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.519 DEBUG:boto:path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.520 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.520 DEBUG:boto:Method: GET 2023-10-17T18:54:19.521 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.521 DEBUG:boto:Data: 2023-10-17T18:54:19.521 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.522 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.522 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.523 DEBUG:boto:Params: {} 2023-10-17T18:54:19.523 DEBUG:boto:Token: None 2023-10-17T18:54:19.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000028 2023-10-17T18:54:19.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BrpyeXGQRX35zzLOUuJqucBCtAY= 2023-10-17T18:54:19.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BrpyeXGQRX35zzLOUuJqucBCtAY=', 'Content-Length': '0'} 2023-10-17T18:54:19.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89b40ae4efdcf15-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.570 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.571 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.571 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.572 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.572 DEBUG:boto:Method: GET 2023-10-17T18:54:19.573 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.573 DEBUG:boto:Data: 2023-10-17T18:54:19.574 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.574 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.575 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.575 DEBUG:boto:Params: {} 2023-10-17T18:54:19.576 DEBUG:boto:Token: None 2023-10-17T18:54:19.576 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.577 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207bb6deb2575c6e-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.580 DEBUG:boto:b'hbmuog-58obj000000000000000000281000falseobj000000000000000000282023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.581 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.582 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.582 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.583 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.583 DEBUG:boto:Method: GET 2023-10-17T18:54:19.583 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000028 2023-10-17T18:54:19.584 DEBUG:boto:Data: 2023-10-17T18:54:19.584 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.585 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.586 DEBUG:boto:Params: {} 2023-10-17T18:54:19.586 DEBUG:boto:Token: None 2023-10-17T18:54:19.587 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd94f9f4091076a-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.592 DEBUG:boto:b'hbmuog-58obj000000000000000000281000falseobj000000000000000000282023-10-17T18:54:01.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-17T18:54:19.593 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.594 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.594 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.595 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.595 DEBUG:boto:Method: GET 2023-10-17T18:54:19.595 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.596 DEBUG:boto:Data: 2023-10-17T18:54:19.596 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.597 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.597 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.598 DEBUG:boto:Params: {} 2023-10-17T18:54:19.598 DEBUG:boto:Token: None 2023-10-17T18:54:19.599 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.599 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY= 2023-10-17T18:54:19.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY=', 'Content-Length': '0'} 2023-10-17T18:54:19.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974078f4b68abc82-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.642 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.643 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.643 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.644 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.644 DEBUG:boto:Method: GET 2023-10-17T18:54:19.645 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.645 DEBUG:boto:Data: 2023-10-17T18:54:19.646 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.646 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.647 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.647 DEBUG:boto:Params: {} 2023-10-17T18:54:19.647 DEBUG:boto:Token: None 2023-10-17T18:54:19.648 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY= 2023-10-17T18:54:19.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY=', 'Content-Length': '0'} 2023-10-17T18:54:19.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005341b993b2f77b21-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.652 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.653 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.653 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.653 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.654 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.654 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.655 DEBUG:boto:Data: 2023-10-17T18:54:19.655 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.655 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.656 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.656 DEBUG:boto:Params: {} 2023-10-17T18:54:19.657 DEBUG:boto:Token: None 2023-10-17T18:54:19.657 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xpSFSdQGmRoZ5glr53C4+ucgmIA= 2023-10-17T18:54:19.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xpSFSdQGmRoZ5glr53C4+ucgmIA=', 'Content-Length': '0'} 2023-10-17T18:54:19.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006784a905ced85e83-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.661 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.661 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.662 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.662 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.662 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.663 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.663 DEBUG:boto:Data: 2023-10-17T18:54:19.664 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.664 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.664 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.665 DEBUG:boto:Params: {} 2023-10-17T18:54:19.665 DEBUG:boto:Token: None 2023-10-17T18:54:19.666 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xpSFSdQGmRoZ5glr53C4+ucgmIA= 2023-10-17T18:54:19.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xpSFSdQGmRoZ5glr53C4+ucgmIA=', 'Content-Length': '0'} 2023-10-17T18:54:19.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7815c56bda90629-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-17T18:54:19.670 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.670 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.671 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.671 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.672 DEBUG:boto:Method: GET 2023-10-17T18:54:19.672 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.673 DEBUG:boto:Data: 2023-10-17T18:54:19.673 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.674 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.674 DEBUG:boto:Params: {} 2023-10-17T18:54:19.675 DEBUG:boto:Token: None 2023-10-17T18:54:19.675 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY= 2023-10-17T18:54:19.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY=', 'Content-Length': '0'} 2023-10-17T18:54:19.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a25de352fa85ae8-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.679 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.680 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.680 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.680 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.681 DEBUG:boto:Method: GET 2023-10-17T18:54:19.681 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.682 DEBUG:boto:Data: 2023-10-17T18:54:19.682 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.683 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.684 DEBUG:boto:Params: {} 2023-10-17T18:54:19.684 DEBUG:boto:Token: None 2023-10-17T18:54:19.685 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029?versionId=null 2023-10-17T18:54:19.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY= 2023-10-17T18:54:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5edGhgoi4ZcDjAzAOzHvJJ8UIUY=', 'Content-Length': '0'} 2023-10-17T18:54:19.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8b552236d857f6-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.688 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.689 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.689 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.689 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.690 DEBUG:boto:Method: GET 2023-10-17T18:54:19.690 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.691 DEBUG:boto:Data: 2023-10-17T18:54:19.691 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.692 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.692 DEBUG:boto:Params: {} 2023-10-17T18:54:19.693 DEBUG:boto:Token: None 2023-10-17T18:54:19.693 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c8ae9d8960cdcf-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.697 DEBUG:boto:b'hbmuog-58obj000000000000000000291000falseobj000000000000000000292023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.698 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.699 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.699 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.700 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.700 DEBUG:boto:Method: GET 2023-10-17T18:54:19.700 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.701 DEBUG:boto:Data: 2023-10-17T18:54:19.701 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.702 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.702 DEBUG:boto:Params: {} 2023-10-17T18:54:19.703 DEBUG:boto:Token: None 2023-10-17T18:54:19.704 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fbbd84dac03aed-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.709 DEBUG:boto:b'hbmuog-58obj000000000000000000291000falseobj000000000000000000292023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.710 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.710 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.711 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.711 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.711 DEBUG:boto:Data: 2023-10-17T18:54:19.712 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.712 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.713 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.713 DEBUG:boto:Params: {} 2023-10-17T18:54:19.714 DEBUG:boto:Token: None 2023-10-17T18:54:19.714 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gJBoYvpkRbRAszgDuzHtMSgbqKw= 2023-10-17T18:54:19.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gJBoYvpkRbRAszgDuzHtMSgbqKw=', 'Content-Length': '0'} 2023-10-17T18:54:19.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e3c6d4ec99415b-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.718 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.718 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.719 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.719 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.720 DEBUG:boto:Data: 2023-10-17T18:54:19.720 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.720 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.721 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.721 DEBUG:boto:Params: {} 2023-10-17T18:54:19.722 DEBUG:boto:Token: None 2023-10-17T18:54:19.722 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gJBoYvpkRbRAszgDuzHtMSgbqKw= 2023-10-17T18:54:19.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gJBoYvpkRbRAszgDuzHtMSgbqKw=', 'Content-Length': '0'} 2023-10-17T18:54:19.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18e7f881e772ecf-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-17T18:54:19.726 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.727 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.727 DEBUG:boto:Method: GET 2023-10-17T18:54:19.727 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.728 DEBUG:boto:Data: 2023-10-17T18:54:19.728 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.729 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.730 DEBUG:boto:Params: {} 2023-10-17T18:54:19.730 DEBUG:boto:Token: None 2023-10-17T18:54:19.731 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2HP+YskO1LCIzgTC74DPgJCZvOA= 2023-10-17T18:54:19.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2HP+YskO1LCIzgTC74DPgJCZvOA=', 'Content-Length': '0'} 2023-10-17T18:54:19.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d30f43ffd24c0af-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.734 DEBUG:boto:path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.735 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.735 DEBUG:boto:Method: GET 2023-10-17T18:54:19.735 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.736 DEBUG:boto:Data: 2023-10-17T18:54:19.736 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.737 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.738 DEBUG:boto:Params: {} 2023-10-17T18:54:19.738 DEBUG:boto:Token: None 2023-10-17T18:54:19.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000029 2023-10-17T18:54:19.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2HP+YskO1LCIzgTC74DPgJCZvOA= 2023-10-17T18:54:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2HP+YskO1LCIzgTC74DPgJCZvOA=', 'Content-Length': '0'} 2023-10-17T18:54:19.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000102a9a4a79479bc-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.782 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.783 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.784 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.784 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.784 DEBUG:boto:Method: GET 2023-10-17T18:54:19.785 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.785 DEBUG:boto:Data: 2023-10-17T18:54:19.786 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.786 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.787 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.787 DEBUG:boto:Params: {} 2023-10-17T18:54:19.788 DEBUG:boto:Token: None 2023-10-17T18:54:19.788 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.789 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb238e208a2e718-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.792 DEBUG:boto:b'hbmuog-58obj000000000000000000291000falseobj000000000000000000292023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.793 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.794 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.794 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.795 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.795 DEBUG:boto:Method: GET 2023-10-17T18:54:19.795 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000029 2023-10-17T18:54:19.796 DEBUG:boto:Data: 2023-10-17T18:54:19.796 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.797 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.798 DEBUG:boto:Params: {} 2023-10-17T18:54:19.798 DEBUG:boto:Token: None 2023-10-17T18:54:19.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.799 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000394635867bb00a6c-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.805 DEBUG:boto:b'hbmuog-58obj000000000000000000291000falseobj000000000000000000292023-10-17T18:54:06.320Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-17T18:54:19.806 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.806 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.807 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.807 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.807 DEBUG:boto:Method: GET 2023-10-17T18:54:19.808 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.808 DEBUG:boto:Data: 2023-10-17T18:54:19.809 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.810 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.810 DEBUG:boto:Params: {} 2023-10-17T18:54:19.811 DEBUG:boto:Token: None 2023-10-17T18:54:19.811 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs= 2023-10-17T18:54:19.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs=', 'Content-Length': '0'} 2023-10-17T18:54:19.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0e1f5836206c0b-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.858 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.859 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.860 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.860 DEBUG:boto:Method: GET 2023-10-17T18:54:19.861 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.861 DEBUG:boto:Data: 2023-10-17T18:54:19.862 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.863 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.863 DEBUG:boto:Params: {} 2023-10-17T18:54:19.863 DEBUG:boto:Token: None 2023-10-17T18:54:19.864 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs= 2023-10-17T18:54:19.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs=', 'Content-Length': '0'} 2023-10-17T18:54:19.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e563bcc7587223f3-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.868 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.868 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.869 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.869 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.870 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.870 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.871 DEBUG:boto:Data: 2023-10-17T18:54:19.871 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.872 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.872 DEBUG:boto:Params: {} 2023-10-17T18:54:19.873 DEBUG:boto:Token: None 2023-10-17T18:54:19.873 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ajMpiCbDgGHmrBCqnvkC1xXhm4= 2023-10-17T18:54:19.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ajMpiCbDgGHmrBCqnvkC1xXhm4=', 'Content-Length': '0'} 2023-10-17T18:54:19.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef73648c43d2bddd-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.877 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.877 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.878 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.878 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.879 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.879 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.880 DEBUG:boto:Data: 2023-10-17T18:54:19.880 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.881 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.881 DEBUG:boto:Params: {} 2023-10-17T18:54:19.882 DEBUG:boto:Token: None 2023-10-17T18:54:19.882 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.883 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7ajMpiCbDgGHmrBCqnvkC1xXhm4= 2023-10-17T18:54:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7ajMpiCbDgGHmrBCqnvkC1xXhm4=', 'Content-Length': '0'} 2023-10-17T18:54:19.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e9fe7666dbf4ff-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-17T18:54:19.886 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.887 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.887 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.888 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.888 DEBUG:boto:Method: GET 2023-10-17T18:54:19.889 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.889 DEBUG:boto:Data: 2023-10-17T18:54:19.889 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.890 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.891 DEBUG:boto:Params: {} 2023-10-17T18:54:19.891 DEBUG:boto:Token: None 2023-10-17T18:54:19.892 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs= 2023-10-17T18:54:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs=', 'Content-Length': '0'} 2023-10-17T18:54:19.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014a06a240cb13cc-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.895 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.896 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.896 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.897 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.897 DEBUG:boto:Method: GET 2023-10-17T18:54:19.898 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.898 DEBUG:boto:Data: 2023-10-17T18:54:19.898 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.899 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.900 DEBUG:boto:Params: {} 2023-10-17T18:54:19.900 DEBUG:boto:Token: None 2023-10-17T18:54:19.901 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030?versionId=null 2023-10-17T18:54:19.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs= 2023-10-17T18:54:19.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZdY/wFE+uM3Eb1oWY9isi/av1xs=', 'Content-Length': '0'} 2023-10-17T18:54:19.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003189c0aacf69e88a-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.905 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.905 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.905 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.906 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.906 DEBUG:boto:Method: GET 2023-10-17T18:54:19.907 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.907 DEBUG:boto:Data: 2023-10-17T18:54:19.908 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.909 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.909 DEBUG:boto:Params: {} 2023-10-17T18:54:19.909 DEBUG:boto:Token: None 2023-10-17T18:54:19.910 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a79d9d9c1b54ce3-00652ed85b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.914 DEBUG:boto:b'hbmuog-58obj000000000000000000301000falseobj000000000000000000302023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.915 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:19.916 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:19.916 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.916 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.917 DEBUG:boto:Method: GET 2023-10-17T18:54:19.917 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:19.918 DEBUG:boto:Data: 2023-10-17T18:54:19.918 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.919 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.919 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.920 DEBUG:boto:Params: {} 2023-10-17T18:54:19.920 DEBUG:boto:Token: None 2023-10-17T18:54:19.921 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/ 2023-10-17T18:54:19.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0= 2023-10-17T18:54:19.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1Qho1wobkd8kYWYq2f1gJ9/pNc0=', 'Content-Length': '0'} 2023-10-17T18:54:19.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63da2fd783c4205-00652ed85b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.927 DEBUG:boto:b'hbmuog-58obj000000000000000000301000falseobj000000000000000000302023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:19.927 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.928 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.928 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.929 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.929 DEBUG:boto:Data: 2023-10-17T18:54:19.929 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.930 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.931 DEBUG:boto:Params: {} 2023-10-17T18:54:19.931 DEBUG:boto:Token: None 2023-10-17T18:54:19.932 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eBaxOXnQX1KwIMCAyUF4kXbWLtA= 2023-10-17T18:54:19.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eBaxOXnQX1KwIMCAyUF4kXbWLtA=', 'Content-Length': '0'} 2023-10-17T18:54:19.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009423ae7294976817-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.935 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.935 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.936 DEBUG:boto:Method: HEAD 2023-10-17T18:54:19.936 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.937 DEBUG:boto:Data: 2023-10-17T18:54:19.937 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.938 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.938 DEBUG:boto:Params: {} 2023-10-17T18:54:19.939 DEBUG:boto:Token: None 2023-10-17T18:54:19.939 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.940 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eBaxOXnQX1KwIMCAyUF4kXbWLtA= 2023-10-17T18:54:19.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eBaxOXnQX1KwIMCAyUF4kXbWLtA=', 'Content-Length': '0'} 2023-10-17T18:54:19.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050640844ed270c00-00652ed85b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-17T18:54:19.943 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.944 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.944 DEBUG:boto:Method: GET 2023-10-17T18:54:19.945 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.945 DEBUG:boto:Data: 2023-10-17T18:54:19.946 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.946 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:19.946 DEBUG:boto:Port: 8001 2023-10-17T18:54:19.947 DEBUG:boto:Params: {} 2023-10-17T18:54:19.947 DEBUG:boto:Token: None 2023-10-17T18:54:19.949 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.949 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fO8iWXcPt8JV8MFyFBP/wwaaGs= 2023-10-17T18:54:19.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fO8iWXcPt8JV8MFyFBP/wwaaGs=', 'Content-Length': '0'} 2023-10-17T18:54:19.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821f45b91dba237d-00652ed85b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:19.994 DEBUG:boto:path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.995 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.996 DEBUG:boto:Method: GET 2023-10-17T18:54:19.996 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:19.996 DEBUG:boto:Data: 2023-10-17T18:54:19.997 DEBUG:boto:Headers: {} 2023-10-17T18:54:19.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:19.998 DEBUG:boto:Port: 8000 2023-10-17T18:54:19.998 DEBUG:boto:Params: {} 2023-10-17T18:54:19.999 DEBUG:boto:Token: None 2023-10-17T18:54:19.999 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:19 GMT /hbmuog-58/obj00000000000000000030 2023-10-17T18:54:20.000 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9fO8iWXcPt8JV8MFyFBP/wwaaGs= 2023-10-17T18:54:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fO8iWXcPt8JV8MFyFBP/wwaaGs=', 'Content-Length': '0'} 2023-10-17T18:54:20.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf2c869a4c00343-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.004 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.004 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.005 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.005 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.005 DEBUG:boto:Method: GET 2023-10-17T18:54:20.006 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.006 DEBUG:boto:Data: 2023-10-17T18:54:20.007 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.007 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.008 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.008 DEBUG:boto:Params: {} 2023-10-17T18:54:20.009 DEBUG:boto:Token: None 2023-10-17T18:54:20.009 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.010 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a688e1d6af24c383-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.014 DEBUG:boto:b'hbmuog-58obj000000000000000000301000falseobj000000000000000000302023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.015 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.015 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.016 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.016 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.016 DEBUG:boto:Method: GET 2023-10-17T18:54:20.017 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000030 2023-10-17T18:54:20.017 DEBUG:boto:Data: 2023-10-17T18:54:20.018 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.019 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.019 DEBUG:boto:Params: {} 2023-10-17T18:54:20.020 DEBUG:boto:Token: None 2023-10-17T18:54:20.020 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055641678666f8684-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.026 DEBUG:boto:b'hbmuog-58obj000000000000000000301000falseobj000000000000000000302023-10-17T18:54:07.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-17T18:54:20.027 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.028 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.028 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.029 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.029 DEBUG:boto:Method: GET 2023-10-17T18:54:20.030 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.030 DEBUG:boto:Data: 2023-10-17T18:54:20.031 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.031 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.032 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.032 DEBUG:boto:Params: {} 2023-10-17T18:54:20.033 DEBUG:boto:Token: None 2023-10-17T18:54:20.033 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs= 2023-10-17T18:54:20.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs=', 'Content-Length': '0'} 2023-10-17T18:54:20.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712958dede6f89c7-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.037 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.038 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.038 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.039 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.039 DEBUG:boto:Method: GET 2023-10-17T18:54:20.040 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.040 DEBUG:boto:Data: 2023-10-17T18:54:20.040 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.041 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.041 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.041 DEBUG:boto:Params: {} 2023-10-17T18:54:20.042 DEBUG:boto:Token: None 2023-10-17T18:54:20.042 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.043 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs= 2023-10-17T18:54:20.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs=', 'Content-Length': '0'} 2023-10-17T18:54:20.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099de425536335696-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.046 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.047 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.047 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.048 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.048 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.048 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.049 DEBUG:boto:Data: 2023-10-17T18:54:20.049 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.050 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.050 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.050 DEBUG:boto:Params: {} 2023-10-17T18:54:20.051 DEBUG:boto:Token: None 2023-10-17T18:54:20.051 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.052 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TV30VpUKXZD8uMLLgo6DFDoYNfo= 2023-10-17T18:54:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TV30VpUKXZD8uMLLgo6DFDoYNfo=', 'Content-Length': '0'} 2023-10-17T18:54:20.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067237f5c7a70d846-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.055 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.055 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.056 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.056 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.057 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.057 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.058 DEBUG:boto:Data: 2023-10-17T18:54:20.058 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.059 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.059 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.059 DEBUG:boto:Params: {} 2023-10-17T18:54:20.060 DEBUG:boto:Token: None 2023-10-17T18:54:20.060 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.061 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TV30VpUKXZD8uMLLgo6DFDoYNfo= 2023-10-17T18:54:20.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TV30VpUKXZD8uMLLgo6DFDoYNfo=', 'Content-Length': '0'} 2023-10-17T18:54:20.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3a6d1e58bc9322-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-17T18:54:20.064 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.065 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.065 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.066 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.066 DEBUG:boto:Method: GET 2023-10-17T18:54:20.067 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.067 DEBUG:boto:Data: 2023-10-17T18:54:20.067 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.068 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.068 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.069 DEBUG:boto:Params: {} 2023-10-17T18:54:20.069 DEBUG:boto:Token: None 2023-10-17T18:54:20.070 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.070 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs= 2023-10-17T18:54:20.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs=', 'Content-Length': '0'} 2023-10-17T18:54:20.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b903624c36700ce-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.073 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.074 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.074 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.075 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.075 DEBUG:boto:Method: GET 2023-10-17T18:54:20.076 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.076 DEBUG:boto:Data: 2023-10-17T18:54:20.077 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.077 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.077 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.078 DEBUG:boto:Params: {} 2023-10-17T18:54:20.078 DEBUG:boto:Token: None 2023-10-17T18:54:20.079 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031?versionId=null 2023-10-17T18:54:20.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs= 2023-10-17T18:54:20.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JYF7YddklqeVVja9rtZ9N9Mlwgs=', 'Content-Length': '0'} 2023-10-17T18:54:20.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000441499d238ae465-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.126 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.127 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.128 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.128 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.128 DEBUG:boto:Method: GET 2023-10-17T18:54:20.129 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.129 DEBUG:boto:Data: 2023-10-17T18:54:20.130 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.131 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.131 DEBUG:boto:Params: {} 2023-10-17T18:54:20.132 DEBUG:boto:Token: None 2023-10-17T18:54:20.132 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d394429407056b7-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.136 DEBUG:boto:b'hbmuog-58obj000000000000000000311000falseobj000000000000000000312023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.137 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.138 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.138 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.138 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.139 DEBUG:boto:Method: GET 2023-10-17T18:54:20.139 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.140 DEBUG:boto:Data: 2023-10-17T18:54:20.140 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.141 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.141 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.142 DEBUG:boto:Params: {} 2023-10-17T18:54:20.142 DEBUG:boto:Token: None 2023-10-17T18:54:20.143 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.143 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c97f485260e7513-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.149 DEBUG:boto:b'hbmuog-58obj000000000000000000311000falseobj000000000000000000312023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.149 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.150 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.150 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.151 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.151 DEBUG:boto:Data: 2023-10-17T18:54:20.152 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.152 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.153 DEBUG:boto:Params: {} 2023-10-17T18:54:20.153 DEBUG:boto:Token: None 2023-10-17T18:54:20.154 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jPtGOHSZCQx+MWS6ni/LyGb0NPQ= 2023-10-17T18:54:20.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jPtGOHSZCQx+MWS6ni/LyGb0NPQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e782df8ce462311-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.157 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.158 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.158 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.159 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.159 DEBUG:boto:Data: 2023-10-17T18:54:20.160 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.161 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.161 DEBUG:boto:Params: {} 2023-10-17T18:54:20.161 DEBUG:boto:Token: None 2023-10-17T18:54:20.162 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.162 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jPtGOHSZCQx+MWS6ni/LyGb0NPQ= 2023-10-17T18:54:20.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jPtGOHSZCQx+MWS6ni/LyGb0NPQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219e706e4b556f93-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-17T18:54:20.166 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.167 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.167 DEBUG:boto:Method: GET 2023-10-17T18:54:20.167 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.168 DEBUG:boto:Data: 2023-10-17T18:54:20.168 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.169 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.169 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.170 DEBUG:boto:Params: {} 2023-10-17T18:54:20.170 DEBUG:boto:Token: None 2023-10-17T18:54:20.171 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FRQhZt+4MQ7xBSld8zPRdaGkVmI= 2023-10-17T18:54:20.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FRQhZt+4MQ7xBSld8zPRdaGkVmI=', 'Content-Length': '0'} 2023-10-17T18:54:20.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfab606e19e53d3c-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.218 DEBUG:boto:path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.219 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.219 DEBUG:boto:Method: GET 2023-10-17T18:54:20.220 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.220 DEBUG:boto:Data: 2023-10-17T18:54:20.221 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.221 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.222 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.222 DEBUG:boto:Params: {} 2023-10-17T18:54:20.223 DEBUG:boto:Token: None 2023-10-17T18:54:20.223 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000031 2023-10-17T18:54:20.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FRQhZt+4MQ7xBSld8zPRdaGkVmI= 2023-10-17T18:54:20.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FRQhZt+4MQ7xBSld8zPRdaGkVmI=', 'Content-Length': '0'} 2023-10-17T18:54:20.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b1355dbb88f396-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.227 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.228 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.228 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.228 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.229 DEBUG:boto:Method: GET 2023-10-17T18:54:20.229 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.230 DEBUG:boto:Data: 2023-10-17T18:54:20.230 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.231 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.232 DEBUG:boto:Params: {} 2023-10-17T18:54:20.232 DEBUG:boto:Token: None 2023-10-17T18:54:20.233 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.233 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b151994204e3d6-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.237 DEBUG:boto:b'hbmuog-58obj000000000000000000311000falseobj000000000000000000312023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.237 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.238 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.238 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.239 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.239 DEBUG:boto:Method: GET 2023-10-17T18:54:20.240 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000031 2023-10-17T18:54:20.240 DEBUG:boto:Data: 2023-10-17T18:54:20.240 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.241 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.242 DEBUG:boto:Params: {} 2023-10-17T18:54:20.242 DEBUG:boto:Token: None 2023-10-17T18:54:20.243 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066d73f7f5fb80970-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.294 DEBUG:boto:b'hbmuog-58obj000000000000000000311000falseobj000000000000000000312023-10-17T18:54:05.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-17T18:54:20.296 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.296 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.296 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.297 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.297 DEBUG:boto:Method: GET 2023-10-17T18:54:20.298 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.298 DEBUG:boto:Data: 2023-10-17T18:54:20.299 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.299 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.300 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.300 DEBUG:boto:Params: {} 2023-10-17T18:54:20.300 DEBUG:boto:Token: None 2023-10-17T18:54:20.301 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.301 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ= 2023-10-17T18:54:20.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57ccda55351e9b0-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.305 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.306 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.306 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.306 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.307 DEBUG:boto:Method: GET 2023-10-17T18:54:20.307 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.308 DEBUG:boto:Data: 2023-10-17T18:54:20.308 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.308 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.309 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.309 DEBUG:boto:Params: {} 2023-10-17T18:54:20.310 DEBUG:boto:Token: None 2023-10-17T18:54:20.310 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ= 2023-10-17T18:54:20.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc58f8f73922108d-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.314 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.315 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.315 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.315 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.316 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.316 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.316 DEBUG:boto:Data: 2023-10-17T18:54:20.317 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.318 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.318 DEBUG:boto:Params: {} 2023-10-17T18:54:20.319 DEBUG:boto:Token: None 2023-10-17T18:54:20.319 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bz4DMtOc9lPCVcvperK0YNG/aJs= 2023-10-17T18:54:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bz4DMtOc9lPCVcvperK0YNG/aJs=', 'Content-Length': '0'} 2023-10-17T18:54:20.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa108bbd404e2fea-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.322 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.323 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.323 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.324 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.324 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.324 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.325 DEBUG:boto:Data: 2023-10-17T18:54:20.325 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.326 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.326 DEBUG:boto:Params: {} 2023-10-17T18:54:20.327 DEBUG:boto:Token: None 2023-10-17T18:54:20.327 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bz4DMtOc9lPCVcvperK0YNG/aJs= 2023-10-17T18:54:20.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bz4DMtOc9lPCVcvperK0YNG/aJs=', 'Content-Length': '0'} 2023-10-17T18:54:20.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef0fc8682e5aa04-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-17T18:54:20.331 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.331 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.332 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.332 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.333 DEBUG:boto:Method: GET 2023-10-17T18:54:20.333 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.334 DEBUG:boto:Data: 2023-10-17T18:54:20.334 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.334 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.335 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.335 DEBUG:boto:Params: {} 2023-10-17T18:54:20.336 DEBUG:boto:Token: None 2023-10-17T18:54:20.336 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ= 2023-10-17T18:54:20.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074c75a4044d0d82c-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.340 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.341 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.341 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.342 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.342 DEBUG:boto:Method: GET 2023-10-17T18:54:20.342 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.343 DEBUG:boto:Data: 2023-10-17T18:54:20.343 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.344 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.344 DEBUG:boto:Params: {} 2023-10-17T18:54:20.345 DEBUG:boto:Token: None 2023-10-17T18:54:20.345 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032?versionId=null 2023-10-17T18:54:20.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ= 2023-10-17T18:54:20.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JHpOVfoi/OjW41UwiMZ/8FSC/VQ=', 'Content-Length': '0'} 2023-10-17T18:54:20.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013618ca5052b6c39-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.349 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.350 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.350 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.351 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.351 DEBUG:boto:Method: GET 2023-10-17T18:54:20.352 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.352 DEBUG:boto:Data: 2023-10-17T18:54:20.352 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.353 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.354 DEBUG:boto:Params: {} 2023-10-17T18:54:20.354 DEBUG:boto:Token: None 2023-10-17T18:54:20.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de737923646caa4-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.359 DEBUG:boto:b'hbmuog-58obj000000000000000000321000falseobj000000000000000000322023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.360 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.361 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.362 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.362 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.363 DEBUG:boto:Method: GET 2023-10-17T18:54:20.363 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.363 DEBUG:boto:Data: 2023-10-17T18:54:20.364 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.364 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.365 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.365 DEBUG:boto:Params: {} 2023-10-17T18:54:20.366 DEBUG:boto:Token: None 2023-10-17T18:54:20.366 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b136708b9556adc9-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.372 DEBUG:boto:b'hbmuog-58obj000000000000000000321000falseobj000000000000000000322023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.373 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.373 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.374 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.374 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.374 DEBUG:boto:Data: 2023-10-17T18:54:20.375 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.375 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.376 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.376 DEBUG:boto:Params: {} 2023-10-17T18:54:20.376 DEBUG:boto:Token: None 2023-10-17T18:54:20.377 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cnp+NychDnEw7Hei6pmh9VSbz5k= 2023-10-17T18:54:20.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cnp+NychDnEw7Hei6pmh9VSbz5k=', 'Content-Length': '0'} 2023-10-17T18:54:20.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c23583a185d7cdd-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.380 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.381 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.381 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.382 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.382 DEBUG:boto:Data: 2023-10-17T18:54:20.382 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.383 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.384 DEBUG:boto:Params: {} 2023-10-17T18:54:20.384 DEBUG:boto:Token: None 2023-10-17T18:54:20.384 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cnp+NychDnEw7Hei6pmh9VSbz5k= 2023-10-17T18:54:20.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cnp+NychDnEw7Hei6pmh9VSbz5k=', 'Content-Length': '0'} 2023-10-17T18:54:20.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26076df6cb97f43-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-17T18:54:20.388 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.389 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.389 DEBUG:boto:Method: GET 2023-10-17T18:54:20.389 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.390 DEBUG:boto:Data: 2023-10-17T18:54:20.390 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.391 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.392 DEBUG:boto:Params: {} 2023-10-17T18:54:20.392 DEBUG:boto:Token: None 2023-10-17T18:54:20.393 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D8e5Z0+rDOo+pafp7NwglqkOZoY= 2023-10-17T18:54:20.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D8e5Z0+rDOo+pafp7NwglqkOZoY=', 'Content-Length': '0'} 2023-10-17T18:54:20.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc945c52ee23f79d-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.396 DEBUG:boto:path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.397 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.397 DEBUG:boto:Method: GET 2023-10-17T18:54:20.398 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.398 DEBUG:boto:Data: 2023-10-17T18:54:20.398 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.399 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.400 DEBUG:boto:Params: {} 2023-10-17T18:54:20.400 DEBUG:boto:Token: None 2023-10-17T18:54:20.400 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000032 2023-10-17T18:54:20.401 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D8e5Z0+rDOo+pafp7NwglqkOZoY= 2023-10-17T18:54:20.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D8e5Z0+rDOo+pafp7NwglqkOZoY=', 'Content-Length': '0'} 2023-10-17T18:54:20.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3e5b8c3ce27d7b-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.450 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.451 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.451 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.452 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.452 DEBUG:boto:Method: GET 2023-10-17T18:54:20.453 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.453 DEBUG:boto:Data: 2023-10-17T18:54:20.453 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.454 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.455 DEBUG:boto:Params: {} 2023-10-17T18:54:20.455 DEBUG:boto:Token: None 2023-10-17T18:54:20.456 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f299f1d0e741a465-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.460 DEBUG:boto:b'hbmuog-58obj000000000000000000321000falseobj000000000000000000322023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.461 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.461 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.462 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.462 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.462 DEBUG:boto:Method: GET 2023-10-17T18:54:20.463 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000032 2023-10-17T18:54:20.463 DEBUG:boto:Data: 2023-10-17T18:54:20.464 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.464 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.465 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.465 DEBUG:boto:Params: {} 2023-10-17T18:54:20.465 DEBUG:boto:Token: None 2023-10-17T18:54:20.466 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.466 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c797f6c2753638f1-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.472 DEBUG:boto:b'hbmuog-58obj000000000000000000321000falseobj000000000000000000322023-10-17T18:54:04.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-17T18:54:20.473 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.473 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.474 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.474 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.475 DEBUG:boto:Method: GET 2023-10-17T18:54:20.475 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.475 DEBUG:boto:Data: 2023-10-17T18:54:20.476 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.476 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.477 DEBUG:boto:Params: {} 2023-10-17T18:54:20.477 DEBUG:boto:Token: None 2023-10-17T18:54:20.478 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.478 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU= 2023-10-17T18:54:20.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU=', 'Content-Length': '0'} 2023-10-17T18:54:20.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4d5c11187d688b-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.522 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.523 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.523 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.523 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.524 DEBUG:boto:Method: GET 2023-10-17T18:54:20.524 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.525 DEBUG:boto:Data: 2023-10-17T18:54:20.525 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.526 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.527 DEBUG:boto:Params: {} 2023-10-17T18:54:20.527 DEBUG:boto:Token: None 2023-10-17T18:54:20.528 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU= 2023-10-17T18:54:20.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU=', 'Content-Length': '0'} 2023-10-17T18:54:20.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f9ec772c3f7df1a-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.531 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.532 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.532 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.533 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.533 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.533 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.534 DEBUG:boto:Data: 2023-10-17T18:54:20.534 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.535 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.535 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.535 DEBUG:boto:Params: {} 2023-10-17T18:54:20.536 DEBUG:boto:Token: None 2023-10-17T18:54:20.536 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.537 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aHdrRBt6IHgAUuoI9PBFLYx2Ldc= 2023-10-17T18:54:20.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aHdrRBt6IHgAUuoI9PBFLYx2Ldc=', 'Content-Length': '0'} 2023-10-17T18:54:20.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be22afeab630417-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.539 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.540 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.540 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.541 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.541 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.541 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.542 DEBUG:boto:Data: 2023-10-17T18:54:20.542 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.543 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.543 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.543 DEBUG:boto:Params: {} 2023-10-17T18:54:20.544 DEBUG:boto:Token: None 2023-10-17T18:54:20.544 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aHdrRBt6IHgAUuoI9PBFLYx2Ldc= 2023-10-17T18:54:20.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aHdrRBt6IHgAUuoI9PBFLYx2Ldc=', 'Content-Length': '0'} 2023-10-17T18:54:20.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998f1c871ef9feb9-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-17T18:54:20.548 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.548 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.549 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.549 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.549 DEBUG:boto:Method: GET 2023-10-17T18:54:20.550 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.550 DEBUG:boto:Data: 2023-10-17T18:54:20.551 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.551 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.552 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.552 DEBUG:boto:Params: {} 2023-10-17T18:54:20.553 DEBUG:boto:Token: None 2023-10-17T18:54:20.553 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.553 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU= 2023-10-17T18:54:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU=', 'Content-Length': '0'} 2023-10-17T18:54:20.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36335a81f264659-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.557 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.557 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.557 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.558 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.558 DEBUG:boto:Method: GET 2023-10-17T18:54:20.559 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.559 DEBUG:boto:Data: 2023-10-17T18:54:20.560 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.560 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.561 DEBUG:boto:Params: {} 2023-10-17T18:54:20.561 DEBUG:boto:Token: None 2023-10-17T18:54:20.562 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033?versionId=null 2023-10-17T18:54:20.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU= 2023-10-17T18:54:20.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F/e41Qc3cGZ7DBkTnmQxPENyhsU=', 'Content-Length': '0'} 2023-10-17T18:54:20.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8007ea670f7a026-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.610 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.611 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.611 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.611 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.612 DEBUG:boto:Method: GET 2023-10-17T18:54:20.612 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.613 DEBUG:boto:Data: 2023-10-17T18:54:20.613 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.613 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.614 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.614 DEBUG:boto:Params: {} 2023-10-17T18:54:20.615 DEBUG:boto:Token: None 2023-10-17T18:54:20.615 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e3ee9e20ff22fb-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.619 DEBUG:boto:b'hbmuog-58obj000000000000000000331000falseobj000000000000000000332023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.620 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.620 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.621 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.621 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.622 DEBUG:boto:Method: GET 2023-10-17T18:54:20.622 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.622 DEBUG:boto:Data: 2023-10-17T18:54:20.623 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.624 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.624 DEBUG:boto:Params: {} 2023-10-17T18:54:20.625 DEBUG:boto:Token: None 2023-10-17T18:54:20.625 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.625 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6637c098175567-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.630 DEBUG:boto:b'hbmuog-58obj000000000000000000331000falseobj000000000000000000332023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.631 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.631 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.632 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.632 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.633 DEBUG:boto:Data: 2023-10-17T18:54:20.633 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.633 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.634 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.634 DEBUG:boto:Params: {} 2023-10-17T18:54:20.634 DEBUG:boto:Token: None 2023-10-17T18:54:20.635 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.635 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lTT048zZdhtRiM2kz8ZlBmJ2BNo= 2023-10-17T18:54:20.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lTT048zZdhtRiM2kz8ZlBmJ2BNo=', 'Content-Length': '0'} 2023-10-17T18:54:20.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000494aa521621531f5-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.638 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.639 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.639 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.639 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.640 DEBUG:boto:Data: 2023-10-17T18:54:20.640 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.641 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.641 DEBUG:boto:Params: {} 2023-10-17T18:54:20.642 DEBUG:boto:Token: None 2023-10-17T18:54:20.642 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lTT048zZdhtRiM2kz8ZlBmJ2BNo= 2023-10-17T18:54:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lTT048zZdhtRiM2kz8ZlBmJ2BNo=', 'Content-Length': '0'} 2023-10-17T18:54:20.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4c58f53d9add69-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-17T18:54:20.646 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.646 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.646 DEBUG:boto:Method: GET 2023-10-17T18:54:20.647 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.647 DEBUG:boto:Data: 2023-10-17T18:54:20.648 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.648 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.649 DEBUG:boto:Params: {} 2023-10-17T18:54:20.649 DEBUG:boto:Token: None 2023-10-17T18:54:20.650 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xwMuxEqoVkq2S954ZdHURCK7A+c= 2023-10-17T18:54:20.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xwMuxEqoVkq2S954ZdHURCK7A+c=', 'Content-Length': '0'} 2023-10-17T18:54:20.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dad688e90a228e6-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.694 DEBUG:boto:path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.695 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.695 DEBUG:boto:Method: GET 2023-10-17T18:54:20.696 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.696 DEBUG:boto:Data: 2023-10-17T18:54:20.696 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.697 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.698 DEBUG:boto:Params: {} 2023-10-17T18:54:20.698 DEBUG:boto:Token: None 2023-10-17T18:54:20.699 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000033 2023-10-17T18:54:20.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xwMuxEqoVkq2S954ZdHURCK7A+c= 2023-10-17T18:54:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xwMuxEqoVkq2S954ZdHURCK7A+c=', 'Content-Length': '0'} 2023-10-17T18:54:20.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000340006c28b78345b-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.703 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.703 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.704 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.704 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.705 DEBUG:boto:Method: GET 2023-10-17T18:54:20.705 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.705 DEBUG:boto:Data: 2023-10-17T18:54:20.706 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.707 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.707 DEBUG:boto:Params: {} 2023-10-17T18:54:20.708 DEBUG:boto:Token: None 2023-10-17T18:54:20.708 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d99ddac4ae9b960-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.712 DEBUG:boto:b'hbmuog-58obj000000000000000000331000falseobj000000000000000000332023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.713 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.713 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.714 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.714 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.715 DEBUG:boto:Method: GET 2023-10-17T18:54:20.715 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000033 2023-10-17T18:54:20.715 DEBUG:boto:Data: 2023-10-17T18:54:20.716 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.717 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.717 DEBUG:boto:Params: {} 2023-10-17T18:54:20.718 DEBUG:boto:Token: None 2023-10-17T18:54:20.718 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006192db9e5ad701ff-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.724 DEBUG:boto:b'hbmuog-58obj000000000000000000331000falseobj000000000000000000332023-10-17T18:54:08.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-17T18:54:20.725 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.725 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.726 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.726 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.727 DEBUG:boto:Method: GET 2023-10-17T18:54:20.727 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.728 DEBUG:boto:Data: 2023-10-17T18:54:20.728 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.729 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.729 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.729 DEBUG:boto:Params: {} 2023-10-17T18:54:20.730 DEBUG:boto:Token: None 2023-10-17T18:54:20.730 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg= 2023-10-17T18:54:20.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg=', 'Content-Length': '0'} 2023-10-17T18:54:20.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7d4e256f383573-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.734 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.735 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.735 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.735 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.736 DEBUG:boto:Method: GET 2023-10-17T18:54:20.736 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.737 DEBUG:boto:Data: 2023-10-17T18:54:20.737 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.738 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.738 DEBUG:boto:Params: {} 2023-10-17T18:54:20.739 DEBUG:boto:Token: None 2023-10-17T18:54:20.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg= 2023-10-17T18:54:20.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg=', 'Content-Length': '0'} 2023-10-17T18:54:20.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9acac6d085f4ae-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.790 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.791 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.791 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.792 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.792 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.793 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.793 DEBUG:boto:Data: 2023-10-17T18:54:20.793 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.794 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.794 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.795 DEBUG:boto:Params: {} 2023-10-17T18:54:20.795 DEBUG:boto:Token: None 2023-10-17T18:54:20.796 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v1nE5Fioi5ztqR8MTWI84hrA8Z0= 2023-10-17T18:54:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v1nE5Fioi5ztqR8MTWI84hrA8Z0=', 'Content-Length': '0'} 2023-10-17T18:54:20.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090aab0e0a9dc6253-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.799 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.800 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.800 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.801 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.801 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.801 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.802 DEBUG:boto:Data: 2023-10-17T18:54:20.802 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.803 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.803 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.804 DEBUG:boto:Params: {} 2023-10-17T18:54:20.804 DEBUG:boto:Token: None 2023-10-17T18:54:20.805 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v1nE5Fioi5ztqR8MTWI84hrA8Z0= 2023-10-17T18:54:20.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v1nE5Fioi5ztqR8MTWI84hrA8Z0=', 'Content-Length': '0'} 2023-10-17T18:54:20.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df71ae23c31245d-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-17T18:54:20.809 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.809 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.810 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.810 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.810 DEBUG:boto:Method: GET 2023-10-17T18:54:20.811 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.811 DEBUG:boto:Data: 2023-10-17T18:54:20.812 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.813 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.813 DEBUG:boto:Params: {} 2023-10-17T18:54:20.814 DEBUG:boto:Token: None 2023-10-17T18:54:20.814 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.815 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg= 2023-10-17T18:54:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg=', 'Content-Length': '0'} 2023-10-17T18:54:20.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003473bf94647f1786-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.858 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.859 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.859 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.860 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.860 DEBUG:boto:Method: GET 2023-10-17T18:54:20.861 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.861 DEBUG:boto:Data: 2023-10-17T18:54:20.862 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.862 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.863 DEBUG:boto:Params: {} 2023-10-17T18:54:20.863 DEBUG:boto:Token: None 2023-10-17T18:54:20.864 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036?versionId=null 2023-10-17T18:54:20.864 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg= 2023-10-17T18:54:20.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6cFov+jhf7hJddjUMAhYoaHWFg=', 'Content-Length': '0'} 2023-10-17T18:54:20.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2593c8f37eed5f-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.868 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.868 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.869 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.869 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.870 DEBUG:boto:Method: GET 2023-10-17T18:54:20.870 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.871 DEBUG:boto:Data: 2023-10-17T18:54:20.871 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.872 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.872 DEBUG:boto:Params: {} 2023-10-17T18:54:20.873 DEBUG:boto:Token: None 2023-10-17T18:54:20.873 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278a3afdaa2571a1-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.878 DEBUG:boto:b'hbmuog-58obj000000000000000000361000falseobj000000000000000000362023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.879 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.879 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.880 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.880 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.881 DEBUG:boto:Method: GET 2023-10-17T18:54:20.881 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.882 DEBUG:boto:Data: 2023-10-17T18:54:20.882 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.883 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.883 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.883 DEBUG:boto:Params: {} 2023-10-17T18:54:20.884 DEBUG:boto:Token: None 2023-10-17T18:54:20.885 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.885 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075375689b349625e-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.891 DEBUG:boto:b'hbmuog-58obj000000000000000000361000falseobj000000000000000000362023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.891 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.892 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.892 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.892 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.893 DEBUG:boto:Data: 2023-10-17T18:54:20.893 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.894 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.894 DEBUG:boto:Params: {} 2023-10-17T18:54:20.895 DEBUG:boto:Token: None 2023-10-17T18:54:20.895 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xw8A9gnV/MPUbzjE+9YOnninqYM= 2023-10-17T18:54:20.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xw8A9gnV/MPUbzjE+9YOnninqYM=', 'Content-Length': '0'} 2023-10-17T18:54:20.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af44deb1e7e0366c-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.899 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.899 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.900 DEBUG:boto:Method: HEAD 2023-10-17T18:54:20.900 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.901 DEBUG:boto:Data: 2023-10-17T18:54:20.901 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.902 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.902 DEBUG:boto:Params: {} 2023-10-17T18:54:20.903 DEBUG:boto:Token: None 2023-10-17T18:54:20.903 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xw8A9gnV/MPUbzjE+9YOnninqYM= 2023-10-17T18:54:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xw8A9gnV/MPUbzjE+9YOnninqYM=', 'Content-Length': '0'} 2023-10-17T18:54:20.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001139b40e01d1895d-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-17T18:54:20.907 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.908 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.908 DEBUG:boto:Method: GET 2023-10-17T18:54:20.909 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.909 DEBUG:boto:Data: 2023-10-17T18:54:20.909 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.910 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.911 DEBUG:boto:Params: {} 2023-10-17T18:54:20.911 DEBUG:boto:Token: None 2023-10-17T18:54:20.912 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fhoU0qOAoU9aqsUGgYSVw58y578= 2023-10-17T18:54:20.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fhoU0qOAoU9aqsUGgYSVw58y578=', 'Content-Length': '0'} 2023-10-17T18:54:20.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e8927a6e2f4d98-00652ed85c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.915 DEBUG:boto:path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.916 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.916 DEBUG:boto:Method: GET 2023-10-17T18:54:20.917 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.917 DEBUG:boto:Data: 2023-10-17T18:54:20.918 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.919 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.919 DEBUG:boto:Params: {} 2023-10-17T18:54:20.920 DEBUG:boto:Token: None 2023-10-17T18:54:20.920 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000036 2023-10-17T18:54:20.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fhoU0qOAoU9aqsUGgYSVw58y578= 2023-10-17T18:54:20.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fhoU0qOAoU9aqsUGgYSVw58y578=', 'Content-Length': '0'} 2023-10-17T18:54:20.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8cba07fde0153dd-00652ed85c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.966 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.967 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.968 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.968 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.969 DEBUG:boto:Method: GET 2023-10-17T18:54:20.969 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.970 DEBUG:boto:Data: 2023-10-17T18:54:20.970 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.971 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.972 DEBUG:boto:Params: {} 2023-10-17T18:54:20.972 DEBUG:boto:Token: None 2023-10-17T18:54:20.973 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000623e611e51bc19af-00652ed85c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.978 DEBUG:boto:b'hbmuog-58obj000000000000000000361000falseobj000000000000000000362023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.979 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:20.979 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:20.980 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.980 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.981 DEBUG:boto:Method: GET 2023-10-17T18:54:20.981 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000036 2023-10-17T18:54:20.982 DEBUG:boto:Data: 2023-10-17T18:54:20.982 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.983 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:20.983 DEBUG:boto:Port: 8000 2023-10-17T18:54:20.984 DEBUG:boto:Params: {} 2023-10-17T18:54:20.984 DEBUG:boto:Token: None 2023-10-17T18:54:20.985 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/ 2023-10-17T18:54:20.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538= 2023-10-17T18:54:20.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLeDf2+snupu+97p7MseVbW7538=', 'Content-Length': '0'} 2023-10-17T18:54:20.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fbf61999c456d53-00652ed85c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:20.991 DEBUG:boto:b'hbmuog-58obj000000000000000000361000falseobj000000000000000000362023-10-17T18:54:08.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:20.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-17T18:54:20.992 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:20.993 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:20.993 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:20.994 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:20.994 DEBUG:boto:Method: GET 2023-10-17T18:54:20.995 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:20.995 DEBUG:boto:Data: 2023-10-17T18:54:20.996 DEBUG:boto:Headers: {} 2023-10-17T18:54:20.996 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:20.997 DEBUG:boto:Port: 8001 2023-10-17T18:54:20.997 DEBUG:boto:Params: {} 2023-10-17T18:54:20.998 DEBUG:boto:Token: None 2023-10-17T18:54:20.998 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:20 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:20.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fMuPda8LfVUY7PUETINF8w0OLDw= 2023-10-17T18:54:20.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fMuPda8LfVUY7PUETINF8w0OLDw=', 'Content-Length': '0'} 2023-10-17T18:54:21.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea4f539052dde37-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.003 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.003 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.004 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.004 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.005 DEBUG:boto:Method: GET 2023-10-17T18:54:21.005 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.006 DEBUG:boto:Data: 2023-10-17T18:54:21.006 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.007 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.007 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.008 DEBUG:boto:Params: {} 2023-10-17T18:54:21.008 DEBUG:boto:Token: None 2023-10-17T18:54:21.009 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.010 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc= 2023-10-17T18:54:21.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc=', 'Content-Length': '0'} 2023-10-17T18:54:21.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000633d1b70f4b66fb3-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.014 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.014 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.015 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.015 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.016 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.016 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.017 DEBUG:boto:Data: 2023-10-17T18:54:21.017 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.018 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.019 DEBUG:boto:Params: {} 2023-10-17T18:54:21.019 DEBUG:boto:Token: None 2023-10-17T18:54:21.020 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:quRJaGjbh3vjH8kiYvpXIDEzrzg= 2023-10-17T18:54:21.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:quRJaGjbh3vjH8kiYvpXIDEzrzg=', 'Content-Length': '0'} 2023-10-17T18:54:21.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000953c0190d829d71-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.023 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.024 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.025 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.025 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.025 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.026 DEBUG:boto:Data: 2023-10-17T18:54:21.026 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.027 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.028 DEBUG:boto:Params: {} 2023-10-17T18:54:21.028 DEBUG:boto:Token: None 2023-10-17T18:54:21.029 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:quRJaGjbh3vjH8kiYvpXIDEzrzg= 2023-10-17T18:54:21.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:quRJaGjbh3vjH8kiYvpXIDEzrzg=', 'Content-Length': '0'} 2023-10-17T18:54:21.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc1a69ee63e3b3e-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-17T18:54:21.033 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.034 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.034 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.034 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.035 DEBUG:boto:Method: GET 2023-10-17T18:54:21.035 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.036 DEBUG:boto:Data: 2023-10-17T18:54:21.036 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.037 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.038 DEBUG:boto:Params: {} 2023-10-17T18:54:21.038 DEBUG:boto:Token: None 2023-10-17T18:54:21.039 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.039 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc= 2023-10-17T18:54:21.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc=', 'Content-Length': '0'} 2023-10-17T18:54:21.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a420b52830cb70-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.086 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.087 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.088 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.088 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.088 DEBUG:boto:Method: GET 2023-10-17T18:54:21.089 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.089 DEBUG:boto:Data: 2023-10-17T18:54:21.090 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.091 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.091 DEBUG:boto:Params: {} 2023-10-17T18:54:21.092 DEBUG:boto:Token: None 2023-10-17T18:54:21.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039?versionId=null 2023-10-17T18:54:21.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc= 2023-10-17T18:54:21.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:umOgdZRLTEnr9H/pT8NuTbA9Ijc=', 'Content-Length': '0'} 2023-10-17T18:54:21.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c92146f02af140a-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.096 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.097 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.097 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.098 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.098 DEBUG:boto:Method: GET 2023-10-17T18:54:21.099 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.099 DEBUG:boto:Data: 2023-10-17T18:54:21.100 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.100 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.101 DEBUG:boto:Params: {} 2023-10-17T18:54:21.101 DEBUG:boto:Token: None 2023-10-17T18:54:21.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb5c4245fe985b51-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.106 DEBUG:boto:b'hbmuog-58obj000000000000000000391000falseobj000000000000000000392023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.107 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.107 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.108 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.108 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.109 DEBUG:boto:Method: GET 2023-10-17T18:54:21.109 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.110 DEBUG:boto:Data: 2023-10-17T18:54:21.110 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.111 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.111 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.112 DEBUG:boto:Params: {} 2023-10-17T18:54:21.112 DEBUG:boto:Token: None 2023-10-17T18:54:21.113 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e66741e3740f13-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.119 DEBUG:boto:b'hbmuog-58obj000000000000000000391000falseobj000000000000000000392023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.120 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.120 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.121 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.121 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.122 DEBUG:boto:Data: 2023-10-17T18:54:21.122 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.123 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.123 DEBUG:boto:Params: {} 2023-10-17T18:54:21.124 DEBUG:boto:Token: None 2023-10-17T18:54:21.124 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bGFN/ajix182VGhbnvw3iR13aaI= 2023-10-17T18:54:21.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bGFN/ajix182VGhbnvw3iR13aaI=', 'Content-Length': '0'} 2023-10-17T18:54:21.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1eacd6315121ab-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.128 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.129 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.129 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.130 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.130 DEBUG:boto:Data: 2023-10-17T18:54:21.130 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.131 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.132 DEBUG:boto:Params: {} 2023-10-17T18:54:21.132 DEBUG:boto:Token: None 2023-10-17T18:54:21.132 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bGFN/ajix182VGhbnvw3iR13aaI= 2023-10-17T18:54:21.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bGFN/ajix182VGhbnvw3iR13aaI=', 'Content-Length': '0'} 2023-10-17T18:54:21.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e406ece5f53de884-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-17T18:54:21.137 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.137 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.138 DEBUG:boto:Method: GET 2023-10-17T18:54:21.138 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.139 DEBUG:boto:Data: 2023-10-17T18:54:21.139 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.140 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.140 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.140 DEBUG:boto:Params: {} 2023-10-17T18:54:21.141 DEBUG:boto:Token: None 2023-10-17T18:54:21.141 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.142 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mqGiY7dBnBkHKxwKF8SgvakFgbA= 2023-10-17T18:54:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mqGiY7dBnBkHKxwKF8SgvakFgbA=', 'Content-Length': '0'} 2023-10-17T18:54:21.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ac0a3796f6ba0a-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.145 DEBUG:boto:path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.146 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.146 DEBUG:boto:Method: GET 2023-10-17T18:54:21.147 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.147 DEBUG:boto:Data: 2023-10-17T18:54:21.148 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.148 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.149 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.149 DEBUG:boto:Params: {} 2023-10-17T18:54:21.149 DEBUG:boto:Token: None 2023-10-17T18:54:21.150 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000039 2023-10-17T18:54:21.151 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mqGiY7dBnBkHKxwKF8SgvakFgbA= 2023-10-17T18:54:21.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mqGiY7dBnBkHKxwKF8SgvakFgbA=', 'Content-Length': '0'} 2023-10-17T18:54:21.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0f34cfee6c760d-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.194 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.195 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.196 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.196 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.197 DEBUG:boto:Method: GET 2023-10-17T18:54:21.197 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.197 DEBUG:boto:Data: 2023-10-17T18:54:21.198 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.198 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.199 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.199 DEBUG:boto:Params: {} 2023-10-17T18:54:21.200 DEBUG:boto:Token: None 2023-10-17T18:54:21.200 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a99a1480df50ef-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.205 DEBUG:boto:b'hbmuog-58obj000000000000000000391000falseobj000000000000000000392023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.206 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.206 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.207 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.207 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.208 DEBUG:boto:Method: GET 2023-10-17T18:54:21.208 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000039 2023-10-17T18:54:21.208 DEBUG:boto:Data: 2023-10-17T18:54:21.209 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.210 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.210 DEBUG:boto:Params: {} 2023-10-17T18:54:21.211 DEBUG:boto:Token: None 2023-10-17T18:54:21.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002385b5943091e083-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.217 DEBUG:boto:b'hbmuog-58obj000000000000000000391000falseobj000000000000000000392023-10-17T18:54:07.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-17T18:54:21.218 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.219 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.219 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.220 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.220 DEBUG:boto:Method: GET 2023-10-17T18:54:21.221 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.221 DEBUG:boto:Data: 2023-10-17T18:54:21.222 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.223 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.223 DEBUG:boto:Params: {} 2023-10-17T18:54:21.223 DEBUG:boto:Token: None 2023-10-17T18:54:21.224 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.225 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU= 2023-10-17T18:54:21.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU=', 'Content-Length': '0'} 2023-10-17T18:54:21.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102f7374f5cc9112-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.228 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.229 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.229 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.229 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.230 DEBUG:boto:Method: GET 2023-10-17T18:54:21.230 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.231 DEBUG:boto:Data: 2023-10-17T18:54:21.231 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.232 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.232 DEBUG:boto:Params: {} 2023-10-17T18:54:21.233 DEBUG:boto:Token: None 2023-10-17T18:54:21.233 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU= 2023-10-17T18:54:21.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU=', 'Content-Length': '0'} 2023-10-17T18:54:21.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000278d0ea09d465d3-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.237 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.238 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.238 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.239 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.239 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.239 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.240 DEBUG:boto:Data: 2023-10-17T18:54:21.240 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.241 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.241 DEBUG:boto:Params: {} 2023-10-17T18:54:21.242 DEBUG:boto:Token: None 2023-10-17T18:54:21.242 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9JALM9PN6ksBVnWeCBYtLndtMYU= 2023-10-17T18:54:21.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9JALM9PN6ksBVnWeCBYtLndtMYU=', 'Content-Length': '0'} 2023-10-17T18:54:21.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001568ea0be0182314-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.246 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.246 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.247 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.247 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.247 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.248 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.248 DEBUG:boto:Data: 2023-10-17T18:54:21.249 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.249 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.250 DEBUG:boto:Params: {} 2023-10-17T18:54:21.250 DEBUG:boto:Token: None 2023-10-17T18:54:21.251 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.251 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9JALM9PN6ksBVnWeCBYtLndtMYU= 2023-10-17T18:54:21.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9JALM9PN6ksBVnWeCBYtLndtMYU=', 'Content-Length': '0'} 2023-10-17T18:54:21.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007984913376a11846-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-17T18:54:21.255 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.255 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.256 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.256 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.257 DEBUG:boto:Method: GET 2023-10-17T18:54:21.257 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.257 DEBUG:boto:Data: 2023-10-17T18:54:21.258 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.259 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.259 DEBUG:boto:Params: {} 2023-10-17T18:54:21.260 DEBUG:boto:Token: None 2023-10-17T18:54:21.260 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU= 2023-10-17T18:54:21.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU=', 'Content-Length': '0'} 2023-10-17T18:54:21.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b75cc21bf827395d-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.310 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.311 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.312 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.312 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.312 DEBUG:boto:Method: GET 2023-10-17T18:54:21.313 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.313 DEBUG:boto:Data: 2023-10-17T18:54:21.314 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.314 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.315 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.315 DEBUG:boto:Params: {} 2023-10-17T18:54:21.316 DEBUG:boto:Token: None 2023-10-17T18:54:21.316 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040?versionId=null 2023-10-17T18:54:21.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU= 2023-10-17T18:54:21.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1j9kZXx5inm1K7BRNJScIdzvOiU=', 'Content-Length': '0'} 2023-10-17T18:54:21.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5117b56292d5f5d-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.320 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.320 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.321 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.321 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.322 DEBUG:boto:Method: GET 2023-10-17T18:54:21.322 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.323 DEBUG:boto:Data: 2023-10-17T18:54:21.323 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.323 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.324 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.324 DEBUG:boto:Params: {} 2023-10-17T18:54:21.325 DEBUG:boto:Token: None 2023-10-17T18:54:21.325 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.326 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d6c5abcb3f959a-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.330 DEBUG:boto:b'hbmuog-58obj000000000000000000401000falseobj000000000000000000402023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.330 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.331 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.331 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.332 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.332 DEBUG:boto:Method: GET 2023-10-17T18:54:21.333 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.333 DEBUG:boto:Data: 2023-10-17T18:54:21.334 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.334 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.335 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.335 DEBUG:boto:Params: {} 2023-10-17T18:54:21.335 DEBUG:boto:Token: None 2023-10-17T18:54:21.336 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.336 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033518c3d10567891-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.382 DEBUG:boto:b'hbmuog-58obj000000000000000000401000falseobj000000000000000000402023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.383 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.383 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.384 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.384 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.385 DEBUG:boto:Data: 2023-10-17T18:54:21.385 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.386 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.387 DEBUG:boto:Params: {} 2023-10-17T18:54:21.387 DEBUG:boto:Token: None 2023-10-17T18:54:21.387 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0uDNaBFgS+uScG24OsN6WpDuKo= 2023-10-17T18:54:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0uDNaBFgS+uScG24OsN6WpDuKo=', 'Content-Length': '0'} 2023-10-17T18:54:21.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1097c150754670-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.391 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.392 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.392 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.393 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.393 DEBUG:boto:Data: 2023-10-17T18:54:21.393 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.394 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.395 DEBUG:boto:Params: {} 2023-10-17T18:54:21.395 DEBUG:boto:Token: None 2023-10-17T18:54:21.396 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:r0uDNaBFgS+uScG24OsN6WpDuKo= 2023-10-17T18:54:21.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:r0uDNaBFgS+uScG24OsN6WpDuKo=', 'Content-Length': '0'} 2023-10-17T18:54:21.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd84497779d93938-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-17T18:54:21.400 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.400 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.401 DEBUG:boto:Method: GET 2023-10-17T18:54:21.401 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.401 DEBUG:boto:Data: 2023-10-17T18:54:21.402 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.403 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.403 DEBUG:boto:Params: {} 2023-10-17T18:54:21.404 DEBUG:boto:Token: None 2023-10-17T18:54:21.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mxIU5li3AmwNtqjqlFR40UPY2sA= 2023-10-17T18:54:21.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mxIU5li3AmwNtqjqlFR40UPY2sA=', 'Content-Length': '0'} 2023-10-17T18:54:21.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001155e7fdf88c5cda-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.408 DEBUG:boto:path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.408 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.409 DEBUG:boto:Method: GET 2023-10-17T18:54:21.409 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.410 DEBUG:boto:Data: 2023-10-17T18:54:21.410 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.411 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.411 DEBUG:boto:Params: {} 2023-10-17T18:54:21.412 DEBUG:boto:Token: None 2023-10-17T18:54:21.412 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000040 2023-10-17T18:54:21.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mxIU5li3AmwNtqjqlFR40UPY2sA= 2023-10-17T18:54:21.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mxIU5li3AmwNtqjqlFR40UPY2sA=', 'Content-Length': '0'} 2023-10-17T18:54:21.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df50197f8a37548d-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.415 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.416 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.416 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.416 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.417 DEBUG:boto:Method: GET 2023-10-17T18:54:21.417 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.418 DEBUG:boto:Data: 2023-10-17T18:54:21.418 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.419 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.419 DEBUG:boto:Params: {} 2023-10-17T18:54:21.420 DEBUG:boto:Token: None 2023-10-17T18:54:21.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d608c39fdaa6d848-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.424 DEBUG:boto:b'hbmuog-58obj000000000000000000401000falseobj000000000000000000402023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.424 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.425 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.425 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.426 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.426 DEBUG:boto:Method: GET 2023-10-17T18:54:21.427 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000040 2023-10-17T18:54:21.427 DEBUG:boto:Data: 2023-10-17T18:54:21.427 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.428 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.429 DEBUG:boto:Params: {} 2023-10-17T18:54:21.429 DEBUG:boto:Token: None 2023-10-17T18:54:21.429 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37fe913d74267af-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.478 DEBUG:boto:b'hbmuog-58obj000000000000000000401000falseobj000000000000000000402023-10-17T18:54:08.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-17T18:54:21.480 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.480 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.480 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.481 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.481 DEBUG:boto:Method: GET 2023-10-17T18:54:21.482 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.482 DEBUG:boto:Data: 2023-10-17T18:54:21.483 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.483 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.483 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.484 DEBUG:boto:Params: {} 2023-10-17T18:54:21.484 DEBUG:boto:Token: None 2023-10-17T18:54:21.485 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.485 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c= 2023-10-17T18:54:21.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c=', 'Content-Length': '0'} 2023-10-17T18:54:21.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f37fa258b2fd099-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.489 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.489 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.490 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.490 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.491 DEBUG:boto:Method: GET 2023-10-17T18:54:21.491 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.491 DEBUG:boto:Data: 2023-10-17T18:54:21.492 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.493 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.493 DEBUG:boto:Params: {} 2023-10-17T18:54:21.493 DEBUG:boto:Token: None 2023-10-17T18:54:21.494 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c= 2023-10-17T18:54:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c=', 'Content-Length': '0'} 2023-10-17T18:54:21.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc13fabb17173acf-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.498 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.498 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.499 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.499 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.499 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.502 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.503 DEBUG:boto:Data: 2023-10-17T18:54:21.503 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.503 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.504 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.504 DEBUG:boto:Params: {} 2023-10-17T18:54:21.505 DEBUG:boto:Token: None 2023-10-17T18:54:21.505 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.506 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KtVoleHCJVOdJdctwCsmBgBLBEk= 2023-10-17T18:54:21.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KtVoleHCJVOdJdctwCsmBgBLBEk=', 'Content-Length': '0'} 2023-10-17T18:54:21.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba96bac3112691f2-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.509 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.509 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.509 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.510 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.510 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.511 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.511 DEBUG:boto:Data: 2023-10-17T18:54:21.511 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.512 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.512 DEBUG:boto:Params: {} 2023-10-17T18:54:21.513 DEBUG:boto:Token: None 2023-10-17T18:54:21.513 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KtVoleHCJVOdJdctwCsmBgBLBEk= 2023-10-17T18:54:21.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KtVoleHCJVOdJdctwCsmBgBLBEk=', 'Content-Length': '0'} 2023-10-17T18:54:21.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad97f6d0f170d7a-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-17T18:54:21.517 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.518 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.518 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.518 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.519 DEBUG:boto:Method: GET 2023-10-17T18:54:21.519 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.520 DEBUG:boto:Data: 2023-10-17T18:54:21.520 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.521 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.521 DEBUG:boto:Params: {} 2023-10-17T18:54:21.522 DEBUG:boto:Token: None 2023-10-17T18:54:21.522 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c= 2023-10-17T18:54:21.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c=', 'Content-Length': '0'} 2023-10-17T18:54:21.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006277bdb8bcb3ee13-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.526 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.527 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.527 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.527 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.528 DEBUG:boto:Method: GET 2023-10-17T18:54:21.528 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.529 DEBUG:boto:Data: 2023-10-17T18:54:21.529 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.530 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.530 DEBUG:boto:Params: {} 2023-10-17T18:54:21.531 DEBUG:boto:Token: None 2023-10-17T18:54:21.531 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042?versionId=null 2023-10-17T18:54:21.532 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c= 2023-10-17T18:54:21.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GQ5H0M9ZPNti1BGSIhULgx6HP3c=', 'Content-Length': '0'} 2023-10-17T18:54:21.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68062b2f3a0e921-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.535 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.535 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.536 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.536 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.537 DEBUG:boto:Method: GET 2023-10-17T18:54:21.537 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.537 DEBUG:boto:Data: 2023-10-17T18:54:21.538 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.539 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.539 DEBUG:boto:Params: {} 2023-10-17T18:54:21.540 DEBUG:boto:Token: None 2023-10-17T18:54:21.540 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b847b1d774bd7901-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.544 DEBUG:boto:b'hbmuog-58obj000000000000000000421000falseobj000000000000000000422023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.544 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.545 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.545 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.546 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.546 DEBUG:boto:Method: GET 2023-10-17T18:54:21.547 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.547 DEBUG:boto:Data: 2023-10-17T18:54:21.547 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.548 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.548 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.549 DEBUG:boto:Params: {} 2023-10-17T18:54:21.549 DEBUG:boto:Token: None 2023-10-17T18:54:21.550 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.550 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1c5c0ff74eef9d5-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.555 DEBUG:boto:b'hbmuog-58obj000000000000000000421000falseobj000000000000000000422023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.556 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.556 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.557 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.557 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.557 DEBUG:boto:Data: 2023-10-17T18:54:21.558 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.559 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.559 DEBUG:boto:Params: {} 2023-10-17T18:54:21.560 DEBUG:boto:Token: None 2023-10-17T18:54:21.560 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:We2td/49SFXiCYVM2sgeRSmV5pw= 2023-10-17T18:54:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:We2td/49SFXiCYVM2sgeRSmV5pw=', 'Content-Length': '0'} 2023-10-17T18:54:21.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b81e4c2e39c09afd-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.564 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.564 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.565 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.565 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.566 DEBUG:boto:Data: 2023-10-17T18:54:21.566 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.566 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.567 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.567 DEBUG:boto:Params: {} 2023-10-17T18:54:21.568 DEBUG:boto:Token: None 2023-10-17T18:54:21.568 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:We2td/49SFXiCYVM2sgeRSmV5pw= 2023-10-17T18:54:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:We2td/49SFXiCYVM2sgeRSmV5pw=', 'Content-Length': '0'} 2023-10-17T18:54:21.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb29bcbfd5aa92da-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-17T18:54:21.572 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.572 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.573 DEBUG:boto:Method: GET 2023-10-17T18:54:21.573 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.574 DEBUG:boto:Data: 2023-10-17T18:54:21.574 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.575 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.575 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.575 DEBUG:boto:Params: {} 2023-10-17T18:54:21.576 DEBUG:boto:Token: None 2023-10-17T18:54:21.576 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.577 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:83ubVMq/JsXffXPYW54/O8pePEk= 2023-10-17T18:54:21.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:83ubVMq/JsXffXPYW54/O8pePEk=', 'Content-Length': '0'} 2023-10-17T18:54:21.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa82b5242cb2e1f-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.622 DEBUG:boto:path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.623 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.623 DEBUG:boto:Method: GET 2023-10-17T18:54:21.624 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.624 DEBUG:boto:Data: 2023-10-17T18:54:21.625 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.625 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.626 DEBUG:boto:Params: {} 2023-10-17T18:54:21.626 DEBUG:boto:Token: None 2023-10-17T18:54:21.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000042 2023-10-17T18:54:21.627 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:83ubVMq/JsXffXPYW54/O8pePEk= 2023-10-17T18:54:21.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:83ubVMq/JsXffXPYW54/O8pePEk=', 'Content-Length': '0'} 2023-10-17T18:54:21.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7002c809d09869-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.630 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.631 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.631 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.632 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.632 DEBUG:boto:Method: GET 2023-10-17T18:54:21.632 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.633 DEBUG:boto:Data: 2023-10-17T18:54:21.633 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.634 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.635 DEBUG:boto:Params: {} 2023-10-17T18:54:21.635 DEBUG:boto:Token: None 2023-10-17T18:54:21.635 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.636 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e53fc659e96726-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.639 DEBUG:boto:b'hbmuog-58obj000000000000000000421000falseobj000000000000000000422023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.640 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.641 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.641 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.641 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.642 DEBUG:boto:Method: GET 2023-10-17T18:54:21.642 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000042 2023-10-17T18:54:21.643 DEBUG:boto:Data: 2023-10-17T18:54:21.643 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.644 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.644 DEBUG:boto:Params: {} 2023-10-17T18:54:21.645 DEBUG:boto:Token: None 2023-10-17T18:54:21.645 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.646 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5c34264b5985a4-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.651 DEBUG:boto:b'hbmuog-58obj000000000000000000421000falseobj000000000000000000422023-10-17T18:54:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-17T18:54:21.652 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.652 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.653 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.653 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.654 DEBUG:boto:Method: GET 2023-10-17T18:54:21.654 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.655 DEBUG:boto:Data: 2023-10-17T18:54:21.655 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.655 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.656 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.656 DEBUG:boto:Params: {} 2023-10-17T18:54:21.657 DEBUG:boto:Token: None 2023-10-17T18:54:21.657 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4= 2023-10-17T18:54:21.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4=', 'Content-Length': '0'} 2023-10-17T18:54:21.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401770ef85927c11-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.661 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.661 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.662 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.662 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.663 DEBUG:boto:Method: GET 2023-10-17T18:54:21.663 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.663 DEBUG:boto:Data: 2023-10-17T18:54:21.664 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.664 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.665 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.665 DEBUG:boto:Params: {} 2023-10-17T18:54:21.665 DEBUG:boto:Token: None 2023-10-17T18:54:21.666 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.666 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4= 2023-10-17T18:54:21.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4=', 'Content-Length': '0'} 2023-10-17T18:54:21.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba721ef64aeb1ed-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.669 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.670 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.670 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.671 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.671 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.671 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.672 DEBUG:boto:Data: 2023-10-17T18:54:21.672 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.673 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.673 DEBUG:boto:Params: {} 2023-10-17T18:54:21.674 DEBUG:boto:Token: None 2023-10-17T18:54:21.674 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vkgm2mFJ8f5fvNSTVf3hLG0bXSo= 2023-10-17T18:54:21.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vkgm2mFJ8f5fvNSTVf3hLG0bXSo=', 'Content-Length': '0'} 2023-10-17T18:54:21.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006274c6a6376a0b37-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.678 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.678 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.678 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.679 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.679 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.680 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.680 DEBUG:boto:Data: 2023-10-17T18:54:21.680 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.681 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.681 DEBUG:boto:Params: {} 2023-10-17T18:54:21.682 DEBUG:boto:Token: None 2023-10-17T18:54:21.682 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.683 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vkgm2mFJ8f5fvNSTVf3hLG0bXSo= 2023-10-17T18:54:21.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vkgm2mFJ8f5fvNSTVf3hLG0bXSo=', 'Content-Length': '0'} 2023-10-17T18:54:21.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003924e227e4649579-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-17T18:54:21.686 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.687 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.687 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.688 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.688 DEBUG:boto:Method: GET 2023-10-17T18:54:21.689 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.689 DEBUG:boto:Data: 2023-10-17T18:54:21.689 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.690 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.691 DEBUG:boto:Params: {} 2023-10-17T18:54:21.691 DEBUG:boto:Token: None 2023-10-17T18:54:21.691 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4= 2023-10-17T18:54:21.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4=', 'Content-Length': '0'} 2023-10-17T18:54:21.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0748c1e05909ac-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.695 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.696 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.696 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.696 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.697 DEBUG:boto:Method: GET 2023-10-17T18:54:21.697 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.698 DEBUG:boto:Data: 2023-10-17T18:54:21.698 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.699 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.699 DEBUG:boto:Params: {} 2023-10-17T18:54:21.700 DEBUG:boto:Token: None 2023-10-17T18:54:21.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045?versionId=null 2023-10-17T18:54:21.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4= 2023-10-17T18:54:21.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTSZ3LJcJayy2TXtK6ASPnrJml4=', 'Content-Length': '0'} 2023-10-17T18:54:21.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144410533007c913-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.704 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.704 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.705 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.705 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.706 DEBUG:boto:Method: GET 2023-10-17T18:54:21.706 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.707 DEBUG:boto:Data: 2023-10-17T18:54:21.707 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.708 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.708 DEBUG:boto:Params: {} 2023-10-17T18:54:21.709 DEBUG:boto:Token: None 2023-10-17T18:54:21.709 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec7403e3378a0d4a-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.754 DEBUG:boto:b'hbmuog-58obj000000000000000000451000falseobj000000000000000000452023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.755 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.756 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.756 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.757 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.757 DEBUG:boto:Method: GET 2023-10-17T18:54:21.757 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.758 DEBUG:boto:Data: 2023-10-17T18:54:21.758 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.759 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.759 DEBUG:boto:Params: {} 2023-10-17T18:54:21.760 DEBUG:boto:Token: None 2023-10-17T18:54:21.760 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f4c97f53e19325-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.767 DEBUG:boto:b'hbmuog-58obj000000000000000000451000falseobj000000000000000000452023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.767 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.768 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.768 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.768 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.769 DEBUG:boto:Data: 2023-10-17T18:54:21.769 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.770 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.770 DEBUG:boto:Params: {} 2023-10-17T18:54:21.771 DEBUG:boto:Token: None 2023-10-17T18:54:21.771 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5xLkA0VWAhWLlfMP7lcQcSxja1A= 2023-10-17T18:54:21.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5xLkA0VWAhWLlfMP7lcQcSxja1A=', 'Content-Length': '0'} 2023-10-17T18:54:21.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923c44711c73fd34-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.775 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.775 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.776 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.776 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.776 DEBUG:boto:Data: 2023-10-17T18:54:21.777 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.777 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.777 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.778 DEBUG:boto:Params: {} 2023-10-17T18:54:21.778 DEBUG:boto:Token: None 2023-10-17T18:54:21.779 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.779 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5xLkA0VWAhWLlfMP7lcQcSxja1A= 2023-10-17T18:54:21.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5xLkA0VWAhWLlfMP7lcQcSxja1A=', 'Content-Length': '0'} 2023-10-17T18:54:21.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c73d0d86d595193b-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-17T18:54:21.782 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.783 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.783 DEBUG:boto:Method: GET 2023-10-17T18:54:21.784 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.784 DEBUG:boto:Data: 2023-10-17T18:54:21.785 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.785 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.786 DEBUG:boto:Params: {} 2023-10-17T18:54:21.786 DEBUG:boto:Token: None 2023-10-17T18:54:21.787 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c4IfcocmJdFOWT1yFFXMZ6rzebA= 2023-10-17T18:54:21.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c4IfcocmJdFOWT1yFFXMZ6rzebA=', 'Content-Length': '0'} 2023-10-17T18:54:21.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2b12d7591b1bbb-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.790 DEBUG:boto:path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.793 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.794 DEBUG:boto:Method: GET 2023-10-17T18:54:21.794 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.795 DEBUG:boto:Data: 2023-10-17T18:54:21.795 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.795 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.796 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.796 DEBUG:boto:Params: {} 2023-10-17T18:54:21.797 DEBUG:boto:Token: None 2023-10-17T18:54:21.797 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000045 2023-10-17T18:54:21.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c4IfcocmJdFOWT1yFFXMZ6rzebA= 2023-10-17T18:54:21.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c4IfcocmJdFOWT1yFFXMZ6rzebA=', 'Content-Length': '0'} 2023-10-17T18:54:21.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088ce413d9703ac25-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.801 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.801 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.802 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.802 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.803 DEBUG:boto:Method: GET 2023-10-17T18:54:21.803 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.804 DEBUG:boto:Data: 2023-10-17T18:54:21.804 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.804 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.805 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.805 DEBUG:boto:Params: {} 2023-10-17T18:54:21.806 DEBUG:boto:Token: None 2023-10-17T18:54:21.806 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.807 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e626e92a70b9d7f-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.810 DEBUG:boto:b'hbmuog-58obj000000000000000000451000falseobj000000000000000000452023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.811 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.811 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.812 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.812 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.813 DEBUG:boto:Method: GET 2023-10-17T18:54:21.813 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000045 2023-10-17T18:54:21.813 DEBUG:boto:Data: 2023-10-17T18:54:21.814 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.814 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.815 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.815 DEBUG:boto:Params: {} 2023-10-17T18:54:21.816 DEBUG:boto:Token: None 2023-10-17T18:54:21.816 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.817 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3961636f6c3fa6b-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.822 DEBUG:boto:b'hbmuog-58obj000000000000000000451000falseobj000000000000000000452023-10-17T18:54:07.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-17T18:54:21.823 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.823 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.824 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.824 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.825 DEBUG:boto:Method: GET 2023-10-17T18:54:21.825 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.825 DEBUG:boto:Data: 2023-10-17T18:54:21.826 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.827 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.827 DEBUG:boto:Params: {} 2023-10-17T18:54:21.828 DEBUG:boto:Token: None 2023-10-17T18:54:21.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os= 2023-10-17T18:54:21.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os=', 'Content-Length': '0'} 2023-10-17T18:54:21.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b714bcb488b2440-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.831 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.832 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.832 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.833 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.833 DEBUG:boto:Method: GET 2023-10-17T18:54:21.833 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.834 DEBUG:boto:Data: 2023-10-17T18:54:21.834 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.835 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.835 DEBUG:boto:Params: {} 2023-10-17T18:54:21.836 DEBUG:boto:Token: None 2023-10-17T18:54:21.836 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os= 2023-10-17T18:54:21.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os=', 'Content-Length': '0'} 2023-10-17T18:54:21.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030ffc6d58caf6fc5-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.882 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.883 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.883 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.884 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.884 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.885 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.885 DEBUG:boto:Data: 2023-10-17T18:54:21.885 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.886 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.887 DEBUG:boto:Params: {} 2023-10-17T18:54:21.887 DEBUG:boto:Token: None 2023-10-17T18:54:21.887 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BBas4eNorrwT9bXVT+e4joqhS/k= 2023-10-17T18:54:21.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBas4eNorrwT9bXVT+e4joqhS/k=', 'Content-Length': '0'} 2023-10-17T18:54:21.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7e79847ec1c3f4-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.891 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.892 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.892 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.892 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.893 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.893 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.893 DEBUG:boto:Data: 2023-10-17T18:54:21.894 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.895 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.895 DEBUG:boto:Params: {} 2023-10-17T18:54:21.895 DEBUG:boto:Token: None 2023-10-17T18:54:21.896 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BBas4eNorrwT9bXVT+e4joqhS/k= 2023-10-17T18:54:21.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBas4eNorrwT9bXVT+e4joqhS/k=', 'Content-Length': '0'} 2023-10-17T18:54:21.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c55c8a416803d3-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-17T18:54:21.900 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.900 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.901 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.901 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.902 DEBUG:boto:Method: GET 2023-10-17T18:54:21.902 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.902 DEBUG:boto:Data: 2023-10-17T18:54:21.903 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.903 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.904 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.904 DEBUG:boto:Params: {} 2023-10-17T18:54:21.904 DEBUG:boto:Token: None 2023-10-17T18:54:21.905 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.905 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os= 2023-10-17T18:54:21.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os=', 'Content-Length': '0'} 2023-10-17T18:54:21.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f1811b5ae37933-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.950 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.951 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.951 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.952 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.952 DEBUG:boto:Method: GET 2023-10-17T18:54:21.953 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.953 DEBUG:boto:Data: 2023-10-17T18:54:21.954 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.954 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.955 DEBUG:boto:Params: {} 2023-10-17T18:54:21.955 DEBUG:boto:Token: None 2023-10-17T18:54:21.956 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046?versionId=null 2023-10-17T18:54:21.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os= 2023-10-17T18:54:21.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:M/b9zP/pcNA0orzIuYTw+0vl3os=', 'Content-Length': '0'} 2023-10-17T18:54:21.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b1c5ca8b9b87df-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.960 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.960 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.961 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.961 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.961 DEBUG:boto:Method: GET 2023-10-17T18:54:21.962 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.962 DEBUG:boto:Data: 2023-10-17T18:54:21.963 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.963 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.963 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.964 DEBUG:boto:Params: {} 2023-10-17T18:54:21.964 DEBUG:boto:Token: None 2023-10-17T18:54:21.965 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.965 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6a5fc6dbc2b10ee-00652ed85d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.969 DEBUG:boto:b'hbmuog-58obj000000000000000000461000falseobj000000000000000000462023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.970 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:21.970 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:21.971 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.971 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.971 DEBUG:boto:Method: GET 2023-10-17T18:54:21.972 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:21.972 DEBUG:boto:Data: 2023-10-17T18:54:21.973 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.973 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.973 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.974 DEBUG:boto:Params: {} 2023-10-17T18:54:21.974 DEBUG:boto:Token: None 2023-10-17T18:54:21.975 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/ 2023-10-17T18:54:21.975 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw= 2023-10-17T18:54:21.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wBPREIiF3Wz3mHFz1gd/jB5lyrw=', 'Content-Length': '0'} 2023-10-17T18:54:21.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cf2d75482a11b6-00652ed85d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.981 DEBUG:boto:b'hbmuog-58obj000000000000000000461000falseobj000000000000000000462023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:21.981 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.982 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.982 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.983 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.983 DEBUG:boto:Data: 2023-10-17T18:54:21.984 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.984 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:21.984 DEBUG:boto:Port: 8001 2023-10-17T18:54:21.985 DEBUG:boto:Params: {} 2023-10-17T18:54:21.985 DEBUG:boto:Token: None 2023-10-17T18:54:21.986 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.986 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K8DaHr9fLfQ+3PPdgrEiu0jVnQ8= 2023-10-17T18:54:21.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K8DaHr9fLfQ+3PPdgrEiu0jVnQ8=', 'Content-Length': '0'} 2023-10-17T18:54:21.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5093db14f4114d1-00652ed85d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.990 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.990 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.991 DEBUG:boto:Method: HEAD 2023-10-17T18:54:21.991 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.991 DEBUG:boto:Data: 2023-10-17T18:54:21.992 DEBUG:boto:Headers: {} 2023-10-17T18:54:21.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:21.993 DEBUG:boto:Port: 8000 2023-10-17T18:54:21.993 DEBUG:boto:Params: {} 2023-10-17T18:54:21.993 DEBUG:boto:Token: None 2023-10-17T18:54:21.994 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:21 GMT /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K8DaHr9fLfQ+3PPdgrEiu0jVnQ8= 2023-10-17T18:54:21.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K8DaHr9fLfQ+3PPdgrEiu0jVnQ8=', 'Content-Length': '0'} 2023-10-17T18:54:21.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2d3b07804b57f9-00652ed85d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:21.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-17T18:54:21.999 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:21.999 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.000 DEBUG:boto:Method: GET 2023-10-17T18:54:22.000 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.001 DEBUG:boto:Data: 2023-10-17T18:54:22.001 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.001 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.002 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.002 DEBUG:boto:Params: {} 2023-10-17T18:54:22.003 DEBUG:boto:Token: None 2023-10-17T18:54:22.003 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FKR7/YZLYw29N4cg7i9MTfODZnw= 2023-10-17T18:54:22.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FKR7/YZLYw29N4cg7i9MTfODZnw=', 'Content-Length': '0'} 2023-10-17T18:54:22.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040396af2f2544a11-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.007 DEBUG:boto:path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.007 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.008 DEBUG:boto:Method: GET 2023-10-17T18:54:22.008 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.008 DEBUG:boto:Data: 2023-10-17T18:54:22.009 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.010 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.010 DEBUG:boto:Params: {} 2023-10-17T18:54:22.011 DEBUG:boto:Token: None 2023-10-17T18:54:22.011 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000046 2023-10-17T18:54:22.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FKR7/YZLYw29N4cg7i9MTfODZnw= 2023-10-17T18:54:22.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FKR7/YZLYw29N4cg7i9MTfODZnw=', 'Content-Length': '0'} 2023-10-17T18:54:22.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4817927ce6ff870-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.058 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.059 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.059 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.060 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.060 DEBUG:boto:Method: GET 2023-10-17T18:54:22.061 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.061 DEBUG:boto:Data: 2023-10-17T18:54:22.061 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.062 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.062 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.063 DEBUG:boto:Params: {} 2023-10-17T18:54:22.063 DEBUG:boto:Token: None 2023-10-17T18:54:22.064 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.064 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003595a53790cde2cd-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.067 DEBUG:boto:b'hbmuog-58obj000000000000000000461000falseobj000000000000000000462023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.068 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.069 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.069 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.069 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.070 DEBUG:boto:Method: GET 2023-10-17T18:54:22.070 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000046 2023-10-17T18:54:22.071 DEBUG:boto:Data: 2023-10-17T18:54:22.071 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.071 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.072 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.072 DEBUG:boto:Params: {} 2023-10-17T18:54:22.072 DEBUG:boto:Token: None 2023-10-17T18:54:22.073 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.073 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1dafa345757a23d-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.079 DEBUG:boto:b'hbmuog-58obj000000000000000000461000falseobj000000000000000000462023-10-17T18:54:03.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-17T18:54:22.080 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.080 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.080 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.081 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.081 DEBUG:boto:Method: GET 2023-10-17T18:54:22.082 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.082 DEBUG:boto:Data: 2023-10-17T18:54:22.082 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.083 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.083 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.084 DEBUG:boto:Params: {} 2023-10-17T18:54:22.084 DEBUG:boto:Token: None 2023-10-17T18:54:22.085 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM= 2023-10-17T18:54:22.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM=', 'Content-Length': '0'} 2023-10-17T18:54:22.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002462ef2f24fed934-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.088 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.088 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.089 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.089 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.090 DEBUG:boto:Method: GET 2023-10-17T18:54:22.090 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.091 DEBUG:boto:Data: 2023-10-17T18:54:22.091 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.092 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.092 DEBUG:boto:Params: {} 2023-10-17T18:54:22.093 DEBUG:boto:Token: None 2023-10-17T18:54:22.093 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.093 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM= 2023-10-17T18:54:22.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM=', 'Content-Length': '0'} 2023-10-17T18:54:22.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e011e7ac6ebdb3b0-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.096 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.097 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.097 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.097 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.098 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.098 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.098 DEBUG:boto:Data: 2023-10-17T18:54:22.099 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.099 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.100 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.100 DEBUG:boto:Params: {} 2023-10-17T18:54:22.100 DEBUG:boto:Token: None 2023-10-17T18:54:22.101 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.101 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+uqrEAzu76ylKjmYFZUEoJRX6/4= 2023-10-17T18:54:22.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+uqrEAzu76ylKjmYFZUEoJRX6/4=', 'Content-Length': '0'} 2023-10-17T18:54:22.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030976707057d1229-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.104 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.104 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.104 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.105 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.105 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.106 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.106 DEBUG:boto:Data: 2023-10-17T18:54:22.106 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.107 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.107 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.107 DEBUG:boto:Params: {} 2023-10-17T18:54:22.108 DEBUG:boto:Token: None 2023-10-17T18:54:22.108 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+uqrEAzu76ylKjmYFZUEoJRX6/4= 2023-10-17T18:54:22.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+uqrEAzu76ylKjmYFZUEoJRX6/4=', 'Content-Length': '0'} 2023-10-17T18:54:22.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004caab4e103e8c146-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-17T18:54:22.112 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.112 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.113 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.113 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.114 DEBUG:boto:Method: GET 2023-10-17T18:54:22.114 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.114 DEBUG:boto:Data: 2023-10-17T18:54:22.115 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.115 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.116 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.116 DEBUG:boto:Params: {} 2023-10-17T18:54:22.116 DEBUG:boto:Token: None 2023-10-17T18:54:22.117 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.117 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM= 2023-10-17T18:54:22.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM=', 'Content-Length': '0'} 2023-10-17T18:54:22.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d0648333adc477-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.120 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.120 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.121 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.121 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.122 DEBUG:boto:Method: GET 2023-10-17T18:54:22.122 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.122 DEBUG:boto:Data: 2023-10-17T18:54:22.123 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.123 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.124 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.124 DEBUG:boto:Params: {} 2023-10-17T18:54:22.124 DEBUG:boto:Token: None 2023-10-17T18:54:22.125 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048?versionId=null 2023-10-17T18:54:22.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM= 2023-10-17T18:54:22.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VhU8JSUI9IzON8ouankkqUZmqmM=', 'Content-Length': '0'} 2023-10-17T18:54:22.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de60154772ad31c1-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.170 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.171 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.171 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.171 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.172 DEBUG:boto:Method: GET 2023-10-17T18:54:22.172 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.173 DEBUG:boto:Data: 2023-10-17T18:54:22.173 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.173 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.174 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.174 DEBUG:boto:Params: {} 2023-10-17T18:54:22.175 DEBUG:boto:Token: None 2023-10-17T18:54:22.175 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.175 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af563f8951747b49-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.179 DEBUG:boto:b'hbmuog-58obj000000000000000000481000falseobj000000000000000000482023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.179 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.180 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.180 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.180 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.181 DEBUG:boto:Method: GET 2023-10-17T18:54:22.181 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.182 DEBUG:boto:Data: 2023-10-17T18:54:22.182 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.182 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.183 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.183 DEBUG:boto:Params: {} 2023-10-17T18:54:22.184 DEBUG:boto:Token: None 2023-10-17T18:54:22.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.185 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f721f6e5d44f46-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.190 DEBUG:boto:b'hbmuog-58obj000000000000000000481000falseobj000000000000000000482023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.190 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.191 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.191 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.191 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.192 DEBUG:boto:Data: 2023-10-17T18:54:22.192 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.193 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.193 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.193 DEBUG:boto:Params: {} 2023-10-17T18:54:22.194 DEBUG:boto:Token: None 2023-10-17T18:54:22.194 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.195 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+KEHEiN0eGQfoYCUFXrGovof55U= 2023-10-17T18:54:22.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+KEHEiN0eGQfoYCUFXrGovof55U=', 'Content-Length': '0'} 2023-10-17T18:54:22.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee00933f17aa320-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.197 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.198 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.198 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.198 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.199 DEBUG:boto:Data: 2023-10-17T18:54:22.201 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.202 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.202 DEBUG:boto:Params: {} 2023-10-17T18:54:22.203 DEBUG:boto:Token: None 2023-10-17T18:54:22.203 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.204 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+KEHEiN0eGQfoYCUFXrGovof55U= 2023-10-17T18:54:22.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+KEHEiN0eGQfoYCUFXrGovof55U=', 'Content-Length': '0'} 2023-10-17T18:54:22.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d90b1af4ec0498f0-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-17T18:54:22.207 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.207 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.208 DEBUG:boto:Method: GET 2023-10-17T18:54:22.208 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.209 DEBUG:boto:Data: 2023-10-17T18:54:22.209 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.209 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.210 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.210 DEBUG:boto:Params: {} 2023-10-17T18:54:22.211 DEBUG:boto:Token: None 2023-10-17T18:54:22.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:im5R9nD2KV/MAwbGBA+ngF09Yio= 2023-10-17T18:54:22.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:im5R9nD2KV/MAwbGBA+ngF09Yio=', 'Content-Length': '0'} 2023-10-17T18:54:22.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a9cf5c5033a8ce-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.254 DEBUG:boto:path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.255 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.255 DEBUG:boto:Method: GET 2023-10-17T18:54:22.256 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.256 DEBUG:boto:Data: 2023-10-17T18:54:22.256 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.257 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.257 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.258 DEBUG:boto:Params: {} 2023-10-17T18:54:22.258 DEBUG:boto:Token: None 2023-10-17T18:54:22.259 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000048 2023-10-17T18:54:22.259 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:im5R9nD2KV/MAwbGBA+ngF09Yio= 2023-10-17T18:54:22.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:im5R9nD2KV/MAwbGBA+ngF09Yio=', 'Content-Length': '0'} 2023-10-17T18:54:22.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a58f01c4095716b-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.262 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.262 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.263 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.263 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.264 DEBUG:boto:Method: GET 2023-10-17T18:54:22.264 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.264 DEBUG:boto:Data: 2023-10-17T18:54:22.265 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.265 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.266 DEBUG:boto:Params: {} 2023-10-17T18:54:22.266 DEBUG:boto:Token: None 2023-10-17T18:54:22.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8d8307c8f8d094-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.271 DEBUG:boto:b'hbmuog-58obj000000000000000000481000falseobj000000000000000000482023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.271 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.272 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.272 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.273 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.273 DEBUG:boto:Method: GET 2023-10-17T18:54:22.273 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000048 2023-10-17T18:54:22.274 DEBUG:boto:Data: 2023-10-17T18:54:22.274 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.275 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.275 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.275 DEBUG:boto:Params: {} 2023-10-17T18:54:22.276 DEBUG:boto:Token: None 2023-10-17T18:54:22.276 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.277 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1cfa646a828e7b-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.282 DEBUG:boto:b'hbmuog-58obj000000000000000000481000falseobj000000000000000000482023-10-17T18:54:07.836Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-17T18:54:22.283 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.283 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.284 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.284 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.285 DEBUG:boto:Method: GET 2023-10-17T18:54:22.285 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.285 DEBUG:boto:Data: 2023-10-17T18:54:22.286 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.287 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.287 DEBUG:boto:Params: {} 2023-10-17T18:54:22.288 DEBUG:boto:Token: None 2023-10-17T18:54:22.288 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4= 2023-10-17T18:54:22.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4=', 'Content-Length': '0'} 2023-10-17T18:54:22.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045322c3bad248945-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.292 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.292 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.293 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.293 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.293 DEBUG:boto:Method: GET 2023-10-17T18:54:22.294 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.294 DEBUG:boto:Data: 2023-10-17T18:54:22.295 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.295 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.296 DEBUG:boto:Params: {} 2023-10-17T18:54:22.296 DEBUG:boto:Token: None 2023-10-17T18:54:22.297 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4= 2023-10-17T18:54:22.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4=', 'Content-Length': '0'} 2023-10-17T18:54:22.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6b35928769d179-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.342 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.343 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.343 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.344 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.344 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.345 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.345 DEBUG:boto:Data: 2023-10-17T18:54:22.345 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.346 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.346 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.347 DEBUG:boto:Params: {} 2023-10-17T18:54:22.347 DEBUG:boto:Token: None 2023-10-17T18:54:22.348 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgyopSy1Yq/E4ZAZNS/7OAszxy8= 2023-10-17T18:54:22.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgyopSy1Yq/E4ZAZNS/7OAszxy8=', 'Content-Length': '0'} 2023-10-17T18:54:22.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006067ed390c72b77-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.351 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.352 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.352 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.352 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.353 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.353 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.353 DEBUG:boto:Data: 2023-10-17T18:54:22.354 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.355 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.355 DEBUG:boto:Params: {} 2023-10-17T18:54:22.355 DEBUG:boto:Token: None 2023-10-17T18:54:22.356 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.356 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zgyopSy1Yq/E4ZAZNS/7OAszxy8= 2023-10-17T18:54:22.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zgyopSy1Yq/E4ZAZNS/7OAszxy8=', 'Content-Length': '0'} 2023-10-17T18:54:22.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac202a1d36dbc120-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-17T18:54:22.360 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.360 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.361 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.361 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.361 DEBUG:boto:Method: GET 2023-10-17T18:54:22.362 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.362 DEBUG:boto:Data: 2023-10-17T18:54:22.363 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.363 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.364 DEBUG:boto:Params: {} 2023-10-17T18:54:22.364 DEBUG:boto:Token: None 2023-10-17T18:54:22.365 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4= 2023-10-17T18:54:22.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4=', 'Content-Length': '0'} 2023-10-17T18:54:22.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b45476e55bb065-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.410 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.411 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.411 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.412 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.412 DEBUG:boto:Method: GET 2023-10-17T18:54:22.413 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.413 DEBUG:boto:Data: 2023-10-17T18:54:22.413 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.414 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.415 DEBUG:boto:Params: {} 2023-10-17T18:54:22.415 DEBUG:boto:Token: None 2023-10-17T18:54:22.416 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049?versionId=null 2023-10-17T18:54:22.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4= 2023-10-17T18:54:22.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NI1R1RN38Wzsd2CFIyjEtqB/yu4=', 'Content-Length': '0'} 2023-10-17T18:54:22.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b41b1c8df23662-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.420 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.420 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.421 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.421 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.421 DEBUG:boto:Method: GET 2023-10-17T18:54:22.422 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.422 DEBUG:boto:Data: 2023-10-17T18:54:22.423 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.423 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.424 DEBUG:boto:Params: {} 2023-10-17T18:54:22.424 DEBUG:boto:Token: None 2023-10-17T18:54:22.425 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.425 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cc10132b368e7e4-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.429 DEBUG:boto:b'hbmuog-58obj000000000000000000491000falseobj000000000000000000492023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.430 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.430 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.431 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.431 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.432 DEBUG:boto:Method: GET 2023-10-17T18:54:22.432 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.432 DEBUG:boto:Data: 2023-10-17T18:54:22.433 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.433 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.434 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.434 DEBUG:boto:Params: {} 2023-10-17T18:54:22.434 DEBUG:boto:Token: None 2023-10-17T18:54:22.435 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.435 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029881604a49b58b2-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.441 DEBUG:boto:b'hbmuog-58obj000000000000000000491000falseobj000000000000000000492023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.441 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.442 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.442 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.442 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.443 DEBUG:boto:Data: 2023-10-17T18:54:22.443 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.444 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.444 DEBUG:boto:Params: {} 2023-10-17T18:54:22.445 DEBUG:boto:Token: None 2023-10-17T18:54:22.445 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.446 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xgjXdHAqtTd+BUgro3+cbRX9+6k= 2023-10-17T18:54:22.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xgjXdHAqtTd+BUgro3+cbRX9+6k=', 'Content-Length': '0'} 2023-10-17T18:54:22.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b381d5ab53f81e1-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.449 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.449 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.450 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.450 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.450 DEBUG:boto:Data: 2023-10-17T18:54:22.451 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.452 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.452 DEBUG:boto:Params: {} 2023-10-17T18:54:22.452 DEBUG:boto:Token: None 2023-10-17T18:54:22.453 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xgjXdHAqtTd+BUgro3+cbRX9+6k= 2023-10-17T18:54:22.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xgjXdHAqtTd+BUgro3+cbRX9+6k=', 'Content-Length': '0'} 2023-10-17T18:54:22.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153ea2d0b16c5adb-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-17T18:54:22.457 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.457 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.458 DEBUG:boto:Method: GET 2023-10-17T18:54:22.458 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.458 DEBUG:boto:Data: 2023-10-17T18:54:22.459 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.459 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.460 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.460 DEBUG:boto:Params: {} 2023-10-17T18:54:22.461 DEBUG:boto:Token: None 2023-10-17T18:54:22.461 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.461 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ozJYLQqvu9KmQY6UUCY6xoSa1OE= 2023-10-17T18:54:22.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ozJYLQqvu9KmQY6UUCY6xoSa1OE=', 'Content-Length': '0'} 2023-10-17T18:54:22.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097d936378b8a79d7-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.464 DEBUG:boto:path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.465 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.465 DEBUG:boto:Method: GET 2023-10-17T18:54:22.466 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.466 DEBUG:boto:Data: 2023-10-17T18:54:22.466 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.467 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.468 DEBUG:boto:Params: {} 2023-10-17T18:54:22.468 DEBUG:boto:Token: None 2023-10-17T18:54:22.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000049 2023-10-17T18:54:22.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ozJYLQqvu9KmQY6UUCY6xoSa1OE= 2023-10-17T18:54:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ozJYLQqvu9KmQY6UUCY6xoSa1OE=', 'Content-Length': '0'} 2023-10-17T18:54:22.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055abdcbab5a2dd87-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.514 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.515 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.516 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.516 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.516 DEBUG:boto:Method: GET 2023-10-17T18:54:22.517 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.517 DEBUG:boto:Data: 2023-10-17T18:54:22.518 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.518 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.519 DEBUG:boto:Params: {} 2023-10-17T18:54:22.519 DEBUG:boto:Token: None 2023-10-17T18:54:22.520 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf330e1804fbbff5-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.524 DEBUG:boto:b'hbmuog-58obj000000000000000000491000falseobj000000000000000000492023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.525 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.525 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.526 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.526 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.527 DEBUG:boto:Method: GET 2023-10-17T18:54:22.527 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000049 2023-10-17T18:54:22.528 DEBUG:boto:Data: 2023-10-17T18:54:22.528 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.529 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.529 DEBUG:boto:Params: {} 2023-10-17T18:54:22.530 DEBUG:boto:Token: None 2023-10-17T18:54:22.530 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692358d97ab0af75-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.537 DEBUG:boto:b'hbmuog-58obj000000000000000000491000falseobj000000000000000000492023-10-17T18:54:04.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-17T18:54:22.538 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.538 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.539 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.539 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.540 DEBUG:boto:Method: GET 2023-10-17T18:54:22.540 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.541 DEBUG:boto:Data: 2023-10-17T18:54:22.541 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.542 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.542 DEBUG:boto:Params: {} 2023-10-17T18:54:22.543 DEBUG:boto:Token: None 2023-10-17T18:54:22.543 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk= 2023-10-17T18:54:22.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk=', 'Content-Length': '0'} 2023-10-17T18:54:22.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d532a12a893593a-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.547 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.548 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.548 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.548 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.549 DEBUG:boto:Method: GET 2023-10-17T18:54:22.549 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.550 DEBUG:boto:Data: 2023-10-17T18:54:22.550 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.550 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.551 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.551 DEBUG:boto:Params: {} 2023-10-17T18:54:22.552 DEBUG:boto:Token: None 2023-10-17T18:54:22.552 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.553 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk= 2023-10-17T18:54:22.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk=', 'Content-Length': '0'} 2023-10-17T18:54:22.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39c5abf6da4ff47-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.556 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.556 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.556 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.557 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.557 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.557 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.558 DEBUG:boto:Data: 2023-10-17T18:54:22.558 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.559 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.560 DEBUG:boto:Params: {} 2023-10-17T18:54:22.560 DEBUG:boto:Token: None 2023-10-17T18:54:22.560 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XkyXIpPq/I+ykvQw5dqxWNKBf3M= 2023-10-17T18:54:22.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XkyXIpPq/I+ykvQw5dqxWNKBf3M=', 'Content-Length': '0'} 2023-10-17T18:54:22.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd13eba179592407-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.564 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.565 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.565 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.565 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.566 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.566 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.567 DEBUG:boto:Data: 2023-10-17T18:54:22.567 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.568 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.568 DEBUG:boto:Params: {} 2023-10-17T18:54:22.568 DEBUG:boto:Token: None 2023-10-17T18:54:22.569 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XkyXIpPq/I+ykvQw5dqxWNKBf3M= 2023-10-17T18:54:22.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XkyXIpPq/I+ykvQw5dqxWNKBf3M=', 'Content-Length': '0'} 2023-10-17T18:54:22.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137be331071214cc-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-17T18:54:22.573 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.573 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.573 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.574 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.574 DEBUG:boto:Method: GET 2023-10-17T18:54:22.575 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.575 DEBUG:boto:Data: 2023-10-17T18:54:22.575 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.576 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.576 DEBUG:boto:Params: {} 2023-10-17T18:54:22.577 DEBUG:boto:Token: None 2023-10-17T18:54:22.577 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk= 2023-10-17T18:54:22.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk=', 'Content-Length': '0'} 2023-10-17T18:54:22.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ae671d0b2df0f6-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.622 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.623 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.623 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.624 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.624 DEBUG:boto:Method: GET 2023-10-17T18:54:22.625 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.625 DEBUG:boto:Data: 2023-10-17T18:54:22.625 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.626 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.627 DEBUG:boto:Params: {} 2023-10-17T18:54:22.627 DEBUG:boto:Token: None 2023-10-17T18:54:22.628 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050?versionId=null 2023-10-17T18:54:22.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk= 2023-10-17T18:54:22.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvOeHza2SjRzbW02Q6jknd/5rPk=', 'Content-Length': '0'} 2023-10-17T18:54:22.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2648e626606932-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.632 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.632 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.633 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.633 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.633 DEBUG:boto:Method: GET 2023-10-17T18:54:22.634 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.634 DEBUG:boto:Data: 2023-10-17T18:54:22.635 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.635 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.635 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.636 DEBUG:boto:Params: {} 2023-10-17T18:54:22.636 DEBUG:boto:Token: None 2023-10-17T18:54:22.637 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d740b2a3d420fcb0-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.641 DEBUG:boto:b'hbmuog-58obj000000000000000000501000falseobj000000000000000000502023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.642 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.642 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.643 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.643 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.644 DEBUG:boto:Method: GET 2023-10-17T18:54:22.644 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.644 DEBUG:boto:Data: 2023-10-17T18:54:22.645 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.646 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.646 DEBUG:boto:Params: {} 2023-10-17T18:54:22.647 DEBUG:boto:Token: None 2023-10-17T18:54:22.647 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a9a41b3e043abd-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.653 DEBUG:boto:b'hbmuog-58obj000000000000000000501000falseobj000000000000000000502023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.654 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.655 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.655 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.655 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.656 DEBUG:boto:Data: 2023-10-17T18:54:22.656 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.657 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.657 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.658 DEBUG:boto:Params: {} 2023-10-17T18:54:22.658 DEBUG:boto:Token: None 2023-10-17T18:54:22.659 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.659 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oCxzS0fa9fUaX23g/obOpXkgkdY= 2023-10-17T18:54:22.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oCxzS0fa9fUaX23g/obOpXkgkdY=', 'Content-Length': '0'} 2023-10-17T18:54:22.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ae7d7eb6aaace4-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.662 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.663 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.663 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.663 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.664 DEBUG:boto:Data: 2023-10-17T18:54:22.664 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.665 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.665 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.666 DEBUG:boto:Params: {} 2023-10-17T18:54:22.666 DEBUG:boto:Token: None 2023-10-17T18:54:22.666 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.667 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oCxzS0fa9fUaX23g/obOpXkgkdY= 2023-10-17T18:54:22.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oCxzS0fa9fUaX23g/obOpXkgkdY=', 'Content-Length': '0'} 2023-10-17T18:54:22.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0df5b4a6c4b4c3e-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-17T18:54:22.670 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.671 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.671 DEBUG:boto:Method: GET 2023-10-17T18:54:22.672 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.672 DEBUG:boto:Data: 2023-10-17T18:54:22.673 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.674 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.674 DEBUG:boto:Params: {} 2023-10-17T18:54:22.675 DEBUG:boto:Token: None 2023-10-17T18:54:22.675 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z8uiun5aFMDdBK/3+iThs7oVjDY= 2023-10-17T18:54:22.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z8uiun5aFMDdBK/3+iThs7oVjDY=', 'Content-Length': '0'} 2023-10-17T18:54:22.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd782e63e4b4d8e-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.679 DEBUG:boto:path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.679 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.680 DEBUG:boto:Method: GET 2023-10-17T18:54:22.680 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.680 DEBUG:boto:Data: 2023-10-17T18:54:22.681 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.682 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.682 DEBUG:boto:Params: {} 2023-10-17T18:54:22.683 DEBUG:boto:Token: None 2023-10-17T18:54:22.683 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000050 2023-10-17T18:54:22.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z8uiun5aFMDdBK/3+iThs7oVjDY= 2023-10-17T18:54:22.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z8uiun5aFMDdBK/3+iThs7oVjDY=', 'Content-Length': '0'} 2023-10-17T18:54:22.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073143e7f1fd6cbfe-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.687 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.687 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.688 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.688 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.689 DEBUG:boto:Method: GET 2023-10-17T18:54:22.689 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.690 DEBUG:boto:Data: 2023-10-17T18:54:22.690 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.691 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.692 DEBUG:boto:Params: {} 2023-10-17T18:54:22.692 DEBUG:boto:Token: None 2023-10-17T18:54:22.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167e4bd3a14f7e5b-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.697 DEBUG:boto:b'hbmuog-58obj000000000000000000501000falseobj000000000000000000502023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.697 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.698 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.698 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.699 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.699 DEBUG:boto:Method: GET 2023-10-17T18:54:22.699 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000050 2023-10-17T18:54:22.700 DEBUG:boto:Data: 2023-10-17T18:54:22.700 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.701 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.702 DEBUG:boto:Params: {} 2023-10-17T18:54:22.702 DEBUG:boto:Token: None 2023-10-17T18:54:22.703 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.703 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005faadd7696b95220-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.750 DEBUG:boto:b'hbmuog-58obj000000000000000000501000falseobj000000000000000000502023-10-17T18:54:06.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-17T18:54:22.752 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.752 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.753 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.753 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.753 DEBUG:boto:Method: GET 2023-10-17T18:54:22.754 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.754 DEBUG:boto:Data: 2023-10-17T18:54:22.755 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.755 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.756 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.756 DEBUG:boto:Params: {} 2023-10-17T18:54:22.757 DEBUG:boto:Token: None 2023-10-17T18:54:22.757 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE= 2023-10-17T18:54:22.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE=', 'Content-Length': '0'} 2023-10-17T18:54:22.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad08a9abf356ce65-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.761 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.762 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.762 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.762 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.763 DEBUG:boto:Method: GET 2023-10-17T18:54:22.763 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.764 DEBUG:boto:Data: 2023-10-17T18:54:22.764 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.764 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.765 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.765 DEBUG:boto:Params: {} 2023-10-17T18:54:22.766 DEBUG:boto:Token: None 2023-10-17T18:54:22.766 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE= 2023-10-17T18:54:22.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE=', 'Content-Length': '0'} 2023-10-17T18:54:22.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0d80b6f0c53645-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.770 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.771 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.771 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.772 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.772 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.772 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.773 DEBUG:boto:Data: 2023-10-17T18:54:22.773 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.774 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.774 DEBUG:boto:Params: {} 2023-10-17T18:54:22.775 DEBUG:boto:Token: None 2023-10-17T18:54:22.775 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v7lDDzHnjmFGw4R2wn3ysYMP9CM= 2023-10-17T18:54:22.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v7lDDzHnjmFGw4R2wn3ysYMP9CM=', 'Content-Length': '0'} 2023-10-17T18:54:22.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262c2bc6232ba3af-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.779 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.779 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.780 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.780 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.780 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.781 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.781 DEBUG:boto:Data: 2023-10-17T18:54:22.782 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.782 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.783 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.783 DEBUG:boto:Params: {} 2023-10-17T18:54:22.783 DEBUG:boto:Token: None 2023-10-17T18:54:22.784 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v7lDDzHnjmFGw4R2wn3ysYMP9CM= 2023-10-17T18:54:22.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v7lDDzHnjmFGw4R2wn3ysYMP9CM=', 'Content-Length': '0'} 2023-10-17T18:54:22.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000734f406b62e16018-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-17T18:54:22.788 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.788 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.789 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.790 DEBUG:boto:Method: GET 2023-10-17T18:54:22.790 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.790 DEBUG:boto:Data: 2023-10-17T18:54:22.791 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.791 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.792 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.792 DEBUG:boto:Params: {} 2023-10-17T18:54:22.793 DEBUG:boto:Token: None 2023-10-17T18:54:22.793 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE= 2023-10-17T18:54:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE=', 'Content-Length': '0'} 2023-10-17T18:54:22.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba68abd272312ebe-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.797 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.798 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.798 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.799 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.799 DEBUG:boto:Method: GET 2023-10-17T18:54:22.799 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.800 DEBUG:boto:Data: 2023-10-17T18:54:22.800 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.801 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.802 DEBUG:boto:Params: {} 2023-10-17T18:54:22.802 DEBUG:boto:Token: None 2023-10-17T18:54:22.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055?versionId=null 2023-10-17T18:54:22.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE= 2023-10-17T18:54:22.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nNteLZ1aHA37IgP0ISB01ybWRYE=', 'Content-Length': '0'} 2023-10-17T18:54:22.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c81e3fd554da02-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.806 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.807 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.807 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.808 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.808 DEBUG:boto:Method: GET 2023-10-17T18:54:22.809 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.809 DEBUG:boto:Data: 2023-10-17T18:54:22.810 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.810 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.811 DEBUG:boto:Params: {} 2023-10-17T18:54:22.811 DEBUG:boto:Token: None 2023-10-17T18:54:22.812 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a17a3a24b9182edf-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.818 DEBUG:boto:b'hbmuog-58obj000000000000000000551000falseobj000000000000000000552023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.819 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.819 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.820 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.820 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.820 DEBUG:boto:Method: GET 2023-10-17T18:54:22.821 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.821 DEBUG:boto:Data: 2023-10-17T18:54:22.822 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.823 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.823 DEBUG:boto:Params: {} 2023-10-17T18:54:22.823 DEBUG:boto:Token: None 2023-10-17T18:54:22.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec0c197c6ca8832-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.830 DEBUG:boto:b'hbmuog-58obj000000000000000000551000falseobj000000000000000000552023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.830 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.831 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.831 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.832 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.832 DEBUG:boto:Data: 2023-10-17T18:54:22.832 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.833 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.834 DEBUG:boto:Params: {} 2023-10-17T18:54:22.834 DEBUG:boto:Token: None 2023-10-17T18:54:22.835 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tT8Qac069E4joNUbSR5OsJjj1NY= 2023-10-17T18:54:22.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tT8Qac069E4joNUbSR5OsJjj1NY=', 'Content-Length': '0'} 2023-10-17T18:54:22.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e008180ce6ac24d-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.838 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.839 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.839 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.839 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.840 DEBUG:boto:Data: 2023-10-17T18:54:22.840 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.841 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.841 DEBUG:boto:Params: {} 2023-10-17T18:54:22.842 DEBUG:boto:Token: None 2023-10-17T18:54:22.842 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tT8Qac069E4joNUbSR5OsJjj1NY= 2023-10-17T18:54:22.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tT8Qac069E4joNUbSR5OsJjj1NY=', 'Content-Length': '0'} 2023-10-17T18:54:22.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d165e94badfb0af-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-17T18:54:22.847 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.847 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.848 DEBUG:boto:Method: GET 2023-10-17T18:54:22.848 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.849 DEBUG:boto:Data: 2023-10-17T18:54:22.849 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.850 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.850 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.850 DEBUG:boto:Params: {} 2023-10-17T18:54:22.851 DEBUG:boto:Token: None 2023-10-17T18:54:22.852 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IwZBU21QA4ckKFO8uFR4EC1qV+0= 2023-10-17T18:54:22.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IwZBU21QA4ckKFO8uFR4EC1qV+0=', 'Content-Length': '0'} 2023-10-17T18:54:22.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3641098af96714-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.902 DEBUG:boto:path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.903 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.904 DEBUG:boto:Method: GET 2023-10-17T18:54:22.904 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.905 DEBUG:boto:Data: 2023-10-17T18:54:22.905 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.906 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.906 DEBUG:boto:Params: {} 2023-10-17T18:54:22.907 DEBUG:boto:Token: None 2023-10-17T18:54:22.908 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000055 2023-10-17T18:54:22.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IwZBU21QA4ckKFO8uFR4EC1qV+0= 2023-10-17T18:54:22.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IwZBU21QA4ckKFO8uFR4EC1qV+0=', 'Content-Length': '0'} 2023-10-17T18:54:22.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568d1aac39715abd-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.911 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.912 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.912 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.913 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.913 DEBUG:boto:Method: GET 2023-10-17T18:54:22.914 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.914 DEBUG:boto:Data: 2023-10-17T18:54:22.915 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.916 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.916 DEBUG:boto:Params: {} 2023-10-17T18:54:22.917 DEBUG:boto:Token: None 2023-10-17T18:54:22.917 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.918 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050acb839b4340677-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.921 DEBUG:boto:b'hbmuog-58obj000000000000000000551000falseobj000000000000000000552023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.922 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.923 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.923 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.924 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.924 DEBUG:boto:Method: GET 2023-10-17T18:54:22.924 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000055 2023-10-17T18:54:22.925 DEBUG:boto:Data: 2023-10-17T18:54:22.925 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.926 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.927 DEBUG:boto:Params: {} 2023-10-17T18:54:22.927 DEBUG:boto:Token: None 2023-10-17T18:54:22.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.928 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f59c575602ba7f-00652ed85e-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.934 DEBUG:boto:b'hbmuog-58obj000000000000000000551000falseobj000000000000000000552023-10-17T18:54:03.852Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:22.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-17T18:54:22.935 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.936 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.936 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.936 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.937 DEBUG:boto:Method: GET 2023-10-17T18:54:22.937 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.938 DEBUG:boto:Data: 2023-10-17T18:54:22.938 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.939 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.939 DEBUG:boto:Params: {} 2023-10-17T18:54:22.940 DEBUG:boto:Token: None 2023-10-17T18:54:22.940 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4= 2023-10-17T18:54:22.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4=', 'Content-Length': '0'} 2023-10-17T18:54:22.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1569444910e5eb-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.944 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.945 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.945 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.945 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.946 DEBUG:boto:Method: GET 2023-10-17T18:54:22.946 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.947 DEBUG:boto:Data: 2023-10-17T18:54:22.947 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.948 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.948 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.948 DEBUG:boto:Params: {} 2023-10-17T18:54:22.949 DEBUG:boto:Token: None 2023-10-17T18:54:22.949 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.950 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4= 2023-10-17T18:54:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4=', 'Content-Length': '0'} 2023-10-17T18:54:22.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60406150c4a3f24-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.953 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.954 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.954 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.955 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.955 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.955 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.956 DEBUG:boto:Data: 2023-10-17T18:54:22.956 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.957 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.957 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.957 DEBUG:boto:Params: {} 2023-10-17T18:54:22.958 DEBUG:boto:Token: None 2023-10-17T18:54:22.958 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ijj9qLhF7cxWQl0U3BRfXwdhEKs= 2023-10-17T18:54:22.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ijj9qLhF7cxWQl0U3BRfXwdhEKs=', 'Content-Length': '0'} 2023-10-17T18:54:22.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4762e4543e3444-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.962 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.962 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.963 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.963 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.963 DEBUG:boto:Method: HEAD 2023-10-17T18:54:22.964 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.964 DEBUG:boto:Data: 2023-10-17T18:54:22.965 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.965 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.965 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.966 DEBUG:boto:Params: {} 2023-10-17T18:54:22.966 DEBUG:boto:Token: None 2023-10-17T18:54:22.967 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.967 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ijj9qLhF7cxWQl0U3BRfXwdhEKs= 2023-10-17T18:54:22.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ijj9qLhF7cxWQl0U3BRfXwdhEKs=', 'Content-Length': '0'} 2023-10-17T18:54:22.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c005011c1106fb8b-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-17T18:54:22.971 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.971 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.972 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.972 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.972 DEBUG:boto:Method: GET 2023-10-17T18:54:22.973 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.973 DEBUG:boto:Data: 2023-10-17T18:54:22.974 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.974 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.975 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.975 DEBUG:boto:Params: {} 2023-10-17T18:54:22.975 DEBUG:boto:Token: None 2023-10-17T18:54:22.976 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.976 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4= 2023-10-17T18:54:22.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4=', 'Content-Length': '0'} 2023-10-17T18:54:22.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f811616019de1656-00652ed85e-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.979 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.980 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:22.980 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.981 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.981 DEBUG:boto:Method: GET 2023-10-17T18:54:22.982 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.982 DEBUG:boto:Data: 2023-10-17T18:54:22.982 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.983 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:22.983 DEBUG:boto:Port: 8000 2023-10-17T18:54:22.984 DEBUG:boto:Params: {} 2023-10-17T18:54:22.984 DEBUG:boto:Token: None 2023-10-17T18:54:22.985 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/obj00000000000000000056?versionId=null 2023-10-17T18:54:22.985 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4= 2023-10-17T18:54:22.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tmNVW5B/InvYPeIf+m5CmNsV9G4=', 'Content-Length': '0'} 2023-10-17T18:54:22.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000557bdfa41940efaf-00652ed85e-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:22.988 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:22.989 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:22.989 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:22.990 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:22.990 DEBUG:boto:Method: GET 2023-10-17T18:54:22.990 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:22.991 DEBUG:boto:Data: 2023-10-17T18:54:22.991 DEBUG:boto:Headers: {} 2023-10-17T18:54:22.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:22.992 DEBUG:boto:Port: 8001 2023-10-17T18:54:22.992 DEBUG:boto:Params: {} 2023-10-17T18:54:22.993 DEBUG:boto:Token: None 2023-10-17T18:54:22.993 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:22 GMT /hbmuog-58/ 2023-10-17T18:54:22.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo= 2023-10-17T18:54:22.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:c3jGk30NOH7skyY1Vn/7d7XZYmo=', 'Content-Length': '0'} 2023-10-17T18:54:22.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047df6faa9cbbd71a-00652ed85e-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.042 DEBUG:boto:b'hbmuog-58obj000000000000000000561000falseobj000000000000000000562023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.043 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.044 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.044 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.044 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.045 DEBUG:boto:Method: GET 2023-10-17T18:54:23.045 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.046 DEBUG:boto:Data: 2023-10-17T18:54:23.046 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.047 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.047 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.047 DEBUG:boto:Params: {} 2023-10-17T18:54:23.048 DEBUG:boto:Token: None 2023-10-17T18:54:23.048 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff819ca12ed3833-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.055 DEBUG:boto:b'hbmuog-58obj000000000000000000561000falseobj000000000000000000562023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.055 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.055 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.056 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.056 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.057 DEBUG:boto:Data: 2023-10-17T18:54:23.057 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.058 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.058 DEBUG:boto:Params: {} 2023-10-17T18:54:23.059 DEBUG:boto:Token: None 2023-10-17T18:54:23.059 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvCoeGsgdysCqr8rg+OAvaZ7GiY= 2023-10-17T18:54:23.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvCoeGsgdysCqr8rg+OAvaZ7GiY=', 'Content-Length': '0'} 2023-10-17T18:54:23.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000563774d1ccd099bc-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.062 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.063 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.063 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.064 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.064 DEBUG:boto:Data: 2023-10-17T18:54:23.064 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.065 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.066 DEBUG:boto:Params: {} 2023-10-17T18:54:23.066 DEBUG:boto:Token: None 2023-10-17T18:54:23.067 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CvCoeGsgdysCqr8rg+OAvaZ7GiY= 2023-10-17T18:54:23.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CvCoeGsgdysCqr8rg+OAvaZ7GiY=', 'Content-Length': '0'} 2023-10-17T18:54:23.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7b0ed7c02e5371-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-17T18:54:23.071 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.072 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.072 DEBUG:boto:Method: GET 2023-10-17T18:54:23.072 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.073 DEBUG:boto:Data: 2023-10-17T18:54:23.073 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.074 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.074 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.075 DEBUG:boto:Params: {} 2023-10-17T18:54:23.075 DEBUG:boto:Token: None 2023-10-17T18:54:23.076 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bYwdpCy/8t6M5RkqmyU7/6N08tM= 2023-10-17T18:54:23.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bYwdpCy/8t6M5RkqmyU7/6N08tM=', 'Content-Length': '0'} 2023-10-17T18:54:23.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004749a5304ad29f97-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.079 DEBUG:boto:path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.080 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.080 DEBUG:boto:Method: GET 2023-10-17T18:54:23.081 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.081 DEBUG:boto:Data: 2023-10-17T18:54:23.082 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.083 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.083 DEBUG:boto:Params: {} 2023-10-17T18:54:23.083 DEBUG:boto:Token: None 2023-10-17T18:54:23.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000056 2023-10-17T18:54:23.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bYwdpCy/8t6M5RkqmyU7/6N08tM= 2023-10-17T18:54:23.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bYwdpCy/8t6M5RkqmyU7/6N08tM=', 'Content-Length': '0'} 2023-10-17T18:54:23.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063087516f45e4b79-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.130 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.131 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.131 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.132 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.132 DEBUG:boto:Method: GET 2023-10-17T18:54:23.133 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.133 DEBUG:boto:Data: 2023-10-17T18:54:23.133 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.134 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.135 DEBUG:boto:Params: {} 2023-10-17T18:54:23.135 DEBUG:boto:Token: None 2023-10-17T18:54:23.136 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdd6c46646990fa3-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.140 DEBUG:boto:b'hbmuog-58obj000000000000000000561000falseobj000000000000000000562023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.140 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.141 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.141 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.141 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.142 DEBUG:boto:Method: GET 2023-10-17T18:54:23.142 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000056 2023-10-17T18:54:23.143 DEBUG:boto:Data: 2023-10-17T18:54:23.143 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.143 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.144 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.144 DEBUG:boto:Params: {} 2023-10-17T18:54:23.145 DEBUG:boto:Token: None 2023-10-17T18:54:23.145 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7a627b65c77326-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.151 DEBUG:boto:b'hbmuog-58obj000000000000000000561000falseobj000000000000000000562023-10-17T18:54:06.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-17T18:54:23.152 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.152 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.152 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.153 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.153 DEBUG:boto:Method: GET 2023-10-17T18:54:23.154 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.154 DEBUG:boto:Data: 2023-10-17T18:54:23.154 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.155 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.156 DEBUG:boto:Params: {} 2023-10-17T18:54:23.156 DEBUG:boto:Token: None 2023-10-17T18:54:23.157 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig= 2023-10-17T18:54:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig=', 'Content-Length': '0'} 2023-10-17T18:54:23.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a923c796b37c6faf-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.160 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.161 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.161 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.162 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.162 DEBUG:boto:Method: GET 2023-10-17T18:54:23.162 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.163 DEBUG:boto:Data: 2023-10-17T18:54:23.163 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.164 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.164 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.164 DEBUG:boto:Params: {} 2023-10-17T18:54:23.165 DEBUG:boto:Token: None 2023-10-17T18:54:23.165 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.166 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig= 2023-10-17T18:54:23.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig=', 'Content-Length': '0'} 2023-10-17T18:54:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb3ea4490e37880-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.170 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.171 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.172 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.172 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.173 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.173 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.174 DEBUG:boto:Data: 2023-10-17T18:54:23.174 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.175 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.176 DEBUG:boto:Params: {} 2023-10-17T18:54:23.176 DEBUG:boto:Token: None 2023-10-17T18:54:23.176 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.177 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:un3/YAT0KGwrjP5kpcHHOBZtB08= 2023-10-17T18:54:23.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:un3/YAT0KGwrjP5kpcHHOBZtB08=', 'Content-Length': '0'} 2023-10-17T18:54:23.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e6cfa518fc1fbf-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.180 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.180 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.181 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.181 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.181 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.182 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.182 DEBUG:boto:Data: 2023-10-17T18:54:23.183 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.183 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.184 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.184 DEBUG:boto:Params: {} 2023-10-17T18:54:23.185 DEBUG:boto:Token: None 2023-10-17T18:54:23.186 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.186 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:un3/YAT0KGwrjP5kpcHHOBZtB08= 2023-10-17T18:54:23.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:un3/YAT0KGwrjP5kpcHHOBZtB08=', 'Content-Length': '0'} 2023-10-17T18:54:23.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f068409ed7cb47-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-17T18:54:23.190 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.191 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.191 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.192 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.192 DEBUG:boto:Method: GET 2023-10-17T18:54:23.193 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.193 DEBUG:boto:Data: 2023-10-17T18:54:23.193 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.194 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.194 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.195 DEBUG:boto:Params: {} 2023-10-17T18:54:23.195 DEBUG:boto:Token: None 2023-10-17T18:54:23.195 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig= 2023-10-17T18:54:23.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig=', 'Content-Length': '0'} 2023-10-17T18:54:23.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a4fac402048b9f-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.242 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.243 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.243 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.244 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.244 DEBUG:boto:Method: GET 2023-10-17T18:54:23.244 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.245 DEBUG:boto:Data: 2023-10-17T18:54:23.245 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.246 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.246 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.246 DEBUG:boto:Params: {} 2023-10-17T18:54:23.247 DEBUG:boto:Token: None 2023-10-17T18:54:23.247 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057?versionId=null 2023-10-17T18:54:23.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig= 2023-10-17T18:54:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tX5SGoXt+/23Z5NQ3zuY3tQaAig=', 'Content-Length': '0'} 2023-10-17T18:54:23.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfcd69e40e935b7b-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.251 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.251 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.252 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.252 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.253 DEBUG:boto:Method: GET 2023-10-17T18:54:23.253 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.253 DEBUG:boto:Data: 2023-10-17T18:54:23.254 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.254 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.255 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.255 DEBUG:boto:Params: {} 2023-10-17T18:54:23.255 DEBUG:boto:Token: None 2023-10-17T18:54:23.256 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.256 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4614fa54fe4a73-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.260 DEBUG:boto:b'hbmuog-58obj000000000000000000571000falseobj000000000000000000572023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.260 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.261 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.261 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.262 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.262 DEBUG:boto:Method: GET 2023-10-17T18:54:23.262 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.263 DEBUG:boto:Data: 2023-10-17T18:54:23.263 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.264 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.264 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.264 DEBUG:boto:Params: {} 2023-10-17T18:54:23.265 DEBUG:boto:Token: None 2023-10-17T18:54:23.265 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.266 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d27251d7565c3a-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.314 DEBUG:boto:b'hbmuog-58obj000000000000000000571000falseobj000000000000000000572023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.315 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.315 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.316 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.316 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.317 DEBUG:boto:Data: 2023-10-17T18:54:23.317 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.318 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.318 DEBUG:boto:Params: {} 2023-10-17T18:54:23.319 DEBUG:boto:Token: None 2023-10-17T18:54:23.319 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6piW/tUS9NtFgdJRZJHgNns/UO8= 2023-10-17T18:54:23.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6piW/tUS9NtFgdJRZJHgNns/UO8=', 'Content-Length': '0'} 2023-10-17T18:54:23.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b7701f9367d095-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.322 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.323 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.323 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.324 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.324 DEBUG:boto:Data: 2023-10-17T18:54:23.324 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.325 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.325 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.325 DEBUG:boto:Params: {} 2023-10-17T18:54:23.326 DEBUG:boto:Token: None 2023-10-17T18:54:23.326 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6piW/tUS9NtFgdJRZJHgNns/UO8= 2023-10-17T18:54:23.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6piW/tUS9NtFgdJRZJHgNns/UO8=', 'Content-Length': '0'} 2023-10-17T18:54:23.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b919d25db833318-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-17T18:54:23.330 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.330 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.331 DEBUG:boto:Method: GET 2023-10-17T18:54:23.331 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.331 DEBUG:boto:Data: 2023-10-17T18:54:23.332 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.332 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.333 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.333 DEBUG:boto:Params: {} 2023-10-17T18:54:23.333 DEBUG:boto:Token: None 2023-10-17T18:54:23.334 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LznWgTHW7jkMM0MXY7uWrAbaMu0= 2023-10-17T18:54:23.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LznWgTHW7jkMM0MXY7uWrAbaMu0=', 'Content-Length': '0'} 2023-10-17T18:54:23.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8c10626ae0dea2-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.337 DEBUG:boto:path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.337 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.338 DEBUG:boto:Method: GET 2023-10-17T18:54:23.338 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.339 DEBUG:boto:Data: 2023-10-17T18:54:23.339 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.340 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.340 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.340 DEBUG:boto:Params: {} 2023-10-17T18:54:23.341 DEBUG:boto:Token: None 2023-10-17T18:54:23.341 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000057 2023-10-17T18:54:23.342 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LznWgTHW7jkMM0MXY7uWrAbaMu0= 2023-10-17T18:54:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LznWgTHW7jkMM0MXY7uWrAbaMu0=', 'Content-Length': '0'} 2023-10-17T18:54:23.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076511625b1c79a97-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.345 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.345 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.346 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.346 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.346 DEBUG:boto:Method: GET 2023-10-17T18:54:23.347 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.347 DEBUG:boto:Data: 2023-10-17T18:54:23.348 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.348 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.349 DEBUG:boto:Params: {} 2023-10-17T18:54:23.349 DEBUG:boto:Token: None 2023-10-17T18:54:23.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7fccd778c3b5b8c-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.354 DEBUG:boto:b'hbmuog-58obj000000000000000000571000falseobj000000000000000000572023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.355 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.355 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.355 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.356 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.356 DEBUG:boto:Method: GET 2023-10-17T18:54:23.356 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000057 2023-10-17T18:54:23.357 DEBUG:boto:Data: 2023-10-17T18:54:23.357 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.358 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.358 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.358 DEBUG:boto:Params: {} 2023-10-17T18:54:23.359 DEBUG:boto:Token: None 2023-10-17T18:54:23.359 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.360 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59c39075059c815-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.365 DEBUG:boto:b'hbmuog-58obj000000000000000000571000falseobj000000000000000000572023-10-17T18:54:08.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-17T18:54:23.366 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.367 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.367 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.367 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.368 DEBUG:boto:Method: GET 2023-10-17T18:54:23.368 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.369 DEBUG:boto:Data: 2023-10-17T18:54:23.369 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.370 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.370 DEBUG:boto:Params: {} 2023-10-17T18:54:23.371 DEBUG:boto:Token: None 2023-10-17T18:54:23.371 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U= 2023-10-17T18:54:23.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U=', 'Content-Length': '0'} 2023-10-17T18:54:23.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741315aed19e90e4-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.375 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.375 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.376 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.376 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.376 DEBUG:boto:Method: GET 2023-10-17T18:54:23.377 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.377 DEBUG:boto:Data: 2023-10-17T18:54:23.378 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.378 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.378 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.379 DEBUG:boto:Params: {} 2023-10-17T18:54:23.379 DEBUG:boto:Token: None 2023-10-17T18:54:23.380 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U= 2023-10-17T18:54:23.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U=', 'Content-Length': '0'} 2023-10-17T18:54:23.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40dee934d1f8914-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.383 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.383 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.384 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.384 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.385 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.385 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.385 DEBUG:boto:Data: 2023-10-17T18:54:23.386 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.386 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.387 DEBUG:boto:Params: {} 2023-10-17T18:54:23.387 DEBUG:boto:Token: None 2023-10-17T18:54:23.388 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O3qsMhJ66H7mKaS/i99NVzsvJF8= 2023-10-17T18:54:23.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O3qsMhJ66H7mKaS/i99NVzsvJF8=', 'Content-Length': '0'} 2023-10-17T18:54:23.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5a761b709686d2-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.391 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.392 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.392 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.392 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.393 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.393 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.393 DEBUG:boto:Data: 2023-10-17T18:54:23.394 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.395 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.395 DEBUG:boto:Params: {} 2023-10-17T18:54:23.395 DEBUG:boto:Token: None 2023-10-17T18:54:23.396 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O3qsMhJ66H7mKaS/i99NVzsvJF8= 2023-10-17T18:54:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O3qsMhJ66H7mKaS/i99NVzsvJF8=', 'Content-Length': '0'} 2023-10-17T18:54:23.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c3f3831c4428a1-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-17T18:54:23.399 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.400 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.400 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.400 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.401 DEBUG:boto:Method: GET 2023-10-17T18:54:23.401 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.402 DEBUG:boto:Data: 2023-10-17T18:54:23.402 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.403 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.403 DEBUG:boto:Params: {} 2023-10-17T18:54:23.404 DEBUG:boto:Token: None 2023-10-17T18:54:23.404 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.405 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U= 2023-10-17T18:54:23.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U=', 'Content-Length': '0'} 2023-10-17T18:54:23.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed636d24ff9b4154-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.450 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.451 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.451 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.452 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.452 DEBUG:boto:Method: GET 2023-10-17T18:54:23.452 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.453 DEBUG:boto:Data: 2023-10-17T18:54:23.453 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.454 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.455 DEBUG:boto:Params: {} 2023-10-17T18:54:23.455 DEBUG:boto:Token: None 2023-10-17T18:54:23.456 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058?versionId=null 2023-10-17T18:54:23.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U= 2023-10-17T18:54:23.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WC0r2xU9BjgGaS4KVbJSGe1Nx4U=', 'Content-Length': '0'} 2023-10-17T18:54:23.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007386186683413fde-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.459 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.460 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.460 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.460 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.461 DEBUG:boto:Method: GET 2023-10-17T18:54:23.461 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.462 DEBUG:boto:Data: 2023-10-17T18:54:23.462 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.463 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.463 DEBUG:boto:Params: {} 2023-10-17T18:54:23.464 DEBUG:boto:Token: None 2023-10-17T18:54:23.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa4fa881fe4a1f0-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.468 DEBUG:boto:b'hbmuog-58obj000000000000000000581000falseobj000000000000000000582023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.468 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.469 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.469 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.470 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.470 DEBUG:boto:Method: GET 2023-10-17T18:54:23.470 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.471 DEBUG:boto:Data: 2023-10-17T18:54:23.471 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.472 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.472 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.472 DEBUG:boto:Params: {} 2023-10-17T18:54:23.473 DEBUG:boto:Token: None 2023-10-17T18:54:23.473 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.474 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5a50fd1f8e8614-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.479 DEBUG:boto:b'hbmuog-58obj000000000000000000581000falseobj000000000000000000582023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.480 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.480 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.480 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.481 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.481 DEBUG:boto:Data: 2023-10-17T18:54:23.482 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.482 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.482 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.483 DEBUG:boto:Params: {} 2023-10-17T18:54:23.483 DEBUG:boto:Token: None 2023-10-17T18:54:23.484 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.484 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K0w/99BDe9OoCrv3oqO5nGLC9iM= 2023-10-17T18:54:23.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K0w/99BDe9OoCrv3oqO5nGLC9iM=', 'Content-Length': '0'} 2023-10-17T18:54:23.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4fefd19cd3d44f4-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.487 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.487 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.488 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.488 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.488 DEBUG:boto:Data: 2023-10-17T18:54:23.489 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.490 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.490 DEBUG:boto:Params: {} 2023-10-17T18:54:23.490 DEBUG:boto:Token: None 2023-10-17T18:54:23.491 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K0w/99BDe9OoCrv3oqO5nGLC9iM= 2023-10-17T18:54:23.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K0w/99BDe9OoCrv3oqO5nGLC9iM=', 'Content-Length': '0'} 2023-10-17T18:54:23.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000166ee3aaba8ea917-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-17T18:54:23.495 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.495 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.496 DEBUG:boto:Method: GET 2023-10-17T18:54:23.496 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.496 DEBUG:boto:Data: 2023-10-17T18:54:23.497 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.497 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.498 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.498 DEBUG:boto:Params: {} 2023-10-17T18:54:23.499 DEBUG:boto:Token: None 2023-10-17T18:54:23.499 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.500 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TfUQ9WskBS4qFIe2pSwSSFqV6EU= 2023-10-17T18:54:23.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TfUQ9WskBS4qFIe2pSwSSFqV6EU=', 'Content-Length': '0'} 2023-10-17T18:54:23.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646ff234514e4e4b-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.546 DEBUG:boto:path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.547 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.547 DEBUG:boto:Method: GET 2023-10-17T18:54:23.548 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.548 DEBUG:boto:Data: 2023-10-17T18:54:23.549 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.550 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.550 DEBUG:boto:Params: {} 2023-10-17T18:54:23.550 DEBUG:boto:Token: None 2023-10-17T18:54:23.551 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000058 2023-10-17T18:54:23.551 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TfUQ9WskBS4qFIe2pSwSSFqV6EU= 2023-10-17T18:54:23.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TfUQ9WskBS4qFIe2pSwSSFqV6EU=', 'Content-Length': '0'} 2023-10-17T18:54:23.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b31f8366e9d7ea-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.555 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.555 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.555 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.556 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.556 DEBUG:boto:Method: GET 2023-10-17T18:54:23.557 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.557 DEBUG:boto:Data: 2023-10-17T18:54:23.557 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.558 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.559 DEBUG:boto:Params: {} 2023-10-17T18:54:23.559 DEBUG:boto:Token: None 2023-10-17T18:54:23.559 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000290dba74e462723a-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.564 DEBUG:boto:b'hbmuog-58obj000000000000000000581000falseobj000000000000000000582023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.564 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.565 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.565 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.565 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.566 DEBUG:boto:Method: GET 2023-10-17T18:54:23.566 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000058 2023-10-17T18:54:23.567 DEBUG:boto:Data: 2023-10-17T18:54:23.567 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.568 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.568 DEBUG:boto:Params: {} 2023-10-17T18:54:23.569 DEBUG:boto:Token: None 2023-10-17T18:54:23.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.569 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbfda772e405b262-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.574 DEBUG:boto:b'hbmuog-58obj000000000000000000581000falseobj000000000000000000582023-10-17T18:54:03.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-17T18:54:23.575 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.576 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.576 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.576 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.577 DEBUG:boto:Method: GET 2023-10-17T18:54:23.577 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.578 DEBUG:boto:Data: 2023-10-17T18:54:23.578 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.579 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.579 DEBUG:boto:Params: {} 2023-10-17T18:54:23.580 DEBUG:boto:Token: None 2023-10-17T18:54:23.580 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck= 2023-10-17T18:54:23.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck=', 'Content-Length': '0'} 2023-10-17T18:54:23.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0bc0fb032664aef-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.584 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.585 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.585 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.585 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.586 DEBUG:boto:Method: GET 2023-10-17T18:54:23.586 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.587 DEBUG:boto:Data: 2023-10-17T18:54:23.587 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.588 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.588 DEBUG:boto:Params: {} 2023-10-17T18:54:23.589 DEBUG:boto:Token: None 2023-10-17T18:54:23.589 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck= 2023-10-17T18:54:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck=', 'Content-Length': '0'} 2023-10-17T18:54:23.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e717db45821db1c4-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.592 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.593 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.593 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.594 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.594 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.594 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.595 DEBUG:boto:Data: 2023-10-17T18:54:23.595 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.596 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.596 DEBUG:boto:Params: {} 2023-10-17T18:54:23.597 DEBUG:boto:Token: None 2023-10-17T18:54:23.597 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yXYTRjnZd2XacPPfItmD5tn9GlM= 2023-10-17T18:54:23.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yXYTRjnZd2XacPPfItmD5tn9GlM=', 'Content-Length': '0'} 2023-10-17T18:54:23.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007847d65f877bad8-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.601 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.601 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.601 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.602 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.602 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.603 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.603 DEBUG:boto:Data: 2023-10-17T18:54:23.604 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.604 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.604 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.605 DEBUG:boto:Params: {} 2023-10-17T18:54:23.605 DEBUG:boto:Token: None 2023-10-17T18:54:23.606 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.606 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yXYTRjnZd2XacPPfItmD5tn9GlM= 2023-10-17T18:54:23.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yXYTRjnZd2XacPPfItmD5tn9GlM=', 'Content-Length': '0'} 2023-10-17T18:54:23.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d18138fbb238a6-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-17T18:54:23.610 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.610 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.611 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.611 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.612 DEBUG:boto:Method: GET 2023-10-17T18:54:23.612 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.612 DEBUG:boto:Data: 2023-10-17T18:54:23.613 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.613 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.614 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.614 DEBUG:boto:Params: {} 2023-10-17T18:54:23.614 DEBUG:boto:Token: None 2023-10-17T18:54:23.615 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.615 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck= 2023-10-17T18:54:23.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck=', 'Content-Length': '0'} 2023-10-17T18:54:23.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004059f61822f0b38a-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.618 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.619 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.619 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.620 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.620 DEBUG:boto:Method: GET 2023-10-17T18:54:23.620 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.621 DEBUG:boto:Data: 2023-10-17T18:54:23.621 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.622 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.622 DEBUG:boto:Params: {} 2023-10-17T18:54:23.623 DEBUG:boto:Token: None 2023-10-17T18:54:23.623 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059?versionId=null 2023-10-17T18:54:23.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck= 2023-10-17T18:54:23.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6Jl/n6ET3rRz8P8mcL06fuc59ck=', 'Content-Length': '0'} 2023-10-17T18:54:23.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e14f5572b9b74c5-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.627 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.627 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.628 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.628 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.628 DEBUG:boto:Method: GET 2023-10-17T18:54:23.629 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.629 DEBUG:boto:Data: 2023-10-17T18:54:23.630 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.630 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.631 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.631 DEBUG:boto:Params: {} 2023-10-17T18:54:23.631 DEBUG:boto:Token: None 2023-10-17T18:54:23.632 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.632 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0daef9ee7eb07ae-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.636 DEBUG:boto:b'hbmuog-58obj000000000000000000591000falseobj000000000000000000592023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.636 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.637 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.637 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.638 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.638 DEBUG:boto:Method: GET 2023-10-17T18:54:23.639 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.639 DEBUG:boto:Data: 2023-10-17T18:54:23.639 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.640 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.641 DEBUG:boto:Params: {} 2023-10-17T18:54:23.641 DEBUG:boto:Token: None 2023-10-17T18:54:23.642 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd4ce1f7b00827c-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.690 DEBUG:boto:b'hbmuog-58obj000000000000000000591000falseobj000000000000000000592023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.691 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.692 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.692 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.692 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.693 DEBUG:boto:Data: 2023-10-17T18:54:23.693 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.694 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.694 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.695 DEBUG:boto:Params: {} 2023-10-17T18:54:23.695 DEBUG:boto:Token: None 2023-10-17T18:54:23.695 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I7268UU5ECjASG8hylyAHXTx9qQ= 2023-10-17T18:54:23.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I7268UU5ECjASG8hylyAHXTx9qQ=', 'Content-Length': '0'} 2023-10-17T18:54:23.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000474d25e716b37981-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.699 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.700 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.700 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.700 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.701 DEBUG:boto:Data: 2023-10-17T18:54:23.701 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.702 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.702 DEBUG:boto:Params: {} 2023-10-17T18:54:23.703 DEBUG:boto:Token: None 2023-10-17T18:54:23.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I7268UU5ECjASG8hylyAHXTx9qQ= 2023-10-17T18:54:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I7268UU5ECjASG8hylyAHXTx9qQ=', 'Content-Length': '0'} 2023-10-17T18:54:23.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf058110a8a6251b-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-17T18:54:23.707 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.707 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.708 DEBUG:boto:Method: GET 2023-10-17T18:54:23.708 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.709 DEBUG:boto:Data: 2023-10-17T18:54:23.709 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.709 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.710 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.710 DEBUG:boto:Params: {} 2023-10-17T18:54:23.711 DEBUG:boto:Token: None 2023-10-17T18:54:23.711 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CQ4+LxqAjHj78Qc/cMfQWakZcNY= 2023-10-17T18:54:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CQ4+LxqAjHj78Qc/cMfQWakZcNY=', 'Content-Length': '0'} 2023-10-17T18:54:23.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de91f434062ae977-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.758 DEBUG:boto:path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.759 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.759 DEBUG:boto:Method: GET 2023-10-17T18:54:23.760 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.760 DEBUG:boto:Data: 2023-10-17T18:54:23.761 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.761 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.761 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.762 DEBUG:boto:Params: {} 2023-10-17T18:54:23.762 DEBUG:boto:Token: None 2023-10-17T18:54:23.763 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000059 2023-10-17T18:54:23.763 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CQ4+LxqAjHj78Qc/cMfQWakZcNY= 2023-10-17T18:54:23.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CQ4+LxqAjHj78Qc/cMfQWakZcNY=', 'Content-Length': '0'} 2023-10-17T18:54:23.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e62c133319d0f2-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.766 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.767 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.767 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.768 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.768 DEBUG:boto:Method: GET 2023-10-17T18:54:23.769 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.769 DEBUG:boto:Data: 2023-10-17T18:54:23.769 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.770 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.771 DEBUG:boto:Params: {} 2023-10-17T18:54:23.771 DEBUG:boto:Token: None 2023-10-17T18:54:23.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f4b1fddbd27eab-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.776 DEBUG:boto:b'hbmuog-58obj000000000000000000591000falseobj000000000000000000592023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.776 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.777 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.777 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.778 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.778 DEBUG:boto:Method: GET 2023-10-17T18:54:23.778 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000059 2023-10-17T18:54:23.779 DEBUG:boto:Data: 2023-10-17T18:54:23.779 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.780 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.780 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.780 DEBUG:boto:Params: {} 2023-10-17T18:54:23.781 DEBUG:boto:Token: None 2023-10-17T18:54:23.781 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.782 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002107a50a75aa7810-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.787 DEBUG:boto:b'hbmuog-58obj000000000000000000591000falseobj000000000000000000592023-10-17T18:54:04.856Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-17T18:54:23.788 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.789 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.789 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.790 DEBUG:boto:Method: GET 2023-10-17T18:54:23.790 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.791 DEBUG:boto:Data: 2023-10-17T18:54:23.791 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.792 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.792 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.793 DEBUG:boto:Params: {} 2023-10-17T18:54:23.793 DEBUG:boto:Token: None 2023-10-17T18:54:23.794 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.794 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o= 2023-10-17T18:54:23.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o=', 'Content-Length': '0'} 2023-10-17T18:54:23.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ca50dc69dd40bf-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.797 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.798 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.798 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.799 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.799 DEBUG:boto:Method: GET 2023-10-17T18:54:23.800 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.800 DEBUG:boto:Data: 2023-10-17T18:54:23.800 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.801 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.802 DEBUG:boto:Params: {} 2023-10-17T18:54:23.802 DEBUG:boto:Token: None 2023-10-17T18:54:23.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.803 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o= 2023-10-17T18:54:23.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o=', 'Content-Length': '0'} 2023-10-17T18:54:23.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad5800a823cbfb5-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.806 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.807 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.807 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.808 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.808 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.808 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.809 DEBUG:boto:Data: 2023-10-17T18:54:23.809 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.809 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.810 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.810 DEBUG:boto:Params: {} 2023-10-17T18:54:23.810 DEBUG:boto:Token: None 2023-10-17T18:54:23.811 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f57Xv0QGcIlOzCUbSYPL3LSOk2A= 2023-10-17T18:54:23.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f57Xv0QGcIlOzCUbSYPL3LSOk2A=', 'Content-Length': '0'} 2023-10-17T18:54:23.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900efc654f7f4526-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.814 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.815 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.815 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.815 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.816 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.816 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.817 DEBUG:boto:Data: 2023-10-17T18:54:23.817 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.818 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.818 DEBUG:boto:Params: {} 2023-10-17T18:54:23.818 DEBUG:boto:Token: None 2023-10-17T18:54:23.819 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f57Xv0QGcIlOzCUbSYPL3LSOk2A= 2023-10-17T18:54:23.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f57Xv0QGcIlOzCUbSYPL3LSOk2A=', 'Content-Length': '0'} 2023-10-17T18:54:23.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46812f2c7cb4e04-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-17T18:54:23.823 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.823 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.824 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.824 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.825 DEBUG:boto:Method: GET 2023-10-17T18:54:23.825 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.825 DEBUG:boto:Data: 2023-10-17T18:54:23.826 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.827 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.827 DEBUG:boto:Params: {} 2023-10-17T18:54:23.828 DEBUG:boto:Token: None 2023-10-17T18:54:23.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.829 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o= 2023-10-17T18:54:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o=', 'Content-Length': '0'} 2023-10-17T18:54:23.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0497cc69034d0a-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.832 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.832 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.833 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.833 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.833 DEBUG:boto:Method: GET 2023-10-17T18:54:23.834 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.834 DEBUG:boto:Data: 2023-10-17T18:54:23.835 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.837 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.837 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.837 DEBUG:boto:Params: {} 2023-10-17T18:54:23.838 DEBUG:boto:Token: None 2023-10-17T18:54:23.839 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060?versionId=null 2023-10-17T18:54:23.839 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o= 2023-10-17T18:54:23.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WSuDVYOPuhL7svOTGgO/s2GjU4o=', 'Content-Length': '0'} 2023-10-17T18:54:23.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a2035ffc92e9e4-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.842 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.843 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.843 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.844 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.844 DEBUG:boto:Method: GET 2023-10-17T18:54:23.844 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.845 DEBUG:boto:Data: 2023-10-17T18:54:23.845 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.846 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.846 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.847 DEBUG:boto:Params: {} 2023-10-17T18:54:23.847 DEBUG:boto:Token: None 2023-10-17T18:54:23.848 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.848 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dffd9fac0808e39c-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.894 DEBUG:boto:b'hbmuog-58obj000000000000000000601000falseobj000000000000000000602023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.895 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.896 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.896 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.896 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.897 DEBUG:boto:Method: GET 2023-10-17T18:54:23.897 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.898 DEBUG:boto:Data: 2023-10-17T18:54:23.898 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.899 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.899 DEBUG:boto:Params: {} 2023-10-17T18:54:23.900 DEBUG:boto:Token: None 2023-10-17T18:54:23.900 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000710ff765da3871b3-00652ed85f-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.906 DEBUG:boto:b'hbmuog-58obj000000000000000000601000falseobj000000000000000000602023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.907 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.907 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.908 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.908 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.909 DEBUG:boto:Data: 2023-10-17T18:54:23.909 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.909 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.910 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.910 DEBUG:boto:Params: {} 2023-10-17T18:54:23.911 DEBUG:boto:Token: None 2023-10-17T18:54:23.911 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iTyEeD5bYucEzuraqnm5eeB1/Ow= 2023-10-17T18:54:23.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iTyEeD5bYucEzuraqnm5eeB1/Ow=', 'Content-Length': '0'} 2023-10-17T18:54:23.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090737f8cdbaa6578-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.915 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.915 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.916 DEBUG:boto:Method: HEAD 2023-10-17T18:54:23.916 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.916 DEBUG:boto:Data: 2023-10-17T18:54:23.917 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.918 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.918 DEBUG:boto:Params: {} 2023-10-17T18:54:23.918 DEBUG:boto:Token: None 2023-10-17T18:54:23.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iTyEeD5bYucEzuraqnm5eeB1/Ow= 2023-10-17T18:54:23.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iTyEeD5bYucEzuraqnm5eeB1/Ow=', 'Content-Length': '0'} 2023-10-17T18:54:23.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c2033079ba8fd7-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-17T18:54:23.923 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.923 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.924 DEBUG:boto:Method: GET 2023-10-17T18:54:23.924 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.925 DEBUG:boto:Data: 2023-10-17T18:54:23.925 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.926 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.926 DEBUG:boto:Params: {} 2023-10-17T18:54:23.927 DEBUG:boto:Token: None 2023-10-17T18:54:23.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.928 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q5BV0eXKRXXa7H/wmAkwrG5IaSg= 2023-10-17T18:54:23.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q5BV0eXKRXXa7H/wmAkwrG5IaSg=', 'Content-Length': '0'} 2023-10-17T18:54:23.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f725271d73f7fd-00652ed85f-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.931 DEBUG:boto:path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.931 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.932 DEBUG:boto:Method: GET 2023-10-17T18:54:23.932 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.933 DEBUG:boto:Data: 2023-10-17T18:54:23.933 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.934 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.935 DEBUG:boto:Params: {} 2023-10-17T18:54:23.935 DEBUG:boto:Token: None 2023-10-17T18:54:23.935 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/obj00000000000000000060 2023-10-17T18:54:23.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q5BV0eXKRXXa7H/wmAkwrG5IaSg= 2023-10-17T18:54:23.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q5BV0eXKRXXa7H/wmAkwrG5IaSg=', 'Content-Length': '0'} 2023-10-17T18:54:23.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0d32021c6b90e6-00652ed85f-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.982 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.983 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.983 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.984 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.984 DEBUG:boto:Method: GET 2023-10-17T18:54:23.985 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.985 DEBUG:boto:Data: 2023-10-17T18:54:23.986 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:23.987 DEBUG:boto:Port: 8001 2023-10-17T18:54:23.987 DEBUG:boto:Params: {} 2023-10-17T18:54:23.988 DEBUG:boto:Token: None 2023-10-17T18:54:23.988 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:23.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8bef2adfd63ea93-00652ed85f-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:23.992 DEBUG:boto:b'hbmuog-58obj000000000000000000601000falseobj000000000000000000602023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:23.993 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:23.993 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:23.994 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.994 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.995 DEBUG:boto:Method: GET 2023-10-17T18:54:23.995 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000060 2023-10-17T18:54:23.996 DEBUG:boto:Data: 2023-10-17T18:54:23.996 DEBUG:boto:Headers: {} 2023-10-17T18:54:23.996 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:23.997 DEBUG:boto:Port: 8000 2023-10-17T18:54:23.997 DEBUG:boto:Params: {} 2023-10-17T18:54:23.998 DEBUG:boto:Token: None 2023-10-17T18:54:23.998 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:23 GMT /hbmuog-58/ 2023-10-17T18:54:23.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw= 2023-10-17T18:54:23.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+av1ZEjWKJgGLeiSoy+R4VPR5Vw=', 'Content-Length': '0'} 2023-10-17T18:54:24.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff8b8ca8477bb50-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.004 DEBUG:boto:b'hbmuog-58obj000000000000000000601000falseobj000000000000000000602023-10-17T18:54:07.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-17T18:54:24.005 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.006 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.006 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.007 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.007 DEBUG:boto:Method: GET 2023-10-17T18:54:24.008 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.008 DEBUG:boto:Data: 2023-10-17T18:54:24.008 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.009 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.010 DEBUG:boto:Params: {} 2023-10-17T18:54:24.010 DEBUG:boto:Token: None 2023-10-17T18:54:24.011 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc= 2023-10-17T18:54:24.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc=', 'Content-Length': '0'} 2023-10-17T18:54:24.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a4b21b1ebf9337-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.015 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.015 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.016 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.016 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.017 DEBUG:boto:Method: GET 2023-10-17T18:54:24.017 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.017 DEBUG:boto:Data: 2023-10-17T18:54:24.018 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.019 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.019 DEBUG:boto:Params: {} 2023-10-17T18:54:24.020 DEBUG:boto:Token: None 2023-10-17T18:54:24.020 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc= 2023-10-17T18:54:24.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc=', 'Content-Length': '0'} 2023-10-17T18:54:24.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ea36f0edd4933d-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.024 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.024 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.025 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.025 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.025 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.026 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.026 DEBUG:boto:Data: 2023-10-17T18:54:24.027 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.028 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.028 DEBUG:boto:Params: {} 2023-10-17T18:54:24.028 DEBUG:boto:Token: None 2023-10-17T18:54:24.029 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WGyHq3mNWU7ifn2GSI/7lrJmAiM= 2023-10-17T18:54:24.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WGyHq3mNWU7ifn2GSI/7lrJmAiM=', 'Content-Length': '0'} 2023-10-17T18:54:24.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ff70e0949acdb6-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.032 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.033 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.033 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.034 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.034 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.034 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.035 DEBUG:boto:Data: 2023-10-17T18:54:24.035 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.036 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.036 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.037 DEBUG:boto:Params: {} 2023-10-17T18:54:24.037 DEBUG:boto:Token: None 2023-10-17T18:54:24.037 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.038 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WGyHq3mNWU7ifn2GSI/7lrJmAiM= 2023-10-17T18:54:24.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WGyHq3mNWU7ifn2GSI/7lrJmAiM=', 'Content-Length': '0'} 2023-10-17T18:54:24.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898466c000dc4c19-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-17T18:54:24.042 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.042 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.043 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.043 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.044 DEBUG:boto:Method: GET 2023-10-17T18:54:24.044 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.045 DEBUG:boto:Data: 2023-10-17T18:54:24.045 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.046 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.046 DEBUG:boto:Params: {} 2023-10-17T18:54:24.047 DEBUG:boto:Token: None 2023-10-17T18:54:24.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc= 2023-10-17T18:54:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc=', 'Content-Length': '0'} 2023-10-17T18:54:24.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c75dc9c89f63994-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.094 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.095 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.096 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.096 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.096 DEBUG:boto:Method: GET 2023-10-17T18:54:24.097 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.097 DEBUG:boto:Data: 2023-10-17T18:54:24.098 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.099 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.099 DEBUG:boto:Params: {} 2023-10-17T18:54:24.100 DEBUG:boto:Token: None 2023-10-17T18:54:24.100 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061?versionId=null 2023-10-17T18:54:24.101 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc= 2023-10-17T18:54:24.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0r8CZvPXOkBHJdaBqg1dbr9yrpc=', 'Content-Length': '0'} 2023-10-17T18:54:24.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f737cff2058ec1ab-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.104 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.104 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.105 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.105 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.106 DEBUG:boto:Method: GET 2023-10-17T18:54:24.106 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.107 DEBUG:boto:Data: 2023-10-17T18:54:24.107 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.107 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.108 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.108 DEBUG:boto:Params: {} 2023-10-17T18:54:24.109 DEBUG:boto:Token: None 2023-10-17T18:54:24.109 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.110 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce73a20e595da7f2-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.113 DEBUG:boto:b'hbmuog-58obj000000000000000000611000falseobj000000000000000000612023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.114 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.114 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.115 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.115 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.115 DEBUG:boto:Method: GET 2023-10-17T18:54:24.116 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.116 DEBUG:boto:Data: 2023-10-17T18:54:24.117 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.117 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.118 DEBUG:boto:Params: {} 2023-10-17T18:54:24.118 DEBUG:boto:Token: None 2023-10-17T18:54:24.119 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.119 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d36047915e8ddd4-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.124 DEBUG:boto:b'hbmuog-58obj000000000000000000611000falseobj000000000000000000612023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.125 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.126 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.126 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.127 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.127 DEBUG:boto:Data: 2023-10-17T18:54:24.127 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.128 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.128 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.129 DEBUG:boto:Params: {} 2023-10-17T18:54:24.129 DEBUG:boto:Token: None 2023-10-17T18:54:24.129 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.130 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TYYutOWgSRubVQ1ZI+qKDBGuuGM= 2023-10-17T18:54:24.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TYYutOWgSRubVQ1ZI+qKDBGuuGM=', 'Content-Length': '0'} 2023-10-17T18:54:24.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310dc7e3bb45e7df-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.133 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.133 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.134 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.134 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.135 DEBUG:boto:Data: 2023-10-17T18:54:24.135 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.136 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.136 DEBUG:boto:Params: {} 2023-10-17T18:54:24.137 DEBUG:boto:Token: None 2023-10-17T18:54:24.137 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TYYutOWgSRubVQ1ZI+qKDBGuuGM= 2023-10-17T18:54:24.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TYYutOWgSRubVQ1ZI+qKDBGuuGM=', 'Content-Length': '0'} 2023-10-17T18:54:24.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b70efeff9a8e6c-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-17T18:54:24.141 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.142 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.142 DEBUG:boto:Method: GET 2023-10-17T18:54:24.143 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.143 DEBUG:boto:Data: 2023-10-17T18:54:24.144 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.144 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.145 DEBUG:boto:Params: {} 2023-10-17T18:54:24.145 DEBUG:boto:Token: None 2023-10-17T18:54:24.146 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5BC3Jh+fbC2vIktlMdF/vbwGi+Q= 2023-10-17T18:54:24.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5BC3Jh+fbC2vIktlMdF/vbwGi+Q=', 'Content-Length': '0'} 2023-10-17T18:54:24.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d5004770b6be7c4-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.149 DEBUG:boto:path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.150 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.150 DEBUG:boto:Method: GET 2023-10-17T18:54:24.151 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.151 DEBUG:boto:Data: 2023-10-17T18:54:24.151 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.152 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.153 DEBUG:boto:Params: {} 2023-10-17T18:54:24.153 DEBUG:boto:Token: None 2023-10-17T18:54:24.153 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000061 2023-10-17T18:54:24.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5BC3Jh+fbC2vIktlMdF/vbwGi+Q= 2023-10-17T18:54:24.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5BC3Jh+fbC2vIktlMdF/vbwGi+Q=', 'Content-Length': '0'} 2023-10-17T18:54:24.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa32148bf306c9d-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.157 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.158 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.158 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.159 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.159 DEBUG:boto:Method: GET 2023-10-17T18:54:24.159 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.160 DEBUG:boto:Data: 2023-10-17T18:54:24.160 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.161 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.161 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.162 DEBUG:boto:Params: {} 2023-10-17T18:54:24.162 DEBUG:boto:Token: None 2023-10-17T18:54:24.163 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.163 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cd622e689d3669-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.167 DEBUG:boto:b'hbmuog-58obj000000000000000000611000falseobj000000000000000000612023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.167 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.168 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.168 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.169 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.169 DEBUG:boto:Method: GET 2023-10-17T18:54:24.169 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000061 2023-10-17T18:54:24.170 DEBUG:boto:Data: 2023-10-17T18:54:24.170 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.171 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.171 DEBUG:boto:Params: {} 2023-10-17T18:54:24.172 DEBUG:boto:Token: None 2023-10-17T18:54:24.172 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa251cb74ca9fac-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.178 DEBUG:boto:b'hbmuog-58obj000000000000000000611000falseobj000000000000000000612023-10-17T18:54:06.879Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-17T18:54:24.179 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.180 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.180 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.181 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.181 DEBUG:boto:Method: GET 2023-10-17T18:54:24.181 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.182 DEBUG:boto:Data: 2023-10-17T18:54:24.182 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.183 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.183 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.183 DEBUG:boto:Params: {} 2023-10-17T18:54:24.184 DEBUG:boto:Token: None 2023-10-17T18:54:24.184 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.185 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI= 2023-10-17T18:54:24.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI=', 'Content-Length': '0'} 2023-10-17T18:54:24.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca564b92a5ddf68-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.188 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.189 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.189 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.189 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.190 DEBUG:boto:Method: GET 2023-10-17T18:54:24.190 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.191 DEBUG:boto:Data: 2023-10-17T18:54:24.191 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.192 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.192 DEBUG:boto:Params: {} 2023-10-17T18:54:24.193 DEBUG:boto:Token: None 2023-10-17T18:54:24.193 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI= 2023-10-17T18:54:24.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI=', 'Content-Length': '0'} 2023-10-17T18:54:24.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042162b14f2753bce-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.238 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.239 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.239 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.240 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.240 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.240 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.241 DEBUG:boto:Data: 2023-10-17T18:54:24.241 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.242 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.242 DEBUG:boto:Params: {} 2023-10-17T18:54:24.243 DEBUG:boto:Token: None 2023-10-17T18:54:24.245 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.246 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Usv0KXq6uQXkmvVixaQFMv+x4v0= 2023-10-17T18:54:24.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Usv0KXq6uQXkmvVixaQFMv+x4v0=', 'Content-Length': '0'} 2023-10-17T18:54:24.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ea6ea2a26158c01-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.249 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.249 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.250 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.250 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.250 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.251 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.251 DEBUG:boto:Data: 2023-10-17T18:54:24.252 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.252 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.252 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.253 DEBUG:boto:Params: {} 2023-10-17T18:54:24.253 DEBUG:boto:Token: None 2023-10-17T18:54:24.254 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.254 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Usv0KXq6uQXkmvVixaQFMv+x4v0= 2023-10-17T18:54:24.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Usv0KXq6uQXkmvVixaQFMv+x4v0=', 'Content-Length': '0'} 2023-10-17T18:54:24.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5f916a17097e69e-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-17T18:54:24.257 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.258 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.258 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.259 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.259 DEBUG:boto:Method: GET 2023-10-17T18:54:24.260 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.260 DEBUG:boto:Data: 2023-10-17T18:54:24.260 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.261 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.261 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.261 DEBUG:boto:Params: {} 2023-10-17T18:54:24.262 DEBUG:boto:Token: None 2023-10-17T18:54:24.262 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.263 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI= 2023-10-17T18:54:24.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI=', 'Content-Length': '0'} 2023-10-17T18:54:24.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b64ef07217de9f24-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.267 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.267 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.267 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.268 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.268 DEBUG:boto:Method: GET 2023-10-17T18:54:24.268 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.269 DEBUG:boto:Data: 2023-10-17T18:54:24.269 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.270 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.270 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.270 DEBUG:boto:Params: {} 2023-10-17T18:54:24.271 DEBUG:boto:Token: None 2023-10-17T18:54:24.271 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062?versionId=null 2023-10-17T18:54:24.272 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI= 2023-10-17T18:54:24.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9hkAv4mje66lsNdyqG4Ao/+ueJI=', 'Content-Length': '0'} 2023-10-17T18:54:24.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058f7cb10848e584-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.275 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.275 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.275 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.276 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.276 DEBUG:boto:Method: GET 2023-10-17T18:54:24.277 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.277 DEBUG:boto:Data: 2023-10-17T18:54:24.277 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.278 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.279 DEBUG:boto:Params: {} 2023-10-17T18:54:24.279 DEBUG:boto:Token: None 2023-10-17T18:54:24.280 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b8342f11f36e34-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.283 DEBUG:boto:b'hbmuog-58obj000000000000000000621000falseobj000000000000000000622023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.284 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.284 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.285 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.285 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.286 DEBUG:boto:Method: GET 2023-10-17T18:54:24.286 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.287 DEBUG:boto:Data: 2023-10-17T18:54:24.287 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.288 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.288 DEBUG:boto:Params: {} 2023-10-17T18:54:24.289 DEBUG:boto:Token: None 2023-10-17T18:54:24.289 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.290 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d414474db1eb4c-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.295 DEBUG:boto:b'hbmuog-58obj000000000000000000621000falseobj000000000000000000622023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.295 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.296 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.296 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.296 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.297 DEBUG:boto:Data: 2023-10-17T18:54:24.297 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.298 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.298 DEBUG:boto:Params: {} 2023-10-17T18:54:24.299 DEBUG:boto:Token: None 2023-10-17T18:54:24.299 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkTov0banOrLtgzlMVPKnU4u7ek= 2023-10-17T18:54:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkTov0banOrLtgzlMVPKnU4u7ek=', 'Content-Length': '0'} 2023-10-17T18:54:24.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce51b19d71845425-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.303 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.303 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.304 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.304 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.304 DEBUG:boto:Data: 2023-10-17T18:54:24.305 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.305 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.306 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.306 DEBUG:boto:Params: {} 2023-10-17T18:54:24.306 DEBUG:boto:Token: None 2023-10-17T18:54:24.307 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.307 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkTov0banOrLtgzlMVPKnU4u7ek= 2023-10-17T18:54:24.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkTov0banOrLtgzlMVPKnU4u7ek=', 'Content-Length': '0'} 2023-10-17T18:54:24.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189c66e9a5dfd889-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-17T18:54:24.310 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.311 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.311 DEBUG:boto:Method: GET 2023-10-17T18:54:24.311 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.312 DEBUG:boto:Data: 2023-10-17T18:54:24.312 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.313 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.314 DEBUG:boto:Params: {} 2023-10-17T18:54:24.314 DEBUG:boto:Token: None 2023-10-17T18:54:24.315 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EPUvOS2O44D6+FwCibyLW7TYpno= 2023-10-17T18:54:24.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EPUvOS2O44D6+FwCibyLW7TYpno=', 'Content-Length': '0'} 2023-10-17T18:54:24.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043844586b3d1383a-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.358 DEBUG:boto:path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.359 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.360 DEBUG:boto:Method: GET 2023-10-17T18:54:24.360 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.360 DEBUG:boto:Data: 2023-10-17T18:54:24.361 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.361 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.362 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.362 DEBUG:boto:Params: {} 2023-10-17T18:54:24.363 DEBUG:boto:Token: None 2023-10-17T18:54:24.363 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000062 2023-10-17T18:54:24.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EPUvOS2O44D6+FwCibyLW7TYpno= 2023-10-17T18:54:24.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EPUvOS2O44D6+FwCibyLW7TYpno=', 'Content-Length': '0'} 2023-10-17T18:54:24.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2d88f3c6403c33-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.367 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.367 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.368 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.368 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.369 DEBUG:boto:Method: GET 2023-10-17T18:54:24.369 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.369 DEBUG:boto:Data: 2023-10-17T18:54:24.370 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.371 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.371 DEBUG:boto:Params: {} 2023-10-17T18:54:24.371 DEBUG:boto:Token: None 2023-10-17T18:54:24.372 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383e5f8b2cecbdd0-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.376 DEBUG:boto:b'hbmuog-58obj000000000000000000621000falseobj000000000000000000622023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.377 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.377 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.377 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.378 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.378 DEBUG:boto:Method: GET 2023-10-17T18:54:24.379 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000062 2023-10-17T18:54:24.379 DEBUG:boto:Data: 2023-10-17T18:54:24.380 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.380 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.381 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.381 DEBUG:boto:Params: {} 2023-10-17T18:54:24.381 DEBUG:boto:Token: None 2023-10-17T18:54:24.382 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.382 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dd670c19bb0a0ce-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.388 DEBUG:boto:b'hbmuog-58obj000000000000000000621000falseobj000000000000000000622023-10-17T18:54:08.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-17T18:54:24.389 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.389 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.390 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.390 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.391 DEBUG:boto:Method: GET 2023-10-17T18:54:24.391 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.391 DEBUG:boto:Data: 2023-10-17T18:54:24.392 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.392 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.393 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.393 DEBUG:boto:Params: {} 2023-10-17T18:54:24.394 DEBUG:boto:Token: None 2023-10-17T18:54:24.394 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.395 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres= 2023-10-17T18:54:24.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres=', 'Content-Length': '0'} 2023-10-17T18:54:24.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc31c7021f3d4093-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.398 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.398 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.399 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.399 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.400 DEBUG:boto:Method: GET 2023-10-17T18:54:24.400 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.401 DEBUG:boto:Data: 2023-10-17T18:54:24.401 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.401 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.402 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.402 DEBUG:boto:Params: {} 2023-10-17T18:54:24.403 DEBUG:boto:Token: None 2023-10-17T18:54:24.403 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres= 2023-10-17T18:54:24.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres=', 'Content-Length': '0'} 2023-10-17T18:54:24.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1671b68eae98f2-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.407 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.407 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.408 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.408 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.409 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.409 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.409 DEBUG:boto:Data: 2023-10-17T18:54:24.410 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.411 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.411 DEBUG:boto:Params: {} 2023-10-17T18:54:24.411 DEBUG:boto:Token: None 2023-10-17T18:54:24.412 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVGdWB7K+tq27u4l2wbJ/PISGaU= 2023-10-17T18:54:24.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVGdWB7K+tq27u4l2wbJ/PISGaU=', 'Content-Length': '0'} 2023-10-17T18:54:24.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f854abc51cd1cd-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.415 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.416 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.416 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.416 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.417 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.417 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.418 DEBUG:boto:Data: 2023-10-17T18:54:24.418 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.418 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.419 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.419 DEBUG:boto:Params: {} 2023-10-17T18:54:24.420 DEBUG:boto:Token: None 2023-10-17T18:54:24.420 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVGdWB7K+tq27u4l2wbJ/PISGaU= 2023-10-17T18:54:24.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVGdWB7K+tq27u4l2wbJ/PISGaU=', 'Content-Length': '0'} 2023-10-17T18:54:24.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eeed931aa2fbfd5-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-17T18:54:24.424 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.425 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.425 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.426 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.426 DEBUG:boto:Method: GET 2023-10-17T18:54:24.426 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.427 DEBUG:boto:Data: 2023-10-17T18:54:24.427 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.428 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.428 DEBUG:boto:Params: {} 2023-10-17T18:54:24.429 DEBUG:boto:Token: None 2023-10-17T18:54:24.429 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres= 2023-10-17T18:54:24.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres=', 'Content-Length': '0'} 2023-10-17T18:54:24.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ff751c72c6d19f-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.478 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.479 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.479 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.480 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.480 DEBUG:boto:Method: GET 2023-10-17T18:54:24.481 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.481 DEBUG:boto:Data: 2023-10-17T18:54:24.482 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.482 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.482 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.483 DEBUG:boto:Params: {} 2023-10-17T18:54:24.483 DEBUG:boto:Token: None 2023-10-17T18:54:24.484 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065?versionId=null 2023-10-17T18:54:24.484 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres= 2023-10-17T18:54:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7RbDpNx7CO9udSoImo93pfkSres=', 'Content-Length': '0'} 2023-10-17T18:54:24.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e502cf3b22ded518-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.488 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.488 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.489 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.489 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.489 DEBUG:boto:Method: GET 2023-10-17T18:54:24.490 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.490 DEBUG:boto:Data: 2023-10-17T18:54:24.491 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.491 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.492 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.492 DEBUG:boto:Params: {} 2023-10-17T18:54:24.493 DEBUG:boto:Token: None 2023-10-17T18:54:24.493 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e6e18622266731-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.497 DEBUG:boto:b'hbmuog-58obj000000000000000000651000falseobj000000000000000000652023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.498 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.498 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.499 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.499 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.500 DEBUG:boto:Method: GET 2023-10-17T18:54:24.500 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.500 DEBUG:boto:Data: 2023-10-17T18:54:24.501 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.502 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.502 DEBUG:boto:Params: {} 2023-10-17T18:54:24.503 DEBUG:boto:Token: None 2023-10-17T18:54:24.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e406681931229e41-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.509 DEBUG:boto:b'hbmuog-58obj000000000000000000651000falseobj000000000000000000652023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.510 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.510 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.510 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.511 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.511 DEBUG:boto:Data: 2023-10-17T18:54:24.512 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.512 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.513 DEBUG:boto:Params: {} 2023-10-17T18:54:24.513 DEBUG:boto:Token: None 2023-10-17T18:54:24.514 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.514 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CBX6a598vjA1bB/TRA+6WiqjqRw= 2023-10-17T18:54:24.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CBX6a598vjA1bB/TRA+6WiqjqRw=', 'Content-Length': '0'} 2023-10-17T18:54:24.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f207c598705ff1fa-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.517 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.518 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.518 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.519 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.519 DEBUG:boto:Data: 2023-10-17T18:54:24.519 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.520 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.520 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.520 DEBUG:boto:Params: {} 2023-10-17T18:54:24.521 DEBUG:boto:Token: None 2023-10-17T18:54:24.521 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.522 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CBX6a598vjA1bB/TRA+6WiqjqRw= 2023-10-17T18:54:24.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CBX6a598vjA1bB/TRA+6WiqjqRw=', 'Content-Length': '0'} 2023-10-17T18:54:24.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d84d2eb6055d747-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-17T18:54:24.525 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.525 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.526 DEBUG:boto:Method: GET 2023-10-17T18:54:24.526 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.526 DEBUG:boto:Data: 2023-10-17T18:54:24.527 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.527 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.528 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.528 DEBUG:boto:Params: {} 2023-10-17T18:54:24.528 DEBUG:boto:Token: None 2023-10-17T18:54:24.529 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.529 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oDOQlqx6kPpb5Fcpc74LX9kjl7Y= 2023-10-17T18:54:24.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oDOQlqx6kPpb5Fcpc74LX9kjl7Y=', 'Content-Length': '0'} 2023-10-17T18:54:24.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d385269c4747043-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.533 DEBUG:boto:path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.533 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.533 DEBUG:boto:Method: GET 2023-10-17T18:54:24.534 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.534 DEBUG:boto:Data: 2023-10-17T18:54:24.535 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.535 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.535 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.536 DEBUG:boto:Params: {} 2023-10-17T18:54:24.536 DEBUG:boto:Token: None 2023-10-17T18:54:24.537 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000065 2023-10-17T18:54:24.537 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oDOQlqx6kPpb5Fcpc74LX9kjl7Y= 2023-10-17T18:54:24.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oDOQlqx6kPpb5Fcpc74LX9kjl7Y=', 'Content-Length': '0'} 2023-10-17T18:54:24.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026eaccbae6a38a45-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.540 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.540 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.541 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.541 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.542 DEBUG:boto:Method: GET 2023-10-17T18:54:24.542 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.543 DEBUG:boto:Data: 2023-10-17T18:54:24.543 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.543 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.544 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.544 DEBUG:boto:Params: {} 2023-10-17T18:54:24.545 DEBUG:boto:Token: None 2023-10-17T18:54:24.545 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.546 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d44ada09ab4f9ce-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.549 DEBUG:boto:b'hbmuog-58obj000000000000000000651000falseobj000000000000000000652023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.550 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.550 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.551 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.551 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.551 DEBUG:boto:Method: GET 2023-10-17T18:54:24.552 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000065 2023-10-17T18:54:24.552 DEBUG:boto:Data: 2023-10-17T18:54:24.553 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.553 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.554 DEBUG:boto:Params: {} 2023-10-17T18:54:24.554 DEBUG:boto:Token: None 2023-10-17T18:54:24.555 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246478366e8f4be1-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.560 DEBUG:boto:b'hbmuog-58obj000000000000000000651000falseobj000000000000000000652023-10-17T18:54:07.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-17T18:54:24.561 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.562 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.562 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.562 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.563 DEBUG:boto:Method: GET 2023-10-17T18:54:24.563 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.564 DEBUG:boto:Data: 2023-10-17T18:54:24.564 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.564 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.565 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.565 DEBUG:boto:Params: {} 2023-10-17T18:54:24.566 DEBUG:boto:Token: None 2023-10-17T18:54:24.566 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.567 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So= 2023-10-17T18:54:24.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So=', 'Content-Length': '0'} 2023-10-17T18:54:24.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f909d08f029347eb-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.570 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.570 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.571 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.571 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.572 DEBUG:boto:Method: GET 2023-10-17T18:54:24.572 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.572 DEBUG:boto:Data: 2023-10-17T18:54:24.573 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.573 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.574 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.574 DEBUG:boto:Params: {} 2023-10-17T18:54:24.574 DEBUG:boto:Token: None 2023-10-17T18:54:24.575 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.575 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So= 2023-10-17T18:54:24.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So=', 'Content-Length': '0'} 2023-10-17T18:54:24.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fcabeddb06f4a6-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.618 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.619 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.619 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.619 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.620 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.620 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.621 DEBUG:boto:Data: 2023-10-17T18:54:24.621 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.621 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.622 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.622 DEBUG:boto:Params: {} 2023-10-17T18:54:24.623 DEBUG:boto:Token: None 2023-10-17T18:54:24.623 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:klLjpIc6Cz8HfhQScDXgzUxYRQ4= 2023-10-17T18:54:24.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:klLjpIc6Cz8HfhQScDXgzUxYRQ4=', 'Content-Length': '0'} 2023-10-17T18:54:24.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93f8398ccb6c48e-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.626 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.627 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.627 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.627 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.628 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.628 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.629 DEBUG:boto:Data: 2023-10-17T18:54:24.629 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.629 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.630 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.630 DEBUG:boto:Params: {} 2023-10-17T18:54:24.630 DEBUG:boto:Token: None 2023-10-17T18:54:24.631 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:klLjpIc6Cz8HfhQScDXgzUxYRQ4= 2023-10-17T18:54:24.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:klLjpIc6Cz8HfhQScDXgzUxYRQ4=', 'Content-Length': '0'} 2023-10-17T18:54:24.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b659698676def55-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-17T18:54:24.635 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.635 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.636 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.636 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.636 DEBUG:boto:Method: GET 2023-10-17T18:54:24.637 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.637 DEBUG:boto:Data: 2023-10-17T18:54:24.638 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.638 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.639 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.639 DEBUG:boto:Params: {} 2023-10-17T18:54:24.639 DEBUG:boto:Token: None 2023-10-17T18:54:24.640 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.640 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So= 2023-10-17T18:54:24.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So=', 'Content-Length': '0'} 2023-10-17T18:54:24.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a862ecf6cbdd5ca9-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.644 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.644 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.645 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.645 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.646 DEBUG:boto:Method: GET 2023-10-17T18:54:24.646 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.646 DEBUG:boto:Data: 2023-10-17T18:54:24.647 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.647 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.648 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.648 DEBUG:boto:Params: {} 2023-10-17T18:54:24.649 DEBUG:boto:Token: None 2023-10-17T18:54:24.649 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133?versionId=null 2023-10-17T18:54:24.649 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So= 2023-10-17T18:54:24.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XhfehHCuYSMZLhli9ggaLB6O2So=', 'Content-Length': '0'} 2023-10-17T18:54:24.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004115f8450e665c4d-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.653 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.653 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.653 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.654 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.654 DEBUG:boto:Method: GET 2023-10-17T18:54:24.655 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.655 DEBUG:boto:Data: 2023-10-17T18:54:24.655 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.656 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.656 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.657 DEBUG:boto:Params: {} 2023-10-17T18:54:24.657 DEBUG:boto:Token: None 2023-10-17T18:54:24.657 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.658 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d7a2fd76c09ef4-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.661 DEBUG:boto:b'hbmuog-58obj000000000000000001331000falseobj000000000000000001332023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.662 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.662 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.663 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.663 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.664 DEBUG:boto:Method: GET 2023-10-17T18:54:24.664 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.664 DEBUG:boto:Data: 2023-10-17T18:54:24.665 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.665 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.666 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.666 DEBUG:boto:Params: {} 2023-10-17T18:54:24.666 DEBUG:boto:Token: None 2023-10-17T18:54:24.667 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.667 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2bbacf793c7eaa-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.672 DEBUG:boto:b'hbmuog-58obj000000000000000001331000falseobj000000000000000001332023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.673 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.673 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.674 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.674 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.675 DEBUG:boto:Data: 2023-10-17T18:54:24.675 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.675 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.676 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.676 DEBUG:boto:Params: {} 2023-10-17T18:54:24.677 DEBUG:boto:Token: None 2023-10-17T18:54:24.677 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.677 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+DsHa9gMKa9l6LMUO9DaK5Co93Q= 2023-10-17T18:54:24.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+DsHa9gMKa9l6LMUO9DaK5Co93Q=', 'Content-Length': '0'} 2023-10-17T18:54:24.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5182bb1b75b7e66-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.680 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.681 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.681 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.681 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.682 DEBUG:boto:Data: 2023-10-17T18:54:24.682 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.683 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.683 DEBUG:boto:Params: {} 2023-10-17T18:54:24.683 DEBUG:boto:Token: None 2023-10-17T18:54:24.684 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+DsHa9gMKa9l6LMUO9DaK5Co93Q= 2023-10-17T18:54:24.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+DsHa9gMKa9l6LMUO9DaK5Co93Q=', 'Content-Length': '0'} 2023-10-17T18:54:24.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6cb1ea50768b781-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-17T18:54:24.687 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.688 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.688 DEBUG:boto:Method: GET 2023-10-17T18:54:24.689 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.689 DEBUG:boto:Data: 2023-10-17T18:54:24.690 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.690 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.691 DEBUG:boto:Params: {} 2023-10-17T18:54:24.691 DEBUG:boto:Token: None 2023-10-17T18:54:24.692 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sP0t4YBUiGYtXBCubObjPYd+Olc= 2023-10-17T18:54:24.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sP0t4YBUiGYtXBCubObjPYd+Olc=', 'Content-Length': '0'} 2023-10-17T18:54:24.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc477de6021aeb64-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.696 DEBUG:boto:path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.696 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.697 DEBUG:boto:Method: GET 2023-10-17T18:54:24.697 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.697 DEBUG:boto:Data: 2023-10-17T18:54:24.698 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.699 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.699 DEBUG:boto:Params: {} 2023-10-17T18:54:24.700 DEBUG:boto:Token: None 2023-10-17T18:54:24.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000133 2023-10-17T18:54:24.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sP0t4YBUiGYtXBCubObjPYd+Olc= 2023-10-17T18:54:24.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sP0t4YBUiGYtXBCubObjPYd+Olc=', 'Content-Length': '0'} 2023-10-17T18:54:24.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0b2a6405ee2c9e-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.746 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.747 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.747 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.748 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.748 DEBUG:boto:Method: GET 2023-10-17T18:54:24.749 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.749 DEBUG:boto:Data: 2023-10-17T18:54:24.749 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.750 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.751 DEBUG:boto:Params: {} 2023-10-17T18:54:24.751 DEBUG:boto:Token: None 2023-10-17T18:54:24.752 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aa73791c5c729c9-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.756 DEBUG:boto:b'hbmuog-58obj000000000000000001331000falseobj000000000000000001332023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.757 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.757 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.757 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.758 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.758 DEBUG:boto:Method: GET 2023-10-17T18:54:24.759 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000133 2023-10-17T18:54:24.759 DEBUG:boto:Data: 2023-10-17T18:54:24.760 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.761 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.761 DEBUG:boto:Params: {} 2023-10-17T18:54:24.761 DEBUG:boto:Token: None 2023-10-17T18:54:24.762 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b11a6007b11492-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.767 DEBUG:boto:b'hbmuog-58obj000000000000000001331000falseobj000000000000000001332023-10-17T18:54:05.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-17T18:54:24.768 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.769 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.769 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.769 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.770 DEBUG:boto:Method: GET 2023-10-17T18:54:24.770 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.771 DEBUG:boto:Data: 2023-10-17T18:54:24.771 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.771 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.772 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.772 DEBUG:boto:Params: {} 2023-10-17T18:54:24.773 DEBUG:boto:Token: None 2023-10-17T18:54:24.773 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4= 2023-10-17T18:54:24.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4=', 'Content-Length': '0'} 2023-10-17T18:54:24.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa741166ca788fcf-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.818 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.819 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.820 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.820 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.820 DEBUG:boto:Method: GET 2023-10-17T18:54:24.821 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.821 DEBUG:boto:Data: 2023-10-17T18:54:24.822 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.823 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.823 DEBUG:boto:Params: {} 2023-10-17T18:54:24.824 DEBUG:boto:Token: None 2023-10-17T18:54:24.824 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4= 2023-10-17T18:54:24.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4=', 'Content-Length': '0'} 2023-10-17T18:54:24.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0c5bb52481c6f4-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.828 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.829 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.829 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.829 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.830 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.830 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.831 DEBUG:boto:Data: 2023-10-17T18:54:24.831 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.832 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.832 DEBUG:boto:Params: {} 2023-10-17T18:54:24.832 DEBUG:boto:Token: None 2023-10-17T18:54:24.833 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ai/VpsYv75y3Ib4BAi1/Rqrt+jw= 2023-10-17T18:54:24.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ai/VpsYv75y3Ib4BAi1/Rqrt+jw=', 'Content-Length': '0'} 2023-10-17T18:54:24.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e36813aa01c9f9-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.836 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.837 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.837 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.837 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.838 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.838 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.839 DEBUG:boto:Data: 2023-10-17T18:54:24.839 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.840 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.840 DEBUG:boto:Params: {} 2023-10-17T18:54:24.841 DEBUG:boto:Token: None 2023-10-17T18:54:24.841 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.841 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ai/VpsYv75y3Ib4BAi1/Rqrt+jw= 2023-10-17T18:54:24.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ai/VpsYv75y3Ib4BAi1/Rqrt+jw=', 'Content-Length': '0'} 2023-10-17T18:54:24.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8557fe4692b9f68-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-17T18:54:24.845 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.845 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.846 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.846 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.847 DEBUG:boto:Method: GET 2023-10-17T18:54:24.847 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.848 DEBUG:boto:Data: 2023-10-17T18:54:24.848 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.848 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.849 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.849 DEBUG:boto:Params: {} 2023-10-17T18:54:24.850 DEBUG:boto:Token: None 2023-10-17T18:54:24.850 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.851 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4= 2023-10-17T18:54:24.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4=', 'Content-Length': '0'} 2023-10-17T18:54:24.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a51be94102706656-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.853 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.854 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.854 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.855 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.855 DEBUG:boto:Method: GET 2023-10-17T18:54:24.855 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.856 DEBUG:boto:Data: 2023-10-17T18:54:24.856 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.857 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.857 DEBUG:boto:Params: {} 2023-10-17T18:54:24.858 DEBUG:boto:Token: None 2023-10-17T18:54:24.858 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271?versionId=null 2023-10-17T18:54:24.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4= 2023-10-17T18:54:24.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V23rOvu8/14Ul/+phfGPsFgiNi4=', 'Content-Length': '0'} 2023-10-17T18:54:24.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b035a05a610b3a-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.862 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.862 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.863 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.863 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.863 DEBUG:boto:Method: GET 2023-10-17T18:54:24.864 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.864 DEBUG:boto:Data: 2023-10-17T18:54:24.864 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.865 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.865 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.866 DEBUG:boto:Params: {} 2023-10-17T18:54:24.866 DEBUG:boto:Token: None 2023-10-17T18:54:24.867 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.867 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21032914291d665-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.871 DEBUG:boto:b'hbmuog-58obj000000000000000002711000falseobj000000000000000002712023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.871 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.872 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.872 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.872 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.873 DEBUG:boto:Method: GET 2023-10-17T18:54:24.873 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.873 DEBUG:boto:Data: 2023-10-17T18:54:24.874 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.874 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.875 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.875 DEBUG:boto:Params: {} 2023-10-17T18:54:24.875 DEBUG:boto:Token: None 2023-10-17T18:54:24.876 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.876 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504e35e73c3222bd-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.881 DEBUG:boto:b'hbmuog-58obj000000000000000002711000falseobj000000000000000002712023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.882 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.882 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.883 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.883 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.883 DEBUG:boto:Data: 2023-10-17T18:54:24.884 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.884 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.884 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.885 DEBUG:boto:Params: {} 2023-10-17T18:54:24.885 DEBUG:boto:Token: None 2023-10-17T18:54:24.886 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.886 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x3jQhh/tu08lYiwJteCGnktaPDU= 2023-10-17T18:54:24.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x3jQhh/tu08lYiwJteCGnktaPDU=', 'Content-Length': '0'} 2023-10-17T18:54:24.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081803e778f0b856b-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.889 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.889 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.890 DEBUG:boto:Method: HEAD 2023-10-17T18:54:24.890 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.891 DEBUG:boto:Data: 2023-10-17T18:54:24.891 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.892 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.892 DEBUG:boto:Params: {} 2023-10-17T18:54:24.893 DEBUG:boto:Token: None 2023-10-17T18:54:24.893 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.894 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x3jQhh/tu08lYiwJteCGnktaPDU= 2023-10-17T18:54:24.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x3jQhh/tu08lYiwJteCGnktaPDU=', 'Content-Length': '0'} 2023-10-17T18:54:24.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1e65bcb8d9e19f-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-17T18:54:24.897 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.898 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.898 DEBUG:boto:Method: GET 2023-10-17T18:54:24.898 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.899 DEBUG:boto:Data: 2023-10-17T18:54:24.899 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.900 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.900 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.900 DEBUG:boto:Params: {} 2023-10-17T18:54:24.901 DEBUG:boto:Token: None 2023-10-17T18:54:24.901 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.902 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7sIxbblNwHmepW5sd0AqYpbYP3s= 2023-10-17T18:54:24.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7sIxbblNwHmepW5sd0AqYpbYP3s=', 'Content-Length': '0'} 2023-10-17T18:54:24.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edae6a9074f68534-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.905 DEBUG:boto:path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.905 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.906 DEBUG:boto:Method: GET 2023-10-17T18:54:24.906 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.907 DEBUG:boto:Data: 2023-10-17T18:54:24.907 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.907 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.908 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.908 DEBUG:boto:Params: {} 2023-10-17T18:54:24.909 DEBUG:boto:Token: None 2023-10-17T18:54:24.909 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000271 2023-10-17T18:54:24.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7sIxbblNwHmepW5sd0AqYpbYP3s= 2023-10-17T18:54:24.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7sIxbblNwHmepW5sd0AqYpbYP3s=', 'Content-Length': '0'} 2023-10-17T18:54:24.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ede32503ffb3516-00652ed860-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.954 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.955 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.955 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.956 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.956 DEBUG:boto:Method: GET 2023-10-17T18:54:24.957 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.957 DEBUG:boto:Data: 2023-10-17T18:54:24.957 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.958 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.958 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.959 DEBUG:boto:Params: {} 2023-10-17T18:54:24.959 DEBUG:boto:Token: None 2023-10-17T18:54:24.960 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.960 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622991142542d8b6-00652ed860-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.963 DEBUG:boto:b'hbmuog-58obj000000000000000002711000falseobj000000000000000002712023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.964 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:24.964 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:24.965 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.965 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.966 DEBUG:boto:Method: GET 2023-10-17T18:54:24.966 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000271 2023-10-17T18:54:24.966 DEBUG:boto:Data: 2023-10-17T18:54:24.967 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.967 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:24.968 DEBUG:boto:Port: 8000 2023-10-17T18:54:24.968 DEBUG:boto:Params: {} 2023-10-17T18:54:24.969 DEBUG:boto:Token: None 2023-10-17T18:54:24.969 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/ 2023-10-17T18:54:24.970 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE= 2023-10-17T18:54:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u9r4ig1RRh3AWCDMp4h829zMDFE=', 'Content-Length': '0'} 2023-10-17T18:54:24.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052de58c23bb28b52-00652ed860-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:24.975 DEBUG:boto:b'hbmuog-58obj000000000000000002711000falseobj000000000000000002712023-10-17T18:54:06.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:24.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-17T18:54:24.976 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:24.977 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:24.977 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:24.977 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:24.978 DEBUG:boto:Method: GET 2023-10-17T18:54:24.978 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:24.979 DEBUG:boto:Data: 2023-10-17T18:54:24.980 DEBUG:boto:Headers: {} 2023-10-17T18:54:24.980 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:24.980 DEBUG:boto:Port: 8001 2023-10-17T18:54:24.981 DEBUG:boto:Params: {} 2023-10-17T18:54:24.981 DEBUG:boto:Token: None 2023-10-17T18:54:24.982 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:24 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:24.982 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WIusxU6+Kkm7KhXE6gakvD38agY= 2023-10-17T18:54:24.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WIusxU6+Kkm7KhXE6gakvD38agY=', 'Content-Length': '0'} 2023-10-17T18:54:24.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54cdb7f3c8c7883-00652ed860-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.026 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.027 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.027 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.028 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.028 DEBUG:boto:Method: GET 2023-10-17T18:54:25.029 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.029 DEBUG:boto:Data: 2023-10-17T18:54:25.030 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.031 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.031 DEBUG:boto:Params: {} 2023-10-17T18:54:25.032 DEBUG:boto:Token: None 2023-10-17T18:54:25.032 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc= 2023-10-17T18:54:25.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc=', 'Content-Length': '0'} 2023-10-17T18:54:25.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d905a27ea1865f-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.036 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.037 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.037 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.037 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.038 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.038 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.039 DEBUG:boto:Data: 2023-10-17T18:54:25.039 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.040 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.040 DEBUG:boto:Params: {} 2023-10-17T18:54:25.041 DEBUG:boto:Token: None 2023-10-17T18:54:25.041 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2x+p7J0kW7JWLrGUbHFH72Dz1pw= 2023-10-17T18:54:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2x+p7J0kW7JWLrGUbHFH72Dz1pw=', 'Content-Length': '0'} 2023-10-17T18:54:25.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007715d67f1914c01a-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.045 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.045 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.045 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.046 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.046 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.047 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.047 DEBUG:boto:Data: 2023-10-17T18:54:25.047 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.048 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.048 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.049 DEBUG:boto:Params: {} 2023-10-17T18:54:25.049 DEBUG:boto:Token: None 2023-10-17T18:54:25.049 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2x+p7J0kW7JWLrGUbHFH72Dz1pw= 2023-10-17T18:54:25.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2x+p7J0kW7JWLrGUbHFH72Dz1pw=', 'Content-Length': '0'} 2023-10-17T18:54:25.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f821b8d77b598572-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-17T18:54:25.053 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.054 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.054 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.054 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.055 DEBUG:boto:Method: GET 2023-10-17T18:54:25.055 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.056 DEBUG:boto:Data: 2023-10-17T18:54:25.056 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.057 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.057 DEBUG:boto:Params: {} 2023-10-17T18:54:25.058 DEBUG:boto:Token: None 2023-10-17T18:54:25.058 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc= 2023-10-17T18:54:25.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc=', 'Content-Length': '0'} 2023-10-17T18:54:25.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3f49dd6d1800bd-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.062 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.063 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.063 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.063 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.064 DEBUG:boto:Method: GET 2023-10-17T18:54:25.064 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.065 DEBUG:boto:Data: 2023-10-17T18:54:25.065 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.067 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.067 DEBUG:boto:Params: {} 2023-10-17T18:54:25.068 DEBUG:boto:Token: None 2023-10-17T18:54:25.069 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283?versionId=null 2023-10-17T18:54:25.070 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc= 2023-10-17T18:54:25.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:m8fOx1w6wSLvYZIV2mRdsuYszsc=', 'Content-Length': '0'} 2023-10-17T18:54:25.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8c8927f6d27dd0-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.074 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.075 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.076 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.077 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.077 DEBUG:boto:Method: GET 2023-10-17T18:54:25.078 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.079 DEBUG:boto:Data: 2023-10-17T18:54:25.080 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.080 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.081 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.082 DEBUG:boto:Params: {} 2023-10-17T18:54:25.083 DEBUG:boto:Token: None 2023-10-17T18:54:25.083 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f4a084dcdcea30-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.089 DEBUG:boto:b'hbmuog-58obj000000000000000002831000falseobj000000000000000002832023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.090 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.091 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.092 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.092 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.093 DEBUG:boto:Method: GET 2023-10-17T18:54:25.094 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.095 DEBUG:boto:Data: 2023-10-17T18:54:25.095 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.097 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.097 DEBUG:boto:Params: {} 2023-10-17T18:54:25.098 DEBUG:boto:Token: None 2023-10-17T18:54:25.099 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037fa44d3bf911dff-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.105 DEBUG:boto:b'hbmuog-58obj000000000000000002831000falseobj000000000000000002832023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.106 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.106 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.107 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.107 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.107 DEBUG:boto:Data: 2023-10-17T18:54:25.108 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.109 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.109 DEBUG:boto:Params: {} 2023-10-17T18:54:25.109 DEBUG:boto:Token: None 2023-10-17T18:54:25.110 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.110 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:old5dPK/FznCBXjZKh1yllwkeOs= 2023-10-17T18:54:25.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:old5dPK/FznCBXjZKh1yllwkeOs=', 'Content-Length': '0'} 2023-10-17T18:54:25.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d215fe88fb9f1695-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.114 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.114 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.114 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.115 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.115 DEBUG:boto:Data: 2023-10-17T18:54:25.115 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.116 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.116 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.117 DEBUG:boto:Params: {} 2023-10-17T18:54:25.117 DEBUG:boto:Token: None 2023-10-17T18:54:25.117 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.118 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:old5dPK/FznCBXjZKh1yllwkeOs= 2023-10-17T18:54:25.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:old5dPK/FznCBXjZKh1yllwkeOs=', 'Content-Length': '0'} 2023-10-17T18:54:25.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd26fdd7212e7c99-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-17T18:54:25.121 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.121 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.122 DEBUG:boto:Method: GET 2023-10-17T18:54:25.122 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.123 DEBUG:boto:Data: 2023-10-17T18:54:25.123 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.124 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.124 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.125 DEBUG:boto:Params: {} 2023-10-17T18:54:25.125 DEBUG:boto:Token: None 2023-10-17T18:54:25.126 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.126 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nvncB+HheQC7muIwyJ9GZjtC44w= 2023-10-17T18:54:25.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvncB+HheQC7muIwyJ9GZjtC44w=', 'Content-Length': '0'} 2023-10-17T18:54:25.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174727e5a3fc9403-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.129 DEBUG:boto:path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.129 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.130 DEBUG:boto:Method: GET 2023-10-17T18:54:25.130 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.131 DEBUG:boto:Data: 2023-10-17T18:54:25.131 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.132 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.132 DEBUG:boto:Params: {} 2023-10-17T18:54:25.133 DEBUG:boto:Token: None 2023-10-17T18:54:25.133 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000283 2023-10-17T18:54:25.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nvncB+HheQC7muIwyJ9GZjtC44w= 2023-10-17T18:54:25.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvncB+HheQC7muIwyJ9GZjtC44w=', 'Content-Length': '0'} 2023-10-17T18:54:25.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002db344b7f2004e7-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.178 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.179 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.180 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.180 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.181 DEBUG:boto:Method: GET 2023-10-17T18:54:25.181 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.182 DEBUG:boto:Data: 2023-10-17T18:54:25.182 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.183 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.183 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.184 DEBUG:boto:Params: {} 2023-10-17T18:54:25.185 DEBUG:boto:Token: None 2023-10-17T18:54:25.185 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.186 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002669c24db9007d99-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.191 DEBUG:boto:b'hbmuog-58obj000000000000000002831000falseobj000000000000000002832023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.192 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.192 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.193 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.193 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.193 DEBUG:boto:Method: GET 2023-10-17T18:54:25.194 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000283 2023-10-17T18:54:25.194 DEBUG:boto:Data: 2023-10-17T18:54:25.195 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.195 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.196 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.196 DEBUG:boto:Params: {} 2023-10-17T18:54:25.196 DEBUG:boto:Token: None 2023-10-17T18:54:25.197 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.197 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005658f1c8798c4d-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.202 DEBUG:boto:b'hbmuog-58obj000000000000000002831000falseobj000000000000000002832023-10-17T18:54:06.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-17T18:54:25.203 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.204 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.204 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.205 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.205 DEBUG:boto:Method: GET 2023-10-17T18:54:25.206 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.206 DEBUG:boto:Data: 2023-10-17T18:54:25.207 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.207 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.208 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.209 DEBUG:boto:Params: {} 2023-10-17T18:54:25.210 DEBUG:boto:Token: None 2023-10-17T18:54:25.211 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI= 2023-10-17T18:54:25.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI=', 'Content-Length': '0'} 2023-10-17T18:54:25.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d73d8df2b792a18-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.215 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.215 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.215 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.216 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.216 DEBUG:boto:Method: GET 2023-10-17T18:54:25.217 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.217 DEBUG:boto:Data: 2023-10-17T18:54:25.217 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.218 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.219 DEBUG:boto:Params: {} 2023-10-17T18:54:25.219 DEBUG:boto:Token: None 2023-10-17T18:54:25.220 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI= 2023-10-17T18:54:25.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI=', 'Content-Length': '0'} 2023-10-17T18:54:25.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19bcd0330f522f2-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.223 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.224 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.224 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.224 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.225 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.225 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.226 DEBUG:boto:Data: 2023-10-17T18:54:25.226 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.226 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.227 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.227 DEBUG:boto:Params: {} 2023-10-17T18:54:25.228 DEBUG:boto:Token: None 2023-10-17T18:54:25.228 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.228 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FmLtxeojblpods+4Tqfi94SINoo= 2023-10-17T18:54:25.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FmLtxeojblpods+4Tqfi94SINoo=', 'Content-Length': '0'} 2023-10-17T18:54:25.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001451b02aacc30246-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.231 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.232 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.232 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.232 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.233 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.233 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.234 DEBUG:boto:Data: 2023-10-17T18:54:25.234 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.234 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.235 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.235 DEBUG:boto:Params: {} 2023-10-17T18:54:25.236 DEBUG:boto:Token: None 2023-10-17T18:54:25.236 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.237 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FmLtxeojblpods+4Tqfi94SINoo= 2023-10-17T18:54:25.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FmLtxeojblpods+4Tqfi94SINoo=', 'Content-Length': '0'} 2023-10-17T18:54:25.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2d3ebba0302003-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-17T18:54:25.240 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.240 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.241 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.241 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.241 DEBUG:boto:Method: GET 2023-10-17T18:54:25.242 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.242 DEBUG:boto:Data: 2023-10-17T18:54:25.243 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.243 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.243 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.244 DEBUG:boto:Params: {} 2023-10-17T18:54:25.244 DEBUG:boto:Token: None 2023-10-17T18:54:25.245 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI= 2023-10-17T18:54:25.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI=', 'Content-Length': '0'} 2023-10-17T18:54:25.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087951ebf7fd93878-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.290 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.291 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.291 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.292 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.292 DEBUG:boto:Method: GET 2023-10-17T18:54:25.292 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.293 DEBUG:boto:Data: 2023-10-17T18:54:25.293 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.294 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.294 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.295 DEBUG:boto:Params: {} 2023-10-17T18:54:25.295 DEBUG:boto:Token: None 2023-10-17T18:54:25.295 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423?versionId=null 2023-10-17T18:54:25.296 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI= 2023-10-17T18:54:25.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1cAwJpeB4YQXGVcnqemhbntz2iI=', 'Content-Length': '0'} 2023-10-17T18:54:25.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee3023a77c5f91c-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.299 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.299 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.300 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.300 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.301 DEBUG:boto:Method: GET 2023-10-17T18:54:25.301 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.301 DEBUG:boto:Data: 2023-10-17T18:54:25.302 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.303 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.303 DEBUG:boto:Params: {} 2023-10-17T18:54:25.303 DEBUG:boto:Token: None 2023-10-17T18:54:25.304 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8d474e1a53c3a1b-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.308 DEBUG:boto:b'hbmuog-58obj000000000000000004231000falseobj000000000000000004232023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.309 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.310 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.311 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.311 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.312 DEBUG:boto:Method: GET 2023-10-17T18:54:25.313 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.313 DEBUG:boto:Data: 2023-10-17T18:54:25.314 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.314 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.315 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.315 DEBUG:boto:Params: {} 2023-10-17T18:54:25.315 DEBUG:boto:Token: None 2023-10-17T18:54:25.316 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733c561caf83382e-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.362 DEBUG:boto:b'hbmuog-58obj000000000000000004231000falseobj000000000000000004232023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.363 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.363 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.364 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.364 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.364 DEBUG:boto:Data: 2023-10-17T18:54:25.365 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.365 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.366 DEBUG:boto:Params: {} 2023-10-17T18:54:25.366 DEBUG:boto:Token: None 2023-10-17T18:54:25.367 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PCv1wWHA0lHclEO2y70kxDWkK3k= 2023-10-17T18:54:25.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PCv1wWHA0lHclEO2y70kxDWkK3k=', 'Content-Length': '0'} 2023-10-17T18:54:25.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141524759fc500b6-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.370 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.370 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.371 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.371 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.371 DEBUG:boto:Data: 2023-10-17T18:54:25.372 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.373 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.373 DEBUG:boto:Params: {} 2023-10-17T18:54:25.373 DEBUG:boto:Token: None 2023-10-17T18:54:25.374 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PCv1wWHA0lHclEO2y70kxDWkK3k= 2023-10-17T18:54:25.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PCv1wWHA0lHclEO2y70kxDWkK3k=', 'Content-Length': '0'} 2023-10-17T18:54:25.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ba76e4febdd2ab-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-17T18:54:25.378 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.378 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.378 DEBUG:boto:Method: GET 2023-10-17T18:54:25.379 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.379 DEBUG:boto:Data: 2023-10-17T18:54:25.380 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.380 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.381 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.381 DEBUG:boto:Params: {} 2023-10-17T18:54:25.382 DEBUG:boto:Token: None 2023-10-17T18:54:25.382 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B9e5CwONgouww9T6yZnE25Sud80= 2023-10-17T18:54:25.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B9e5CwONgouww9T6yZnE25Sud80=', 'Content-Length': '0'} 2023-10-17T18:54:25.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5677c40c481a2fe-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.386 DEBUG:boto:path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.386 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.387 DEBUG:boto:Method: GET 2023-10-17T18:54:25.387 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.388 DEBUG:boto:Data: 2023-10-17T18:54:25.388 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.389 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.389 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.389 DEBUG:boto:Params: {} 2023-10-17T18:54:25.390 DEBUG:boto:Token: None 2023-10-17T18:54:25.390 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000423 2023-10-17T18:54:25.391 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B9e5CwONgouww9T6yZnE25Sud80= 2023-10-17T18:54:25.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B9e5CwONgouww9T6yZnE25Sud80=', 'Content-Length': '0'} 2023-10-17T18:54:25.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eab2f6776bf121e-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.394 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.395 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.395 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.396 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.396 DEBUG:boto:Method: GET 2023-10-17T18:54:25.396 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.397 DEBUG:boto:Data: 2023-10-17T18:54:25.397 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.398 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.398 DEBUG:boto:Params: {} 2023-10-17T18:54:25.399 DEBUG:boto:Token: None 2023-10-17T18:54:25.399 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba556ca993555b0-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.403 DEBUG:boto:b'hbmuog-58obj000000000000000004231000falseobj000000000000000004232023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.404 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.404 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.405 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.405 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.406 DEBUG:boto:Method: GET 2023-10-17T18:54:25.407 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000423 2023-10-17T18:54:25.407 DEBUG:boto:Data: 2023-10-17T18:54:25.407 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.409 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.409 DEBUG:boto:Params: {} 2023-10-17T18:54:25.410 DEBUG:boto:Token: None 2023-10-17T18:54:25.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d21e9cf2674df24-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.418 DEBUG:boto:b'hbmuog-58obj000000000000000004231000falseobj000000000000000004232023-10-17T18:54:06.562Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-17T18:54:25.419 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.419 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.419 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.420 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.420 DEBUG:boto:Method: GET 2023-10-17T18:54:25.421 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.421 DEBUG:boto:Data: 2023-10-17T18:54:25.421 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.422 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.423 DEBUG:boto:Params: {} 2023-10-17T18:54:25.423 DEBUG:boto:Token: None 2023-10-17T18:54:25.423 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.424 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8= 2023-10-17T18:54:25.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8=', 'Content-Length': '0'} 2023-10-17T18:54:25.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005383ffbf30080678-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.427 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.427 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.428 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.428 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.429 DEBUG:boto:Method: GET 2023-10-17T18:54:25.429 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.429 DEBUG:boto:Data: 2023-10-17T18:54:25.430 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.430 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.431 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.431 DEBUG:boto:Params: {} 2023-10-17T18:54:25.431 DEBUG:boto:Token: None 2023-10-17T18:54:25.432 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.432 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8= 2023-10-17T18:54:25.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8=', 'Content-Length': '0'} 2023-10-17T18:54:25.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ea97fa3595da90-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.436 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.436 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.436 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.437 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.437 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.437 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.438 DEBUG:boto:Data: 2023-10-17T18:54:25.438 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.439 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.439 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.439 DEBUG:boto:Params: {} 2023-10-17T18:54:25.440 DEBUG:boto:Token: None 2023-10-17T18:54:25.440 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.441 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2PhzROF7OIcii1itI92GQ8QC8h8= 2023-10-17T18:54:25.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2PhzROF7OIcii1itI92GQ8QC8h8=', 'Content-Length': '0'} 2023-10-17T18:54:25.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c2b94fc71fe22e-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.444 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.444 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.445 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.445 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.445 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.446 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.446 DEBUG:boto:Data: 2023-10-17T18:54:25.446 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.447 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.447 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.447 DEBUG:boto:Params: {} 2023-10-17T18:54:25.448 DEBUG:boto:Token: None 2023-10-17T18:54:25.448 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2PhzROF7OIcii1itI92GQ8QC8h8= 2023-10-17T18:54:25.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2PhzROF7OIcii1itI92GQ8QC8h8=', 'Content-Length': '0'} 2023-10-17T18:54:25.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeaa19da0868c75d-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-17T18:54:25.452 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.452 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.453 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.453 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.453 DEBUG:boto:Method: GET 2023-10-17T18:54:25.454 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.454 DEBUG:boto:Data: 2023-10-17T18:54:25.455 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.455 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.456 DEBUG:boto:Params: {} 2023-10-17T18:54:25.456 DEBUG:boto:Token: None 2023-10-17T18:54:25.457 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8= 2023-10-17T18:54:25.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8=', 'Content-Length': '0'} 2023-10-17T18:54:25.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5446326c4841356-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.506 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.507 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.507 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.508 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.508 DEBUG:boto:Method: GET 2023-10-17T18:54:25.509 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.509 DEBUG:boto:Data: 2023-10-17T18:54:25.509 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.510 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.510 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.511 DEBUG:boto:Params: {} 2023-10-17T18:54:25.512 DEBUG:boto:Token: None 2023-10-17T18:54:25.512 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467?versionId=null 2023-10-17T18:54:25.513 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8= 2023-10-17T18:54:25.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dLCAQ+6YV/yhr413z+zpQhwxLp8=', 'Content-Length': '0'} 2023-10-17T18:54:25.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502fcfc22523acec-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.517 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.518 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.518 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.519 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.519 DEBUG:boto:Method: GET 2023-10-17T18:54:25.519 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.520 DEBUG:boto:Data: 2023-10-17T18:54:25.520 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.521 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.522 DEBUG:boto:Params: {} 2023-10-17T18:54:25.522 DEBUG:boto:Token: None 2023-10-17T18:54:25.523 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.523 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f428113b1480480-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.526 DEBUG:boto:b'hbmuog-58obj000000000000000004671000falseobj000000000000000004672023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.527 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.527 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.528 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.528 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.529 DEBUG:boto:Method: GET 2023-10-17T18:54:25.529 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.529 DEBUG:boto:Data: 2023-10-17T18:54:25.530 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.531 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.531 DEBUG:boto:Params: {} 2023-10-17T18:54:25.533 DEBUG:boto:Token: None 2023-10-17T18:54:25.533 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.534 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db02be61b64b8c4-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.539 DEBUG:boto:b'hbmuog-58obj000000000000000004671000falseobj000000000000000004672023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.540 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.540 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.541 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.541 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.541 DEBUG:boto:Data: 2023-10-17T18:54:25.542 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.542 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.543 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.543 DEBUG:boto:Params: {} 2023-10-17T18:54:25.543 DEBUG:boto:Token: None 2023-10-17T18:54:25.544 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.544 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iygqIfzynbcGS7L5kxIieJB/L+A= 2023-10-17T18:54:25.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iygqIfzynbcGS7L5kxIieJB/L+A=', 'Content-Length': '0'} 2023-10-17T18:54:25.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a8bf559cb27444-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.548 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.548 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.548 DEBUG:boto:Method: HEAD 2023-10-17T18:54:25.549 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.549 DEBUG:boto:Data: 2023-10-17T18:54:25.550 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.550 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.551 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.551 DEBUG:boto:Params: {} 2023-10-17T18:54:25.551 DEBUG:boto:Token: None 2023-10-17T18:54:25.552 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iygqIfzynbcGS7L5kxIieJB/L+A= 2023-10-17T18:54:25.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iygqIfzynbcGS7L5kxIieJB/L+A=', 'Content-Length': '0'} 2023-10-17T18:54:25.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d63181333f85686-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-17T18:54:25.556 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.556 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.557 DEBUG:boto:Method: GET 2023-10-17T18:54:25.557 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.557 DEBUG:boto:Data: 2023-10-17T18:54:25.558 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.560 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.560 DEBUG:boto:Params: {} 2023-10-17T18:54:25.561 DEBUG:boto:Token: None 2023-10-17T18:54:25.562 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aeTIuC0aVtOSEm2A4wulqYRo6tM= 2023-10-17T18:54:25.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aeTIuC0aVtOSEm2A4wulqYRo6tM=', 'Content-Length': '0'} 2023-10-17T18:54:25.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-58:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001594dce16a0f9578-00652ed861-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.567 DEBUG:boto:path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.568 DEBUG:boto:auth_path=/hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.569 DEBUG:boto:Method: GET 2023-10-17T18:54:25.570 DEBUG:boto:Path: /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.570 DEBUG:boto:Data: 2023-10-17T18:54:25.571 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.572 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.573 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.574 DEBUG:boto:Params: {} 2023-10-17T18:54:25.575 DEBUG:boto:Token: None 2023-10-17T18:54:25.575 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/obj00000000000000000467 2023-10-17T18:54:25.576 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aeTIuC0aVtOSEm2A4wulqYRo6tM= 2023-10-17T18:54:25.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aeTIuC0aVtOSEm2A4wulqYRo6tM=', 'Content-Length': '0'} 2023-10-17T18:54:25.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:54:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf661b80b39a19de-00652ed861-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.581 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.582 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.582 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.583 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.584 DEBUG:boto:Method: GET 2023-10-17T18:54:25.585 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.585 DEBUG:boto:Data: 2023-10-17T18:54:25.586 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:54:25.588 DEBUG:boto:Port: 8001 2023-10-17T18:54:25.588 DEBUG:boto:Params: {} 2023-10-17T18:54:25.589 DEBUG:boto:Token: None 2023-10-17T18:54:25.590 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103ca5daba6712f6-00652ed861-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.638 DEBUG:boto:b'hbmuog-58obj000000000000000004671000falseobj000000000000000004672023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.639 DEBUG:boto:path=/hbmuog-58/ 2023-10-17T18:54:25.639 DEBUG:boto:auth_path=/hbmuog-58/ 2023-10-17T18:54:25.640 DEBUG:boto:path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.640 DEBUG:boto:auth_path=/hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.641 DEBUG:boto:Method: GET 2023-10-17T18:54:25.641 DEBUG:boto:Path: /hbmuog-58/?prefix=obj00000000000000000467 2023-10-17T18:54:25.641 DEBUG:boto:Data: 2023-10-17T18:54:25.642 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.643 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.643 DEBUG:boto:Params: {} 2023-10-17T18:54:25.644 DEBUG:boto:Token: None 2023-10-17T18:54:25.644 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-58/ 2023-10-17T18:54:25.645 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU= 2023-10-17T18:54:25.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QxRjLTt4/WlmpuVu8DAGNmLadMU=', 'Content-Length': '0'} 2023-10-17T18:54:25.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054435b68d3408461-00652ed861-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.650 DEBUG:boto:b'hbmuog-58obj000000000000000004671000falseobj000000000000000004672023-10-17T18:54:06.940Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:54:25.651 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-58 zones={a2, a1} 2023-10-17T18:54:25.652 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-10-17T18:54:25.654 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:54:25.655 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:54:25.655 DEBUG:boto:Method: PUT 2023-10-17T18:54:25.655 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:54:25.656 DEBUG:boto:Data: 2023-10-17T18:54:25.656 DEBUG:boto:Headers: {} 2023-10-17T18:54:25.656 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:54:25.657 DEBUG:boto:Port: 8000 2023-10-17T18:54:25.657 DEBUG:boto:Params: {} 2023-10-17T18:54:25.658 DEBUG:boto:Token: None 2023-10-17T18:54:25.658 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 18:54:25 GMT /hbmuog-59/ 2023-10-17T18:54:25.659 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:th/h0vYaiC9QEIukF9vy6Kkfub0= 2023-10-17T18:54:25.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:th/h0vYaiC9QEIukF9vy6Kkfub0=', 'Content-Length': '0'} 2023-10-17T18:54:25.679 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054eed05f41aa77d2-00652ed861-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:54:25.679 DEBUG:rgw_multi.tests:created bucket=hbmuog-59 2023-10-17T18:54:25.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:25.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:25.681 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:25.737 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:25.737 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:25.773 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T18:54:25.787 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T18:54:25.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.787 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568865.677043_735.1", 2023-10-17T18:54:25.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:54:25.677043Z" 2023-10-17T18:54:25.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.788 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:25.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T18:54:25.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.789 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:25.789 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:54:25.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.790 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:25.790 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T18:54:25.790 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.791 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T18:54:25.792 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:25.792 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T18:54:25.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:25.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:25.794 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:25.847 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:25.847 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:25.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:25.879+0000 7fd67ba8b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:25.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:25.880+0000 7fd67ba8b780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:25.893 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:25.903 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:25.904 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:25.905 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:25.906 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:25.907 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.908 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:25.909 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:25Z" 2023-10-17T18:54:25.909 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:25.910 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:25Z" } 2023-10-17T18:54:25.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:25.912 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:25.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:25.913 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:30.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:30.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:30.915 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:30.974 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:30.975 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:31.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:31.007+0000 7f7977856780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:31.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:31.008+0000 7f7977856780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:31.022 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:31.030 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:31.031 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:31.032 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:31.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:31.035 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:31Z" 2023-10-17T18:54:31.035 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:31.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:31Z" } 2023-10-17T18:54:31.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:31.037 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:31.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:31.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:36.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:36.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:36.040 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:36.099 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:36.099 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:36.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:36.132+0000 7f6c858af780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:36.133+0000 7f6c858af780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:36.147 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:36Z" 2023-10-17T18:54:36.163 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:36.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:36Z" } 2023-10-17T18:54:36.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:36.165 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:36.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:36.166 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:41.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:41.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:41.169 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:41.228 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:41.228 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:41.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:41.262+0000 7f998ad9d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:41.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:41.263+0000 7f998ad9d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:41Z" 2023-10-17T18:54:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:41.294 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:41Z" } 2023-10-17T18:54:41.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:41.295 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:41.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:41.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:46.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:46.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:46.298 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:46.357 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:46.357 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:46.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:46.389+0000 7f076fc01780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:46.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:46.390+0000 7f076fc01780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:46.405 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:46.413 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:46.413 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:46.413 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:46.413 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:46.416 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:46.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:46.416 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:46.416 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:46.416 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:46.417 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:46.417 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:46.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:46.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:46.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:46.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.418 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:46.418 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:46.418 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:46.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:46.419 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:46Z" 2023-10-17T18:54:46.419 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:46.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:46Z" } 2023-10-17T18:54:46.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:46.421 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:46.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:46.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:51.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:51.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:51.424 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:51.484 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:51.485 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:51.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:51.517+0000 7fe7e2081780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:51.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:51.518+0000 7fe7e2081780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:51.533 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:51.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:51.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:51.540 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:51.541 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:51.541 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:51.541 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:51.541 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:51.541 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:51.542 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:51.542 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:51.542 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:51.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:51.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:51.543 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:51.543 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:51.543 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:51.544 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:51.544 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:51.544 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:51.544 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:51.547 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:51Z" 2023-10-17T18:54:51.547 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:51.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:51Z" } 2023-10-17T18:54:51.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:51.549 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:51.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:51.550 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:54:56.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:56.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:54:56.552 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:54:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:54:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:54:56.644 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:56.643+0000 7f9487f24780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:56.645 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:54:56.644+0000 7f9487f24780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:54:56.659 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:54:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:54:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:54:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:54:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:54:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:54:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:54:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:54:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:54:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:54:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:54:56.664 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:56.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:56.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:54:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:54:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:54:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:54:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:56.670 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:56.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:56.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:54:56.671 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:54:56.671 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:54:56.671 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:54:56.672 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:54:56.672 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:54:56.672 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:54:56.672 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:54:56.673 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:54:56.673 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:54:56.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:56.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:54:56.674 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:54:56.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.674 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:54:56.674 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:54:56.675 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:54:56.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:54:56.675 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:54:56Z" 2023-10-17T18:54:56.675 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:54:56.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:54:56Z" } 2023-10-17T18:54:56.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:54:56.677 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:56.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:54:56.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:01.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:01.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:01.680 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:01.738 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:01.739 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:01.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:01.771+0000 7fab4a045780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:01.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:01.772+0000 7fab4a045780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:01.786 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:01Z" 2023-10-17T18:55:01.798 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:01.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:01Z" } 2023-10-17T18:55:01.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:01.800 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:01.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:01.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:06.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:06.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:06.803 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:06.860 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:06.860 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:06.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:06.893+0000 7fa4a2dd6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:06.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:06.894+0000 7fa4a2dd6780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:06.908 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:06.908 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:06.908 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:06.908 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:06.909 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:06.909 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:06.909 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:06.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.910 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:06.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:06.910 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:06.910 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:06.911 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:06.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:06.913 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:06.914 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:06.915 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:06.916 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:06.917 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:06.919 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:06.920 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:06.921 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:06.922 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:06Z" 2023-10-17T18:55:06.922 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:06.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:06Z" } 2023-10-17T18:55:06.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:06.924 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:06.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:06.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:11.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:11.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:11.927 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:11.986 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:11.986 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:12.020+0000 7fe1543e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:12.021+0000 7fe1543e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:12.035 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:12.035 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:12.035 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:12.035 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:12.036 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:12.036 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:12.036 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:12.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.036 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:12.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:12.037 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:12.037 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:12.037 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:12.037 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:12.038 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:12.038 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:12.038 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:12.038 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:12.038 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:12.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:12.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:12.039 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:12.039 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:12.040 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:12.040 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:12.040 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:12.040 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:12.040 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:12.041 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:12.042 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:12.042 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:12.042 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:12.042 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:12.043 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:12.043 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:12.043 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:12.043 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:12.043 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:12.044 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:12.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:12.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:12.046 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:12.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.047 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:12.047 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:12.047 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:12.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:12.047 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:12Z" 2023-10-17T18:55:12.048 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:12.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:12Z" } 2023-10-17T18:55:12.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:12.049 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:12.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:12.050 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:17.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:17.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:17.052 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:17.110 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:17.143+0000 7fba7639f780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:17.144+0000 7fba7639f780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:17.159 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:17.159 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:17.160 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:17.160 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:17.160 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:17.160 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:17.160 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:17.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.161 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:17.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:17.161 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:17.161 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:17.162 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:17.162 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:17.162 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:17.162 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:17.162 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:17.163 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:17.163 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:17.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:17.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:17.164 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:17.164 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:17.164 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:17.164 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:17.164 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:17.165 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:17.165 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:17.165 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:17.165 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:17.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:17.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:17.166 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:17.166 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:17.166 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:17.167 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:17.167 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:17.167 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:17.167 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:17.168 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:17.168 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:17.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:17.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:17.169 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:17.170 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:17.170 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:17.170 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:17.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:17.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:17.171 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:17.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.171 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:17.171 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:17.171 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:17.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:17.172 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:17Z" 2023-10-17T18:55:17.172 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:17.173 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:17Z" } 2023-10-17T18:55:17.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:17.174 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:17.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:17.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:22.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:22.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:22.177 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:22.235 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:22.235 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:22.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:22.267+0000 7fcbf240e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:22.270 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:22.269+0000 7fcbf240e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:22.284 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:22.284 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:22.285 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:22.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:22.287 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:22.287 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:22.287 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:22.287 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:22.288 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:22.288 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:22.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:22.288 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:22.289 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:22.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:22.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:22.290 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:22.290 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:22.290 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:22.290 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:22.290 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:22.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:22.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:22.291 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:22.291 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:22.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:22.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:22.292 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:22.293 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:22.293 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:22.293 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:22.293 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:22.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:22.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:22.294 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:22.294 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:22.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:22.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:22.295 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:22.295 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:22.296 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:22.296 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:22.296 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:22.296 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:22.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:22.297 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:22.297 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:22.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:22.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:22.297 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:22.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.298 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:22.298 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:22.298 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:22.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:22.299 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:22Z" 2023-10-17T18:55:22.299 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:22.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:22Z" } 2023-10-17T18:55:22.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:22.301 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:22.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:22.302 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:27.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:27.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:27.305 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:27.361 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:27.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:27.394 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:27.393+0000 7f009765d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:27.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:27.394+0000 7f009765d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:27.409 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:27.409 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:27.409 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:27.410 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:27.410 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:27.410 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:27.410 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:27.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.411 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:27.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:27.411 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:27.411 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:27.412 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:27.412 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:27.412 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:27.412 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:27.413 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:27.413 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:27.413 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:27.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:27.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:27.414 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:27.414 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:27.415 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:27.415 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:27.415 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:27.415 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:27.415 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:27.416 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:27.416 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:27.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:27.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:27.417 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:27.417 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:27.417 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:27.417 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:27.418 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:27.418 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:27.418 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:27.418 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:27.419 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:27.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:27.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:27.419 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:27.420 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:27.420 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:27.420 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:27.420 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:27.420 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:27.421 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:27.421 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:27.421 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:27.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:27.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:27.422 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:27.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.422 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:27.422 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:27.423 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:27.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:27.423 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:27Z" 2023-10-17T18:55:27.423 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:27.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:27Z" } 2023-10-17T18:55:27.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:27.425 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:27.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:27.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:32.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:32.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:32.429 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:32.487 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:32.488 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:32.521+0000 7f33c28cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:32.523+0000 7f33c28cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:32.539 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:32Z" 2023-10-17T18:55:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:32.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:32Z" } 2023-10-17T18:55:32.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:32.555 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:32.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:32.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:37.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:37.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:37.558 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:37.617 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:37.618 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:37.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:37.650+0000 7f8773bd9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:37.652 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:37.651+0000 7f8773bd9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:37.666 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:37.666 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:37.667 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:37.667 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:37.667 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:37.667 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:37.668 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:37.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.668 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:37.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:37.669 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:37.669 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:37.669 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:37.669 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:37.670 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:37.670 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:37.670 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:37.670 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:37.670 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:37.672 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:37.672 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:37.672 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:37.672 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:37.673 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:37.673 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:37.673 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:37.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:37.676 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:37.676 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:37.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:37.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:37.677 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:37.677 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:37.677 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:37.677 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:37.677 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:37.678 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:37.678 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:37.678 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:37.678 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:37.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:37.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:37.679 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:37.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.679 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:37Z" 2023-10-17T18:55:37.681 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:37.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:37Z" } 2023-10-17T18:55:37.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:37.683 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:37.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:37.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:42.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:42.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:42.686 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:42.742 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:42.743 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:42.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:42.775+0000 7f14986ce780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:42.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:42.776+0000 7f14986ce780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:42.791 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:42.791 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:42.791 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:42.792 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:42.792 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:42.792 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:42.792 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:42.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.793 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:42.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:42.793 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:42.793 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:42.794 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:42.794 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:42.794 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:42.794 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:42.794 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:42.795 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:42.795 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:42.795 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:42.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:42.796 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:42.796 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:42.796 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:42.796 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:42.797 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:42.797 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:42.797 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:42.797 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:42.797 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:42.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:42.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:42.798 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:42.799 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:42.799 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:42.799 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:42.799 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:42.800 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:42.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:42.800 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:42.800 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:42.800 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:42.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:42.801 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:42.801 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:42.802 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:42.802 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:42.802 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:42.802 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:42.803 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:42.803 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:42.803 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:42.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:42.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:42.804 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:42.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.804 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:42.804 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:42.805 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:42.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:42.805 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:42Z" 2023-10-17T18:55:42.805 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:42.806 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:42Z" } 2023-10-17T18:55:42.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:42.807 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:42.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:42.808 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:47.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:47.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:47.810 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:47.869 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:47.869 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:47.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:47.906+0000 7fccf348d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:47.908 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:47.908+0000 7fccf348d780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:47.922 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:47.922 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:47.922 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:47.922 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:47.923 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:47.923 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:47.923 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:47.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.924 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:47.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:47.924 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:47.924 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:47.924 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:47.925 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:47.925 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:47.925 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:47.925 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:47.925 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:47.926 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:47.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:47.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:47.927 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:47.927 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:47.927 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:47.927 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:47.927 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:47.928 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:47.928 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:47.928 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:47.928 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:47.929 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:47.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:47.929 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:47.929 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:47.930 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:47.930 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:47.930 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:47.930 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:47.931 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:47.931 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:47.931 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:47.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:47.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:47.932 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:47.932 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:47.932 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:47.933 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:47.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:47.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:47.936 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:47Z" 2023-10-17T18:55:47.936 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:47.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:47Z" } 2023-10-17T18:55:47.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:47.938 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:47.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:47.938 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:52.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:52.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:52.941 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:52.999 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:52.999 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:53.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:53.032+0000 7f75958dc780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:53.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:53.033+0000 7f75958dc780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:53.048 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:53.048 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:53.049 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:53.049 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:53.049 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:53.049 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:53.050 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:53.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.050 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:53.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:53.051 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:53.051 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:53.051 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:53.051 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568816.812003_700.1", 2023-10-17T18:55:53.051 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:53.052 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:53.052 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:53.052 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T18:55:53.052 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:53.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:53.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:53.053 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:53.053 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:53.055 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:53.055 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:53.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:53.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:53.056 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:53.056 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:53.056 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:53.056 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:53.056 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:53.057 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:53.057 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:53.057 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:53.057 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:53.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:53.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:53.058 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:53.058 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:53.059 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:53.059 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:53.059 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:53.059 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:53.059 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:53.060 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:53.060 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:53.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:53.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:53.060 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:53.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.061 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:53.061 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:53.061 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:53.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:53.062 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:53Z" 2023-10-17T18:55:53.062 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:53.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568816.812003_700.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:53:36.812003Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:53Z" } 2023-10-17T18:55:53.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:53.065 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:53.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:53.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T18:55:58.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:58.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:55:58.068 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:55:58.127 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:55:58.127 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:55:58.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:58.160+0000 7f29fb1e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:58.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:55:58.161+0000 7f29fb1e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:55:58.177 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:55:58.177 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:55:58.177 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:55:58.177 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:55:58.177 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:55:58.178 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:55:58.178 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:58.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.178 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:55:58.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:58.179 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:55:58.179 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:58.179 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:58.179 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568841.365181_703.1", 2023-10-17T18:55:58.179 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:54:01.365181Z", 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:58.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:58.181 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:55:58.181 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:58.181 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:58.181 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:55:58.182 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:58.182 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:58.182 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:58.182 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:55:58.182 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:58.183 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:58.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:58.183 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:55:58.183 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:58.184 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:58.185 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:58.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:58.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:55:58.185 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:55:58.186 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:55:58.186 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:55:58.186 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:55:58.186 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:55:58.186 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:55:58.187 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:55:58.187 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:55:58.187 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:55:58.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:58.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:55:58.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:55:58.189 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:55:58Z" 2023-10-17T18:55:58.189 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:55:58.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568841.365181_703.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:54:01.365181Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:55:58Z" } 2023-10-17T18:55:58.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568841.365181_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:54:01.365181Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:55:58.191 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:58.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568841.365181_703.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:55:58.191 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568841.365181_703.1 2023-10-17T18:56:03.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:56:03.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T18:56:03.194 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T18:56:03.254 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:56:03.255 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:56:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:03.288+0000 7f412a38e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:56:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:03.290+0000 7f412a38e780 20 RGW-SYNC:meta: read sync status 2023-10-17T18:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T18:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T18:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T18:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T18:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T18:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T18:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T18:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T18:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697568865.677043_735.1", 2023-10-17T18:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T18:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:54:25.677043Z", 2023-10-17T18:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T18:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697566654.606901_130.1", 2023-10-17T18:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T18:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T18:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T18:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T18:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T18:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T18:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T18:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T18:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T18:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T18:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T18:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T18:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T18:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T18:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T18:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T18:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T18:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T18:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T18:56:03Z" 2023-10-17T18:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T18:56:03.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697568865.677043_735.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:54:25.677043Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697566654.606901_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:17:34.606901Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T18:56:03Z" } 2023-10-17T18:56:03.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568865.677043_735.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:54:25.677043Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T18:56:03.322 DEBUG:rgw_multi.tests:log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:56:03.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T18:56:03.323 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T18:56:03.323 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:03.324 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:03.324 DEBUG:boto:Method: HEAD 2023-10-17T18:56:03.325 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:03.325 DEBUG:boto:Data: 2023-10-17T18:56:03.326 DEBUG:boto:Headers: {} 2023-10-17T18:56:03.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.327 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.327 DEBUG:boto:Params: {} 2023-10-17T18:56:03.328 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:56:03.329 DEBUG:boto:Token: None 2023-10-17T18:56:03.329 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/ 2023-10-17T18:56:03.330 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T18:56:03.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T18:56:03.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ee7d734d2251824-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.336 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:03.337 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:03.337 DEBUG:boto:Method: PUT 2023-10-17T18:56:03.337 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:56:03.338 DEBUG:boto:Data: 2023-10-17T18:56:03.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:03.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.339 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.340 DEBUG:boto:Params: {} 2023-10-17T18:56:03.340 DEBUG:boto:Token: None 2023-10-17T18:56:03.340 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/obj11 2023-10-17T18:56:03.341 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GmMkVJogk5r6wfjVekxbq8dvVvE= 2023-10-17T18:56:03.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmMkVJogk5r6wfjVekxbq8dvVvE='} 2023-10-17T18:56:03.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e4b1a9bb0872f8-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.349 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:03.349 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:03.350 DEBUG:boto:Method: HEAD 2023-10-17T18:56:03.350 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:03.350 DEBUG:boto:Data: 2023-10-17T18:56:03.351 DEBUG:boto:Headers: {} 2023-10-17T18:56:03.351 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.352 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.352 DEBUG:boto:Params: {} 2023-10-17T18:56:03.352 DEBUG:boto:Token: None 2023-10-17T18:56:03.353 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/ 2023-10-17T18:56:03.353 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T18:56:03.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T18:56:03.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225a805abdd81031-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.356 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:03.357 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:03.357 DEBUG:boto:Method: PUT 2023-10-17T18:56:03.358 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:56:03.358 DEBUG:boto:Data: 2023-10-17T18:56:03.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:03.359 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.359 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.360 DEBUG:boto:Params: {} 2023-10-17T18:56:03.360 DEBUG:boto:Token: None 2023-10-17T18:56:03.361 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/obj22 2023-10-17T18:56:03.361 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/Zj7ugw/EGQASmX7VpgDBmGvH9s= 2023-10-17T18:56:03.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/Zj7ugw/EGQASmX7VpgDBmGvH9s='} 2023-10-17T18:56:03.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b205584a34381f-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.366 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:03.366 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:03.367 DEBUG:boto:Method: HEAD 2023-10-17T18:56:03.367 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:03.367 DEBUG:boto:Data: 2023-10-17T18:56:03.368 DEBUG:boto:Headers: {} 2023-10-17T18:56:03.368 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.368 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.369 DEBUG:boto:Params: {} 2023-10-17T18:56:03.369 DEBUG:boto:Token: None 2023-10-17T18:56:03.370 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/ 2023-10-17T18:56:03.370 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T18:56:03.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T18:56:03.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e608a69cebc63e79-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.374 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:03.374 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:03.374 DEBUG:boto:Method: PUT 2023-10-17T18:56:03.375 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:56:03.375 DEBUG:boto:Data: 2023-10-17T18:56:03.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:03.376 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.376 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.377 DEBUG:boto:Params: {} 2023-10-17T18:56:03.377 DEBUG:boto:Token: None 2023-10-17T18:56:03.378 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/obj33 2023-10-17T18:56:03.378 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hlBZEqZYr99AF15rJsV6rXST2J0= 2023-10-17T18:56:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hlBZEqZYr99AF15rJsV6rXST2J0='} 2023-10-17T18:56:03.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb8dcf2b2804a72e-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.384 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:03.385 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:03.385 DEBUG:boto:Method: HEAD 2023-10-17T18:56:03.385 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:03.386 DEBUG:boto:Data: 2023-10-17T18:56:03.386 DEBUG:boto:Headers: {} 2023-10-17T18:56:03.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.387 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.387 DEBUG:boto:Params: {} 2023-10-17T18:56:03.387 DEBUG:boto:Token: None 2023-10-17T18:56:03.388 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/ 2023-10-17T18:56:03.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T18:56:03.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T18:56:03.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b846ea310e598f5f-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.392 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:03.392 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:03.392 DEBUG:boto:Method: PUT 2023-10-17T18:56:03.393 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:56:03.393 DEBUG:boto:Data: 2023-10-17T18:56:03.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:03.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.394 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.395 DEBUG:boto:Params: {} 2023-10-17T18:56:03.395 DEBUG:boto:Token: None 2023-10-17T18:56:03.395 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/obj44 2023-10-17T18:56:03.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ydryy37zKLeiF/knNNa27gvYGUU= 2023-10-17T18:56:03.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ydryy37zKLeiF/knNNa27gvYGUU='} 2023-10-17T18:56:03.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000319013750d9b34f4-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.439 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:03.439 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:03.440 DEBUG:boto:Method: HEAD 2023-10-17T18:56:03.440 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:03.440 DEBUG:boto:Data: 2023-10-17T18:56:03.441 DEBUG:boto:Headers: {} 2023-10-17T18:56:03.441 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.442 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.442 DEBUG:boto:Params: {} 2023-10-17T18:56:03.442 DEBUG:boto:Token: None 2023-10-17T18:56:03.443 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/ 2023-10-17T18:56:03.443 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T18:56:03.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T18:56:03.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a4820f761b980d5-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.447 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:03.447 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:03.447 DEBUG:boto:Method: PUT 2023-10-17T18:56:03.448 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:56:03.448 DEBUG:boto:Data: 2023-10-17T18:56:03.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:03.449 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:03.449 DEBUG:boto:Port: 8000 2023-10-17T18:56:03.449 DEBUG:boto:Params: {} 2023-10-17T18:56:03.450 DEBUG:boto:Token: None 2023-10-17T18:56:03.450 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:03 GMT /hbmuog-59/obj55 2023-10-17T18:56:03.451 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SvSfDvltaEHSj02HGwVcsSXsbJA= 2023-10-17T18:56:03.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvSfDvltaEHSj02HGwVcsSXsbJA='} 2023-10-17T18:56:03.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036a3d1fe9fd0aa68-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:03.456 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T18:56:03.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:03.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:03.457 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:56:03.517 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:56:03.539 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:56:03.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:03.550+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:08.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:08.550+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:13.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:13.551+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:18.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:18.552+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:23.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:23.553+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:28.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:28.552+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T18:56:33.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:33.554+0000 7fcc492ad780 1 bucket sync caught up with source: 2023-10-17T18:56:33.556 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.74.6, 00000000002.76.6, 00000000002.48.6] 2023-10-17T18:56:33.556 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.74.6, 00000000002.76.6, 00000000002.48.6] 2023-10-17T18:56:33.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:33.554+0000 7fcc492ad780 0 bucket checkpoint complete 2023-10-17T18:56:33.573 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-59 2023-10-17T18:56:33.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:33.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:33.575 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:56:33.639 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:56:33.639 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:56:33.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:33.678+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T18:56:33.680 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:33.680 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:38.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:38.679+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T18:56:38.681 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:38.682 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:43.682 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:43.681+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T18:56:43.683 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:43.683 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:48.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:48.682+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T18:56:48.684 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:48.684 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:53.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:53.684+0000 7f09990f4780 1 bucket sync caught up with source: 2023-10-17T18:56:53.685 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:53.686 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T18:56:53.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:53.684+0000 7f09990f4780 0 bucket checkpoint complete 2023-10-17T18:56:53.703 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-59 zones={a2, a1} 2023-10-17T18:56:53.704 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.704 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.704 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.705 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:53.705 DEBUG:boto:Data: 2023-10-17T18:56:53.706 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.706 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.706 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.707 DEBUG:boto:Params: {} 2023-10-17T18:56:53.707 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:56:53.708 DEBUG:boto:Token: None 2023-10-17T18:56:53.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI= 2023-10-17T18:56:53.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI=', 'Content-Length': '0'} 2023-10-17T18:56:53.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acccc2dcbae9e7ac-00652ed8f5-1b46-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.714 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.715 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.715 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.716 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:53.716 DEBUG:boto:Data: 2023-10-17T18:56:53.716 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.717 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.718 DEBUG:boto:Params: {} 2023-10-17T18:56:53.718 DEBUG:boto:Token: None 2023-10-17T18:56:53.718 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI= 2023-10-17T18:56:53.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI=', 'Content-Length': '0'} 2023-10-17T18:56:53.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000274bfedb3f6267e8-00652ed8f5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:56:53.723 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.723 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.724 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.724 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.725 DEBUG:boto:Method: GET 2023-10-17T18:56:53.725 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.725 DEBUG:boto:Data: 2023-10-17T18:56:53.726 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.727 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.727 DEBUG:boto:Params: {} 2023-10-17T18:56:53.728 DEBUG:boto:Token: None 2023-10-17T18:56:53.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/?versions 2023-10-17T18:56:53.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T18:56:53.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T18:56:53.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907a50b1daa8c931-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.732 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T18:56:53.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T18:56:53.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T18:56:53.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T18:56:53.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T18:56:53.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:56:53.735 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.736 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.736 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.737 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.737 DEBUG:boto:Method: GET 2023-10-17T18:56:53.738 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.738 DEBUG:boto:Data: 2023-10-17T18:56:53.738 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.739 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.739 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.740 DEBUG:boto:Params: {} 2023-10-17T18:56:53.740 DEBUG:boto:Token: None 2023-10-17T18:56:53.741 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/?versions 2023-10-17T18:56:53.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T18:56:53.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T18:56:53.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e255e1d361e9e5-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.745 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T18:56:53.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T18:56:53.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T18:56:53.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T18:56:53.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T18:56:53.748 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.748 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.749 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.749 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.749 DEBUG:boto:Method: GET 2023-10-17T18:56:53.750 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.750 DEBUG:boto:Data: 2023-10-17T18:56:53.751 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.751 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.752 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.752 DEBUG:boto:Params: {} 2023-10-17T18:56:53.752 DEBUG:boto:Token: None 2023-10-17T18:56:53.753 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/?versions 2023-10-17T18:56:53.754 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T18:56:53.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T18:56:53.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007f2ebb61ad68e69-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.757 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.758 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.759 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.759 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.759 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.760 DEBUG:boto:Method: GET 2023-10-17T18:56:53.760 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:56:53.761 DEBUG:boto:Data: 2023-10-17T18:56:53.761 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.762 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.762 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.762 DEBUG:boto:Params: {} 2023-10-17T18:56:53.763 DEBUG:boto:Token: None 2023-10-17T18:56:53.764 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/?versions 2023-10-17T18:56:53.764 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T18:56:53.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T18:56:53.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91af2fb4c23ba42-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.768 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:56:53.769 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.770 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.770 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.771 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.771 DEBUG:boto:Method: GET 2023-10-17T18:56:53.772 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.772 DEBUG:boto:Data: 2023-10-17T18:56:53.772 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.773 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.773 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.774 DEBUG:boto:Params: {} 2023-10-17T18:56:53.774 DEBUG:boto:Token: None 2023-10-17T18:56:53.775 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.775 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T18:56:53.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T18:56:53.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d10b73d54d1236-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.779 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.779 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.780 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.780 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.781 DEBUG:boto:Method: GET 2023-10-17T18:56:53.781 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.782 DEBUG:boto:Data: 2023-10-17T18:56:53.782 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.783 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.784 DEBUG:boto:Params: {} 2023-10-17T18:56:53.784 DEBUG:boto:Token: None 2023-10-17T18:56:53.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T18:56:53.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T18:56:53.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286fd603ead895ee-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.789 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.789 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.790 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.790 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.791 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.791 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.792 DEBUG:boto:Data: 2023-10-17T18:56:53.792 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.793 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.793 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.794 DEBUG:boto:Params: {} 2023-10-17T18:56:53.794 DEBUG:boto:Token: None 2023-10-17T18:56:53.795 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.795 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM= 2023-10-17T18:56:53.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM=', 'Content-Length': '0'} 2023-10-17T18:56:53.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6574dd9e215d73d-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.798 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.799 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.799 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.800 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.800 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.801 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.801 DEBUG:boto:Data: 2023-10-17T18:56:53.802 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.803 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.803 DEBUG:boto:Params: {} 2023-10-17T18:56:53.804 DEBUG:boto:Token: None 2023-10-17T18:56:53.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM= 2023-10-17T18:56:53.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM=', 'Content-Length': '0'} 2023-10-17T18:56:53.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf473a7b5db3592-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:56:53.808 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.808 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.809 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.809 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.810 DEBUG:boto:Method: GET 2023-10-17T18:56:53.810 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.811 DEBUG:boto:Data: 2023-10-17T18:56:53.811 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.812 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.813 DEBUG:boto:Params: {} 2023-10-17T18:56:53.813 DEBUG:boto:Token: None 2023-10-17T18:56:53.814 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.814 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T18:56:53.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T18:56:53.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b45ba7210919f2b-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.817 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.818 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.818 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.819 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.819 DEBUG:boto:Method: GET 2023-10-17T18:56:53.820 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.820 DEBUG:boto:Data: 2023-10-17T18:56:53.821 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.822 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.822 DEBUG:boto:Params: {} 2023-10-17T18:56:53.823 DEBUG:boto:Token: None 2023-10-17T18:56:53.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:56:53.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T18:56:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T18:56:53.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015537fe6649643a7-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.874 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.875 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.876 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.876 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.876 DEBUG:boto:Method: GET 2023-10-17T18:56:53.877 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.877 DEBUG:boto:Data: 2023-10-17T18:56:53.878 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.879 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.879 DEBUG:boto:Params: {} 2023-10-17T18:56:53.880 DEBUG:boto:Token: None 2023-10-17T18:56:53.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.881 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T18:56:53.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T18:56:53.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce63c4b33681d8ee-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.885 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.886 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.886 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.887 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.887 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.887 DEBUG:boto:Method: GET 2023-10-17T18:56:53.888 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.888 DEBUG:boto:Data: 2023-10-17T18:56:53.889 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.889 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.890 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.890 DEBUG:boto:Params: {} 2023-10-17T18:56:53.891 DEBUG:boto:Token: None 2023-10-17T18:56:53.891 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.892 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T18:56:53.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T18:56:53.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ade935389c834c-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.895 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.896 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.897 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.897 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.897 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:56:53.898 DEBUG:boto:Data: 2023-10-17T18:56:53.898 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.899 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.900 DEBUG:boto:Params: {} 2023-10-17T18:56:53.900 DEBUG:boto:Token: None 2023-10-17T18:56:53.901 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11 2023-10-17T18:56:53.901 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE= 2023-10-17T18:56:53.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE=', 'Content-Length': '0'} 2023-10-17T18:56:53.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006405065e7bb7e1a7-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.904 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.905 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.905 DEBUG:boto:Method: HEAD 2023-10-17T18:56:53.906 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:56:53.906 DEBUG:boto:Data: 2023-10-17T18:56:53.907 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.907 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.908 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.908 DEBUG:boto:Params: {} 2023-10-17T18:56:53.909 DEBUG:boto:Token: None 2023-10-17T18:56:53.909 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11 2023-10-17T18:56:53.910 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE= 2023-10-17T18:56:53.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE=', 'Content-Length': '0'} 2023-10-17T18:56:53.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a4e4645089d5af-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:56:53.914 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.914 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.915 DEBUG:boto:Method: GET 2023-10-17T18:56:53.915 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:56:53.915 DEBUG:boto:Data: 2023-10-17T18:56:53.916 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.917 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.917 DEBUG:boto:Params: {} 2023-10-17T18:56:53.918 DEBUG:boto:Token: None 2023-10-17T18:56:53.918 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11 2023-10-17T18:56:53.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM= 2023-10-17T18:56:53.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM=', 'Content-Length': '0'} 2023-10-17T18:56:53.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943a84a9cacfbaba-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.922 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:56:53.922 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:56:53.923 DEBUG:boto:Method: GET 2023-10-17T18:56:53.923 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:56:53.924 DEBUG:boto:Data: 2023-10-17T18:56:53.924 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.925 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.926 DEBUG:boto:Params: {} 2023-10-17T18:56:53.926 DEBUG:boto:Token: None 2023-10-17T18:56:53.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/obj11 2023-10-17T18:56:53.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM= 2023-10-17T18:56:53.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM=', 'Content-Length': '0'} 2023-10-17T18:56:53.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8fefe7755f03486-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.931 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.931 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.932 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.932 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.933 DEBUG:boto:Method: GET 2023-10-17T18:56:53.933 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.933 DEBUG:boto:Data: 2023-10-17T18:56:53.934 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.934 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:53.935 DEBUG:boto:Port: 8001 2023-10-17T18:56:53.935 DEBUG:boto:Params: {} 2023-10-17T18:56:53.936 DEBUG:boto:Token: None 2023-10-17T18:56:53.936 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.937 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T18:56:53.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T18:56:53.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060d559aeaa91e382-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.941 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.942 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:53.942 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:53.943 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.943 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.943 DEBUG:boto:Method: GET 2023-10-17T18:56:53.944 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:56:53.944 DEBUG:boto:Data: 2023-10-17T18:56:53.945 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.945 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:53.946 DEBUG:boto:Port: 8000 2023-10-17T18:56:53.946 DEBUG:boto:Params: {} 2023-10-17T18:56:53.947 DEBUG:boto:Token: None 2023-10-17T18:56:53.947 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:53 GMT /hbmuog-59/ 2023-10-17T18:56:53.948 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T18:56:53.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T18:56:53.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000084885bd0166a9c7-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:53.994 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:53.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:56:53.996 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:53.996 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:53.996 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:53.997 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:53.997 DEBUG:boto:Method: GET 2023-10-17T18:56:53.998 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:53.998 DEBUG:boto:Data: 2023-10-17T18:56:53.999 DEBUG:boto:Headers: {} 2023-10-17T18:56:53.999 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.000 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.000 DEBUG:boto:Params: {} 2023-10-17T18:56:54.001 DEBUG:boto:Token: None 2023-10-17T18:56:54.001 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.002 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T18:56:54.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T18:56:54.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc070428713d31f-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.005 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.006 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.006 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.007 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.007 DEBUG:boto:Method: GET 2023-10-17T18:56:54.007 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.008 DEBUG:boto:Data: 2023-10-17T18:56:54.008 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.009 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.010 DEBUG:boto:Params: {} 2023-10-17T18:56:54.010 DEBUG:boto:Token: None 2023-10-17T18:56:54.011 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T18:56:54.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T18:56:54.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5934142d6f9f466-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.015 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.015 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.016 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.016 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.016 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.017 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.017 DEBUG:boto:Data: 2023-10-17T18:56:54.018 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.019 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.019 DEBUG:boto:Params: {} 2023-10-17T18:56:54.020 DEBUG:boto:Token: None 2023-10-17T18:56:54.020 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.021 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0= 2023-10-17T18:56:54.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0=', 'Content-Length': '0'} 2023-10-17T18:56:54.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af21f2c5ff0e4010-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.024 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.025 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.025 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.025 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.026 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.026 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.027 DEBUG:boto:Data: 2023-10-17T18:56:54.027 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.028 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.028 DEBUG:boto:Params: {} 2023-10-17T18:56:54.029 DEBUG:boto:Token: None 2023-10-17T18:56:54.029 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0= 2023-10-17T18:56:54.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0=', 'Content-Length': '0'} 2023-10-17T18:56:54.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d7e1b2c8e94294-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:56:54.033 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.034 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.034 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.035 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.035 DEBUG:boto:Method: GET 2023-10-17T18:56:54.035 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.036 DEBUG:boto:Data: 2023-10-17T18:56:54.036 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.037 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.038 DEBUG:boto:Params: {} 2023-10-17T18:56:54.038 DEBUG:boto:Token: None 2023-10-17T18:56:54.039 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.039 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T18:56:54.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T18:56:54.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064017165a57366a9-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.043 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.043 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.044 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.044 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.045 DEBUG:boto:Method: GET 2023-10-17T18:56:54.045 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.045 DEBUG:boto:Data: 2023-10-17T18:56:54.046 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.047 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.047 DEBUG:boto:Params: {} 2023-10-17T18:56:54.048 DEBUG:boto:Token: None 2023-10-17T18:56:54.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:56:54.049 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T18:56:54.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T18:56:54.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f28efaf94910d2-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.053 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.053 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.054 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.054 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.055 DEBUG:boto:Method: GET 2023-10-17T18:56:54.055 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.056 DEBUG:boto:Data: 2023-10-17T18:56:54.056 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.057 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.058 DEBUG:boto:Params: {} 2023-10-17T18:56:54.058 DEBUG:boto:Token: None 2023-10-17T18:56:54.059 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75c926c5fb2a222-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.063 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.064 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.064 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.065 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.065 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.066 DEBUG:boto:Method: GET 2023-10-17T18:56:54.066 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.067 DEBUG:boto:Data: 2023-10-17T18:56:54.067 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.068 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.068 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.069 DEBUG:boto:Params: {} 2023-10-17T18:56:54.069 DEBUG:boto:Token: None 2023-10-17T18:56:54.070 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.071 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002cb46bed3f6b66-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.074 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.075 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.075 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.076 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.076 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:56:54.077 DEBUG:boto:Data: 2023-10-17T18:56:54.077 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.078 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.078 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.078 DEBUG:boto:Params: {} 2023-10-17T18:56:54.079 DEBUG:boto:Token: None 2023-10-17T18:56:54.079 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22 2023-10-17T18:56:54.080 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g= 2023-10-17T18:56:54.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g=', 'Content-Length': '0'} 2023-10-17T18:56:54.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011714f76066aada6-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.083 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.084 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.084 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.084 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:56:54.085 DEBUG:boto:Data: 2023-10-17T18:56:54.085 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.086 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.086 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.087 DEBUG:boto:Params: {} 2023-10-17T18:56:54.087 DEBUG:boto:Token: None 2023-10-17T18:56:54.088 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22 2023-10-17T18:56:54.088 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g= 2023-10-17T18:56:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g=', 'Content-Length': '0'} 2023-10-17T18:56:54.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7ac6cc1b3a9706-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:56:54.092 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.092 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.093 DEBUG:boto:Method: GET 2023-10-17T18:56:54.093 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:56:54.094 DEBUG:boto:Data: 2023-10-17T18:56:54.094 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.094 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.095 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.095 DEBUG:boto:Params: {} 2023-10-17T18:56:54.096 DEBUG:boto:Token: None 2023-10-17T18:56:54.096 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22 2023-10-17T18:56:54.097 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw= 2023-10-17T18:56:54.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw=', 'Content-Length': '0'} 2023-10-17T18:56:54.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee5555128f36d57-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.146 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:56:54.147 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:56:54.148 DEBUG:boto:Method: GET 2023-10-17T18:56:54.149 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:56:54.150 DEBUG:boto:Data: 2023-10-17T18:56:54.150 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.151 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.151 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.152 DEBUG:boto:Params: {} 2023-10-17T18:56:54.152 DEBUG:boto:Token: None 2023-10-17T18:56:54.153 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj22 2023-10-17T18:56:54.153 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw= 2023-10-17T18:56:54.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw=', 'Content-Length': '0'} 2023-10-17T18:56:54.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d9eae0c83016b0-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.157 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.157 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.158 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.158 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.159 DEBUG:boto:Method: GET 2023-10-17T18:56:54.160 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.160 DEBUG:boto:Data: 2023-10-17T18:56:54.161 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.161 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.162 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.163 DEBUG:boto:Params: {} 2023-10-17T18:56:54.164 DEBUG:boto:Token: None 2023-10-17T18:56:54.165 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.165 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5d80c2c1ee72de-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.170 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.171 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.171 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.172 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.172 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.173 DEBUG:boto:Method: GET 2023-10-17T18:56:54.173 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:56:54.174 DEBUG:boto:Data: 2023-10-17T18:56:54.174 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.175 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.175 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.176 DEBUG:boto:Params: {} 2023-10-17T18:56:54.177 DEBUG:boto:Token: None 2023-10-17T18:56:54.177 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.178 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8a06206108c5d1-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.183 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:56:54.184 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.185 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.185 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.186 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.186 DEBUG:boto:Method: GET 2023-10-17T18:56:54.187 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.187 DEBUG:boto:Data: 2023-10-17T18:56:54.187 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.188 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.189 DEBUG:boto:Params: {} 2023-10-17T18:56:54.189 DEBUG:boto:Token: None 2023-10-17T18:56:54.190 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.190 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T18:56:54.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T18:56:54.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a57f75621012c5d-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.193 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.194 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.194 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.195 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.195 DEBUG:boto:Method: GET 2023-10-17T18:56:54.196 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.196 DEBUG:boto:Data: 2023-10-17T18:56:54.196 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.197 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.198 DEBUG:boto:Params: {} 2023-10-17T18:56:54.198 DEBUG:boto:Token: None 2023-10-17T18:56:54.199 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.199 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T18:56:54.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T18:56:54.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002395ce9f720a0067-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.202 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.203 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.203 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.204 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.204 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.204 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.205 DEBUG:boto:Data: 2023-10-17T18:56:54.205 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.206 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.206 DEBUG:boto:Params: {} 2023-10-17T18:56:54.207 DEBUG:boto:Token: None 2023-10-17T18:56:54.207 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA= 2023-10-17T18:56:54.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA=', 'Content-Length': '0'} 2023-10-17T18:56:54.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fcb3487eca092a3-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.211 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.211 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.211 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.212 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.212 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.213 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.213 DEBUG:boto:Data: 2023-10-17T18:56:54.214 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.214 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.214 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.215 DEBUG:boto:Params: {} 2023-10-17T18:56:54.215 DEBUG:boto:Token: None 2023-10-17T18:56:54.216 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.216 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA= 2023-10-17T18:56:54.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA=', 'Content-Length': '0'} 2023-10-17T18:56:54.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102c58fbde5aff15-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:56:54.220 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.220 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.221 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.221 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.221 DEBUG:boto:Method: GET 2023-10-17T18:56:54.222 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.222 DEBUG:boto:Data: 2023-10-17T18:56:54.223 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.223 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.224 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.224 DEBUG:boto:Params: {} 2023-10-17T18:56:54.225 DEBUG:boto:Token: None 2023-10-17T18:56:54.225 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.225 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T18:56:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T18:56:54.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd4bd5030606413-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.229 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.229 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.230 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.230 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.230 DEBUG:boto:Method: GET 2023-10-17T18:56:54.231 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.231 DEBUG:boto:Data: 2023-10-17T18:56:54.232 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.232 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.233 DEBUG:boto:Params: {} 2023-10-17T18:56:54.233 DEBUG:boto:Token: None 2023-10-17T18:56:54.234 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:56:54.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T18:56:54.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T18:56:54.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3906afe4e81ee71-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.278 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.279 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.279 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.280 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.280 DEBUG:boto:Method: GET 2023-10-17T18:56:54.280 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.281 DEBUG:boto:Data: 2023-10-17T18:56:54.281 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.282 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.282 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.283 DEBUG:boto:Params: {} 2023-10-17T18:56:54.283 DEBUG:boto:Token: None 2023-10-17T18:56:54.284 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.284 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c45c1fade8c665-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.288 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.289 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.289 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.290 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.290 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.290 DEBUG:boto:Method: GET 2023-10-17T18:56:54.291 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.291 DEBUG:boto:Data: 2023-10-17T18:56:54.292 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.292 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.293 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.293 DEBUG:boto:Params: {} 2023-10-17T18:56:54.294 DEBUG:boto:Token: None 2023-10-17T18:56:54.294 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f26fd2874d24f67-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.298 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.299 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.299 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.300 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.300 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:56:54.301 DEBUG:boto:Data: 2023-10-17T18:56:54.301 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.302 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.302 DEBUG:boto:Params: {} 2023-10-17T18:56:54.303 DEBUG:boto:Token: None 2023-10-17T18:56:54.303 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33 2023-10-17T18:56:54.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs= 2023-10-17T18:56:54.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs=', 'Content-Length': '0'} 2023-10-17T18:56:54.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c538763ae6d57731-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.307 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.307 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.308 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.308 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:56:54.309 DEBUG:boto:Data: 2023-10-17T18:56:54.309 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.310 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.310 DEBUG:boto:Params: {} 2023-10-17T18:56:54.311 DEBUG:boto:Token: None 2023-10-17T18:56:54.311 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33 2023-10-17T18:56:54.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs= 2023-10-17T18:56:54.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs=', 'Content-Length': '0'} 2023-10-17T18:56:54.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b7707307931827-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:56:54.316 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.316 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.317 DEBUG:boto:Method: GET 2023-10-17T18:56:54.317 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:56:54.317 DEBUG:boto:Data: 2023-10-17T18:56:54.318 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.318 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.319 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.319 DEBUG:boto:Params: {} 2023-10-17T18:56:54.320 DEBUG:boto:Token: None 2023-10-17T18:56:54.321 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33 2023-10-17T18:56:54.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0= 2023-10-17T18:56:54.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0=', 'Content-Length': '0'} 2023-10-17T18:56:54.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13a525292fec653-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.366 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:56:54.367 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:56:54.367 DEBUG:boto:Method: GET 2023-10-17T18:56:54.368 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:56:54.368 DEBUG:boto:Data: 2023-10-17T18:56:54.369 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.369 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.370 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.370 DEBUG:boto:Params: {} 2023-10-17T18:56:54.371 DEBUG:boto:Token: None 2023-10-17T18:56:54.371 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj33 2023-10-17T18:56:54.372 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0= 2023-10-17T18:56:54.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0=', 'Content-Length': '0'} 2023-10-17T18:56:54.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b7c28637aad379-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.375 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.375 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.376 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.376 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.377 DEBUG:boto:Method: GET 2023-10-17T18:56:54.377 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.378 DEBUG:boto:Data: 2023-10-17T18:56:54.378 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.378 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.379 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.379 DEBUG:boto:Params: {} 2023-10-17T18:56:54.380 DEBUG:boto:Token: None 2023-10-17T18:56:54.380 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.381 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb71b443de1fb51d-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.384 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.385 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.386 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.386 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.387 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.387 DEBUG:boto:Method: GET 2023-10-17T18:56:54.387 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:56:54.388 DEBUG:boto:Data: 2023-10-17T18:56:54.388 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.389 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.389 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.390 DEBUG:boto:Params: {} 2023-10-17T18:56:54.390 DEBUG:boto:Token: None 2023-10-17T18:56:54.391 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.391 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a208e00b63855d-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.395 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:56:54.396 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.397 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.397 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.398 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.398 DEBUG:boto:Method: GET 2023-10-17T18:56:54.399 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.399 DEBUG:boto:Data: 2023-10-17T18:56:54.400 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.400 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.401 DEBUG:boto:Params: {} 2023-10-17T18:56:54.401 DEBUG:boto:Token: None 2023-10-17T18:56:54.402 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T18:56:54.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T18:56:54.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6d85a0630c7ff3-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.406 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.406 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.407 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.407 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.408 DEBUG:boto:Method: GET 2023-10-17T18:56:54.408 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.408 DEBUG:boto:Data: 2023-10-17T18:56:54.409 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.410 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.410 DEBUG:boto:Params: {} 2023-10-17T18:56:54.411 DEBUG:boto:Token: None 2023-10-17T18:56:54.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.412 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T18:56:54.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T18:56:54.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198f52aababbc24c-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.458 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.459 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.459 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.460 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.460 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.461 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.461 DEBUG:boto:Data: 2023-10-17T18:56:54.461 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.462 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.463 DEBUG:boto:Params: {} 2023-10-17T18:56:54.463 DEBUG:boto:Token: None 2023-10-17T18:56:54.464 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM= 2023-10-17T18:56:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM=', 'Content-Length': '0'} 2023-10-17T18:56:54.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b3f58d1aae7985-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.467 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.468 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.468 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.469 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.470 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.470 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.471 DEBUG:boto:Data: 2023-10-17T18:56:54.471 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.472 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.473 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.474 DEBUG:boto:Params: {} 2023-10-17T18:56:54.475 DEBUG:boto:Token: None 2023-10-17T18:56:54.475 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.476 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM= 2023-10-17T18:56:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM=', 'Content-Length': '0'} 2023-10-17T18:56:54.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ab8e263b573afa-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:56:54.480 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.481 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.481 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.481 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.482 DEBUG:boto:Method: GET 2023-10-17T18:56:54.482 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.483 DEBUG:boto:Data: 2023-10-17T18:56:54.483 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.484 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.484 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.485 DEBUG:boto:Params: {} 2023-10-17T18:56:54.485 DEBUG:boto:Token: None 2023-10-17T18:56:54.486 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.486 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T18:56:54.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T18:56:54.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63e75a1e4435313-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.534 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.535 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.535 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.536 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.536 DEBUG:boto:Method: GET 2023-10-17T18:56:54.537 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.537 DEBUG:boto:Data: 2023-10-17T18:56:54.537 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.538 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.538 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.539 DEBUG:boto:Params: {} 2023-10-17T18:56:54.539 DEBUG:boto:Token: None 2023-10-17T18:56:54.540 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:56:54.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T18:56:54.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T18:56:54.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da82ef902a95bd2-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.544 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.545 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.545 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.546 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.546 DEBUG:boto:Method: GET 2023-10-17T18:56:54.547 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.547 DEBUG:boto:Data: 2023-10-17T18:56:54.547 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.548 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.548 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.549 DEBUG:boto:Params: {} 2023-10-17T18:56:54.549 DEBUG:boto:Token: None 2023-10-17T18:56:54.549 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.550 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad6c73e2a0e0487-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.554 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.555 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.555 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.556 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.556 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.557 DEBUG:boto:Method: GET 2023-10-17T18:56:54.557 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.557 DEBUG:boto:Data: 2023-10-17T18:56:54.558 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.559 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.559 DEBUG:boto:Params: {} 2023-10-17T18:56:54.560 DEBUG:boto:Token: None 2023-10-17T18:56:54.560 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eec324d860473720-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.564 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.565 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.565 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.566 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.566 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:56:54.566 DEBUG:boto:Data: 2023-10-17T18:56:54.567 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.568 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.568 DEBUG:boto:Params: {} 2023-10-17T18:56:54.569 DEBUG:boto:Token: None 2023-10-17T18:56:54.569 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44 2023-10-17T18:56:54.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg= 2023-10-17T18:56:54.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg=', 'Content-Length': '0'} 2023-10-17T18:56:54.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e493d105b6bcb0b-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.573 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.573 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.574 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.574 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:56:54.574 DEBUG:boto:Data: 2023-10-17T18:56:54.575 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.575 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.576 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.576 DEBUG:boto:Params: {} 2023-10-17T18:56:54.576 DEBUG:boto:Token: None 2023-10-17T18:56:54.577 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44 2023-10-17T18:56:54.577 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg= 2023-10-17T18:56:54.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg=', 'Content-Length': '0'} 2023-10-17T18:56:54.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a5e206eea057af-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:56:54.582 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.583 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.583 DEBUG:boto:Method: GET 2023-10-17T18:56:54.583 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:56:54.584 DEBUG:boto:Data: 2023-10-17T18:56:54.584 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.585 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.585 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.586 DEBUG:boto:Params: {} 2023-10-17T18:56:54.586 DEBUG:boto:Token: None 2023-10-17T18:56:54.587 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44 2023-10-17T18:56:54.587 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw= 2023-10-17T18:56:54.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw=', 'Content-Length': '0'} 2023-10-17T18:56:54.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e2f98ae14e3ce8e-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.591 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:56:54.591 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:56:54.592 DEBUG:boto:Method: GET 2023-10-17T18:56:54.592 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:56:54.592 DEBUG:boto:Data: 2023-10-17T18:56:54.593 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.593 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.594 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.594 DEBUG:boto:Params: {} 2023-10-17T18:56:54.595 DEBUG:boto:Token: None 2023-10-17T18:56:54.595 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj44 2023-10-17T18:56:54.596 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw= 2023-10-17T18:56:54.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw=', 'Content-Length': '0'} 2023-10-17T18:56:54.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095c0b08ce40d534-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.646 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.647 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.647 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.648 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.648 DEBUG:boto:Method: GET 2023-10-17T18:56:54.649 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.650 DEBUG:boto:Data: 2023-10-17T18:56:54.650 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.650 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.651 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.651 DEBUG:boto:Params: {} 2023-10-17T18:56:54.652 DEBUG:boto:Token: None 2023-10-17T18:56:54.653 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712fa29a17fb8adb-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.657 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.658 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.659 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.659 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.659 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.660 DEBUG:boto:Method: GET 2023-10-17T18:56:54.660 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:56:54.661 DEBUG:boto:Data: 2023-10-17T18:56:54.661 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.662 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.662 DEBUG:boto:Params: {} 2023-10-17T18:56:54.663 DEBUG:boto:Token: None 2023-10-17T18:56:54.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa14d184f158d54-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.668 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:56:54.669 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.670 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.670 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.670 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.671 DEBUG:boto:Method: GET 2023-10-17T18:56:54.671 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.672 DEBUG:boto:Data: 2023-10-17T18:56:54.672 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.673 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.673 DEBUG:boto:Params: {} 2023-10-17T18:56:54.674 DEBUG:boto:Token: None 2023-10-17T18:56:54.674 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T18:56:54.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T18:56:54.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f0b0c130078677-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.678 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.679 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.679 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.679 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.680 DEBUG:boto:Method: GET 2023-10-17T18:56:54.680 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.681 DEBUG:boto:Data: 2023-10-17T18:56:54.681 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.682 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.682 DEBUG:boto:Params: {} 2023-10-17T18:56:54.683 DEBUG:boto:Token: None 2023-10-17T18:56:54.684 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T18:56:54.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T18:56:54.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26e93db29a29284-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.687 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.688 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.688 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.689 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.689 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.690 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.690 DEBUG:boto:Data: 2023-10-17T18:56:54.691 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.691 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.692 DEBUG:boto:Params: {} 2023-10-17T18:56:54.692 DEBUG:boto:Token: None 2023-10-17T18:56:54.693 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds= 2023-10-17T18:56:54.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds=', 'Content-Length': '0'} 2023-10-17T18:56:54.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec1f19022af33a8-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.696 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.696 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.697 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.697 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.697 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.698 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.698 DEBUG:boto:Data: 2023-10-17T18:56:54.699 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.700 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.700 DEBUG:boto:Params: {} 2023-10-17T18:56:54.701 DEBUG:boto:Token: None 2023-10-17T18:56:54.701 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.702 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds= 2023-10-17T18:56:54.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds=', 'Content-Length': '0'} 2023-10-17T18:56:54.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006388f1d6d95859d7-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:56:54.705 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.706 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.706 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.707 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.707 DEBUG:boto:Method: GET 2023-10-17T18:56:54.708 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.708 DEBUG:boto:Data: 2023-10-17T18:56:54.708 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.709 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.709 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.710 DEBUG:boto:Params: {} 2023-10-17T18:56:54.710 DEBUG:boto:Token: None 2023-10-17T18:56:54.711 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.711 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T18:56:54.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T18:56:54.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1d9ff61d6d2a5c-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.714 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.715 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.718 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.718 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.719 DEBUG:boto:Method: GET 2023-10-17T18:56:54.719 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.720 DEBUG:boto:Data: 2023-10-17T18:56:54.720 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.721 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.721 DEBUG:boto:Params: {} 2023-10-17T18:56:54.722 DEBUG:boto:Token: None 2023-10-17T18:56:54.722 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:56:54.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T18:56:54.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T18:56:54.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441f269621c135d4-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.766 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.767 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.767 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.768 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.768 DEBUG:boto:Method: GET 2023-10-17T18:56:54.769 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.769 DEBUG:boto:Data: 2023-10-17T18:56:54.770 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.771 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.771 DEBUG:boto:Params: {} 2023-10-17T18:56:54.771 DEBUG:boto:Token: None 2023-10-17T18:56:54.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.773 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009098963d60ff4623-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.777 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.777 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.778 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.778 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.779 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.779 DEBUG:boto:Method: GET 2023-10-17T18:56:54.780 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.780 DEBUG:boto:Data: 2023-10-17T18:56:54.781 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.781 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.781 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.782 DEBUG:boto:Params: {} 2023-10-17T18:56:54.782 DEBUG:boto:Token: None 2023-10-17T18:56:54.783 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.783 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f1dc55b1abc51f-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.787 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.788 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.788 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.789 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.789 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:56:54.789 DEBUG:boto:Data: 2023-10-17T18:56:54.790 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.790 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.791 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.791 DEBUG:boto:Params: {} 2023-10-17T18:56:54.791 DEBUG:boto:Token: None 2023-10-17T18:56:54.792 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55 2023-10-17T18:56:54.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0= 2023-10-17T18:56:54.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0=', 'Content-Length': '0'} 2023-10-17T18:56:54.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe134ba75171cf2-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.795 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.796 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.796 DEBUG:boto:Method: HEAD 2023-10-17T18:56:54.797 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:56:54.797 DEBUG:boto:Data: 2023-10-17T18:56:54.798 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.799 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.799 DEBUG:boto:Params: {} 2023-10-17T18:56:54.800 DEBUG:boto:Token: None 2023-10-17T18:56:54.800 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55 2023-10-17T18:56:54.801 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0= 2023-10-17T18:56:54.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0=', 'Content-Length': '0'} 2023-10-17T18:56:54.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d13071c24261066-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:56:54.804 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.804 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.805 DEBUG:boto:Method: GET 2023-10-17T18:56:54.805 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:56:54.806 DEBUG:boto:Data: 2023-10-17T18:56:54.806 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.807 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.807 DEBUG:boto:Params: {} 2023-10-17T18:56:54.808 DEBUG:boto:Token: None 2023-10-17T18:56:54.808 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55 2023-10-17T18:56:54.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g= 2023-10-17T18:56:54.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g=', 'Content-Length': '0'} 2023-10-17T18:56:54.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8979d3751495104-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.854 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:56:54.855 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:56:54.855 DEBUG:boto:Method: GET 2023-10-17T18:56:54.855 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:56:54.856 DEBUG:boto:Data: 2023-10-17T18:56:54.856 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.857 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.858 DEBUG:boto:Params: {} 2023-10-17T18:56:54.858 DEBUG:boto:Token: None 2023-10-17T18:56:54.859 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/obj55 2023-10-17T18:56:54.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g= 2023-10-17T18:56:54.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g=', 'Content-Length': '0'} 2023-10-17T18:56:54.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf4b73c8e81adc8-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.862 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.863 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.863 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.864 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.864 DEBUG:boto:Method: GET 2023-10-17T18:56:54.865 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.865 DEBUG:boto:Data: 2023-10-17T18:56:54.866 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.866 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:56:54.867 DEBUG:boto:Port: 8001 2023-10-17T18:56:54.867 DEBUG:boto:Params: {} 2023-10-17T18:56:54.868 DEBUG:boto:Token: None 2023-10-17T18:56:54.868 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e6fdd5c6a9eafe2-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.873 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.873 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:54.874 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:54.874 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.875 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.875 DEBUG:boto:Method: GET 2023-10-17T18:56:54.876 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:56:54.876 DEBUG:boto:Data: 2023-10-17T18:56:54.877 DEBUG:boto:Headers: {} 2023-10-17T18:56:54.877 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:54.878 DEBUG:boto:Port: 8000 2023-10-17T18:56:54.878 DEBUG:boto:Params: {} 2023-10-17T18:56:54.879 DEBUG:boto:Token: None 2023-10-17T18:56:54.879 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:56:54 GMT /hbmuog-59/ 2023-10-17T18:56:54.880 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T18:56:54.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T18:56:54.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87f96a2defa62ad-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:54.884 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:56:54.884 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-59 zones={a2, a1} 2023-10-17T18:56:54.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:56:54.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:56:54.886 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:56:54.948 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:56:54.948 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:56:54.994 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:56:54.994 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-59 2023-10-17T18:56:54.998 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 5 2023-10-17T18:56:55.036 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:55.036 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:55.037 DEBUG:boto:Method: HEAD 2023-10-17T18:56:55.037 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:55.038 DEBUG:boto:Data: 2023-10-17T18:56:55.038 DEBUG:boto:Headers: {} 2023-10-17T18:56:55.038 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.039 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.039 DEBUG:boto:Params: {} 2023-10-17T18:56:55.040 DEBUG:boto:Token: None 2023-10-17T18:56:55.040 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/ 2023-10-17T18:56:55.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T18:56:55.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T18:56:55.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa00845e07e62fbe-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.045 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:56:55.045 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:56:55.046 DEBUG:boto:Method: PUT 2023-10-17T18:56:55.046 DEBUG:boto:Path: /hbmuog-59/obj17 2023-10-17T18:56:55.047 DEBUG:boto:Data: 2023-10-17T18:56:55.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:55.048 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.048 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.049 DEBUG:boto:Params: {} 2023-10-17T18:56:55.049 DEBUG:boto:Token: None 2023-10-17T18:56:55.050 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/obj17 2023-10-17T18:56:55.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3SkwdlXJKTSlyLt3F9HUUHyZjxk= 2023-10-17T18:56:55.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3SkwdlXJKTSlyLt3F9HUUHyZjxk='} 2023-10-17T18:56:55.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004eb75eac1674b90e-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.095 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:55.095 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:55.096 DEBUG:boto:Method: HEAD 2023-10-17T18:56:55.096 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:55.097 DEBUG:boto:Data: 2023-10-17T18:56:55.097 DEBUG:boto:Headers: {} 2023-10-17T18:56:55.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.098 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.098 DEBUG:boto:Params: {} 2023-10-17T18:56:55.099 DEBUG:boto:Token: None 2023-10-17T18:56:55.099 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/ 2023-10-17T18:56:55.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T18:56:55.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T18:56:55.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000499a44e737db71dd-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.103 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:56:55.104 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:56:55.104 DEBUG:boto:Method: PUT 2023-10-17T18:56:55.104 DEBUG:boto:Path: /hbmuog-59/obj34 2023-10-17T18:56:55.105 DEBUG:boto:Data: 2023-10-17T18:56:55.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:55.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.106 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.107 DEBUG:boto:Params: {} 2023-10-17T18:56:55.107 DEBUG:boto:Token: None 2023-10-17T18:56:55.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/obj34 2023-10-17T18:56:55.108 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bmg7f26GqRUPssuvwxb7cGFMgPo= 2023-10-17T18:56:55.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bmg7f26GqRUPssuvwxb7cGFMgPo='} 2023-10-17T18:56:55.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3e2ca4aad15f273-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.115 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:55.116 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:55.117 DEBUG:boto:Method: HEAD 2023-10-17T18:56:55.117 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:55.118 DEBUG:boto:Data: 2023-10-17T18:56:55.118 DEBUG:boto:Headers: {} 2023-10-17T18:56:55.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.119 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.119 DEBUG:boto:Params: {} 2023-10-17T18:56:55.120 DEBUG:boto:Token: None 2023-10-17T18:56:55.120 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/ 2023-10-17T18:56:55.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T18:56:55.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T18:56:55.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006581bcbb3b15ef6a-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.125 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:56:55.125 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:56:55.125 DEBUG:boto:Method: PUT 2023-10-17T18:56:55.126 DEBUG:boto:Path: /hbmuog-59/obj51 2023-10-17T18:56:55.126 DEBUG:boto:Data: 2023-10-17T18:56:55.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:55.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.128 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.128 DEBUG:boto:Params: {} 2023-10-17T18:56:55.128 DEBUG:boto:Token: None 2023-10-17T18:56:55.129 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/obj51 2023-10-17T18:56:55.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ibt3dzDMCj/HEZoS1ndI8aPuiII= 2023-10-17T18:56:55.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ibt3dzDMCj/HEZoS1ndI8aPuiII='} 2023-10-17T18:56:55.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d193ebf346935e3-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.175 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:55.175 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:55.176 DEBUG:boto:Method: HEAD 2023-10-17T18:56:55.176 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:55.177 DEBUG:boto:Data: 2023-10-17T18:56:55.177 DEBUG:boto:Headers: {} 2023-10-17T18:56:55.177 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.178 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.178 DEBUG:boto:Params: {} 2023-10-17T18:56:55.179 DEBUG:boto:Token: None 2023-10-17T18:56:55.179 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/ 2023-10-17T18:56:55.180 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T18:56:55.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T18:56:55.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e746dabc1c7cd24-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.183 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:56:55.184 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:56:55.184 DEBUG:boto:Method: PUT 2023-10-17T18:56:55.185 DEBUG:boto:Path: /hbmuog-59/obj68 2023-10-17T18:56:55.185 DEBUG:boto:Data: 2023-10-17T18:56:55.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:55.186 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.186 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.187 DEBUG:boto:Params: {} 2023-10-17T18:56:55.187 DEBUG:boto:Token: None 2023-10-17T18:56:55.188 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/obj68 2023-10-17T18:56:55.188 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s0+j0GIcLrYk5x80btP+IpM6UEo= 2023-10-17T18:56:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s0+j0GIcLrYk5x80btP+IpM6UEo='} 2023-10-17T18:56:55.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a02a88bd1046ef1c-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.194 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:56:55.195 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:56:55.195 DEBUG:boto:Method: HEAD 2023-10-17T18:56:55.196 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:56:55.196 DEBUG:boto:Data: 2023-10-17T18:56:55.197 DEBUG:boto:Headers: {} 2023-10-17T18:56:55.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.197 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.198 DEBUG:boto:Params: {} 2023-10-17T18:56:55.198 DEBUG:boto:Token: None 2023-10-17T18:56:55.199 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/ 2023-10-17T18:56:55.199 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T18:56:55.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T18:56:55.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021838152fb3180aa-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.203 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:56:55.203 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:56:55.204 DEBUG:boto:Method: PUT 2023-10-17T18:56:55.204 DEBUG:boto:Path: /hbmuog-59/obj85 2023-10-17T18:56:55.205 DEBUG:boto:Data: 2023-10-17T18:56:55.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:56:55.205 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:56:55.206 DEBUG:boto:Port: 8000 2023-10-17T18:56:55.206 DEBUG:boto:Params: {} 2023-10-17T18:56:55.207 DEBUG:boto:Token: None 2023-10-17T18:56:55.207 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:56:55 GMT /hbmuog-59/obj85 2023-10-17T18:56:55.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:guQP2slNtf1AMh+rmim8Sv4pJFU= 2023-10-17T18:56:55.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:guQP2slNtf1AMh+rmim8Sv4pJFU='} 2023-10-17T18:56:55.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce11d71e323fec88-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:56:55.251 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T18:56:55.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:55.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:56:55.253 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:56:55.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:56:55.311 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:56:55.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:56:55.347+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:57:00.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:00.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:57:05.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:05.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:57:10.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:10.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T18:57:15.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:15.350+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:15.352 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T18:57:15.352 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:20.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:20.352+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:20.354 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T18:57:20.354 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:25.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:25.353+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:25.355 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T18:57:25.355 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:30.356 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:30.356+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:30.356 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T18:57:30.357 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:35.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:35.357+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:35.358 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:57:35.358 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:40.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:40.357+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:40.359 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:57:40.360 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:45.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:45.358+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:45.360 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:57:45.360 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:50.360+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:50.362 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:57:50.362 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:57:55.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:57:55.361+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:57:55.363 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:57:55.363 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:00.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:00.363+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:00.365 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:58:00.365 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:05.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:05.365+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:05.366 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:58:05.367 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:10.367+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:10.368 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , 00000000001.60.6, , , ] 2023-10-17T18:58:10.368 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:15.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:15.369+0000 7f4bc9f56780 1 bucket sync caught up with source: 2023-10-17T18:58:15.369 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:15.370 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T18:58:15.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:15.369+0000 7f4bc9f56780 0 bucket checkpoint complete 2023-10-17T18:58:15.387 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-59 2023-10-17T18:58:15.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:58:15.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:58:15.389 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:58:15.454 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:58:15.455 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:58:15.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:15.494+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:15.494 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T18:58:15.495 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:20.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:20.494+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:20.496 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T18:58:20.497 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:25.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:25.496+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:25.498 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T18:58:25.498 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:30.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:30.497+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:30.499 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T18:58:30.499 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:35.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:35.498+0000 7fd26ed66780 1 bucket sync caught up with source: 2023-10-17T18:58:35.500 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:35.500 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T18:58:35.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:35.498+0000 7fd26ed66780 0 bucket checkpoint complete 2023-10-17T18:58:35.522 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-59 zones={a2, a1} 2023-10-17T18:58:35.523 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.524 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.525 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.526 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:35.526 DEBUG:boto:Data: 2023-10-17T18:58:35.527 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.529 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.529 DEBUG:boto:Params: {} 2023-10-17T18:58:35.531 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T18:58:35.532 DEBUG:boto:Token: None 2023-10-17T18:58:35.533 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.534 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0= 2023-10-17T18:58:35.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0=', 'Content-Length': '0'} 2023-10-17T18:58:35.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb8177c373b87688-00652ed95b-1b46-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.541 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.542 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.542 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.543 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:35.544 DEBUG:boto:Data: 2023-10-17T18:58:35.544 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.545 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.545 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.546 DEBUG:boto:Params: {} 2023-10-17T18:58:35.547 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T18:58:35.548 DEBUG:boto:Token: None 2023-10-17T18:58:35.548 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0= 2023-10-17T18:58:35.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0=', 'Content-Length': '0'} 2023-10-17T18:58:35.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e853f1499cc01c30-00652ed95b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T18:58:35.555 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.555 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.555 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.556 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.556 DEBUG:boto:Method: GET 2023-10-17T18:58:35.557 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.557 DEBUG:boto:Data: 2023-10-17T18:58:35.558 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.559 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.559 DEBUG:boto:Params: {} 2023-10-17T18:58:35.560 DEBUG:boto:Token: None 2023-10-17T18:58:35.560 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/?versions 2023-10-17T18:58:35.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T18:58:35.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T18:58:35.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b33c10dd8d1c5b-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.566 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T18:58:35.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-17T18:58:35.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T18:58:35.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T18:58:35.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-17T18:58:35.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T18:58:35.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-17T18:58:35.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T18:58:35.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-17T18:58:35.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-17T18:58:35.572 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T18:58:35.573 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.573 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.574 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.574 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.574 DEBUG:boto:Method: GET 2023-10-17T18:58:35.575 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.575 DEBUG:boto:Data: 2023-10-17T18:58:35.576 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.577 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.578 DEBUG:boto:Params: {} 2023-10-17T18:58:35.579 DEBUG:boto:Token: None 2023-10-17T18:58:35.580 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/?versions 2023-10-17T18:58:35.580 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T18:58:35.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T18:58:35.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db987f21832655eb-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.585 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T18:58:35.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-17T18:58:35.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T18:58:35.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T18:58:35.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-17T18:58:35.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T18:58:35.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-17T18:58:35.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T18:58:35.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-17T18:58:35.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-17T18:58:35.592 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.592 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.593 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.594 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.594 DEBUG:boto:Method: GET 2023-10-17T18:58:35.595 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.596 DEBUG:boto:Data: 2023-10-17T18:58:35.596 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.597 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.597 DEBUG:boto:Params: {} 2023-10-17T18:58:35.598 DEBUG:boto:Token: None 2023-10-17T18:58:35.598 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/?versions 2023-10-17T18:58:35.599 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T18:58:35.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T18:58:35.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a9c0884297d5c0-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.602 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.603 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.604 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.604 DEBUG:boto:path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.605 DEBUG:boto:auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.605 DEBUG:boto:Method: GET 2023-10-17T18:58:35.606 DEBUG:boto:Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T18:58:35.606 DEBUG:boto:Data: 2023-10-17T18:58:35.606 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.607 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.608 DEBUG:boto:Params: {} 2023-10-17T18:58:35.608 DEBUG:boto:Token: None 2023-10-17T18:58:35.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/?versions 2023-10-17T18:58:35.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T18:58:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T18:58:35.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8da6364700b1b0-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.613 DEBUG:boto:b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:58:35.614 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.615 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.615 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.616 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.616 DEBUG:boto:Method: GET 2023-10-17T18:58:35.617 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.617 DEBUG:boto:Data: 2023-10-17T18:58:35.617 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.618 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.618 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.619 DEBUG:boto:Params: {} 2023-10-17T18:58:35.619 DEBUG:boto:Token: None 2023-10-17T18:58:35.620 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T18:58:35.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1edf8943516eb8-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.625 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.626 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.627 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.628 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.628 DEBUG:boto:Method: GET 2023-10-17T18:58:35.629 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.630 DEBUG:boto:Data: 2023-10-17T18:58:35.630 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.631 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.631 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.632 DEBUG:boto:Params: {} 2023-10-17T18:58:35.632 DEBUG:boto:Token: None 2023-10-17T18:58:35.632 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.633 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T18:58:35.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6685ef802c71391-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.636 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.636 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.637 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.637 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.637 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.638 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.638 DEBUG:boto:Data: 2023-10-17T18:58:35.639 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.639 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.639 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.640 DEBUG:boto:Params: {} 2023-10-17T18:58:35.640 DEBUG:boto:Token: None 2023-10-17T18:58:35.641 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.641 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI= 2023-10-17T18:58:35.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI=', 'Content-Length': '0'} 2023-10-17T18:58:35.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea338c4c13e5abe-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.644 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.644 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.645 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.645 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.646 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.646 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.646 DEBUG:boto:Data: 2023-10-17T18:58:35.647 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.647 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.647 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.648 DEBUG:boto:Params: {} 2023-10-17T18:58:35.648 DEBUG:boto:Token: None 2023-10-17T18:58:35.649 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.649 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI= 2023-10-17T18:58:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI=', 'Content-Length': '0'} 2023-10-17T18:58:35.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a686b9bf142489b-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:58:35.653 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.653 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.654 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.654 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.655 DEBUG:boto:Method: GET 2023-10-17T18:58:35.655 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.656 DEBUG:boto:Data: 2023-10-17T18:58:35.657 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.657 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.658 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.659 DEBUG:boto:Params: {} 2023-10-17T18:58:35.660 DEBUG:boto:Token: None 2023-10-17T18:58:35.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T18:58:35.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096466aa678f06d50-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.664 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.664 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.664 DEBUG:boto:path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.665 DEBUG:boto:auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.665 DEBUG:boto:Method: GET 2023-10-17T18:58:35.666 DEBUG:boto:Path: /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.666 DEBUG:boto:Data: 2023-10-17T18:58:35.666 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.667 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.667 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.668 DEBUG:boto:Params: {} 2023-10-17T18:58:35.668 DEBUG:boto:Token: None 2023-10-17T18:58:35.668 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11?versionId=null 2023-10-17T18:58:35.669 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T18:58:35.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca807ce5c910eb0a-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.672 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.672 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.673 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.673 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.673 DEBUG:boto:Method: GET 2023-10-17T18:58:35.674 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.674 DEBUG:boto:Data: 2023-10-17T18:58:35.675 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.675 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.675 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.676 DEBUG:boto:Params: {} 2023-10-17T18:58:35.676 DEBUG:boto:Token: None 2023-10-17T18:58:35.677 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.677 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b985ed6529481a4-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.681 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.682 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.682 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.683 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.683 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.683 DEBUG:boto:Method: GET 2023-10-17T18:58:35.684 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.684 DEBUG:boto:Data: 2023-10-17T18:58:35.685 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.685 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.685 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.686 DEBUG:boto:Params: {} 2023-10-17T18:58:35.686 DEBUG:boto:Token: None 2023-10-17T18:58:35.687 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.687 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e8388b000f7e369-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.734 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.735 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.735 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.736 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.736 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:58:35.737 DEBUG:boto:Data: 2023-10-17T18:58:35.737 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.738 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.738 DEBUG:boto:Params: {} 2023-10-17T18:58:35.739 DEBUG:boto:Token: None 2023-10-17T18:58:35.739 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11 2023-10-17T18:58:35.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM= 2023-10-17T18:58:35.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM=', 'Content-Length': '0'} 2023-10-17T18:58:35.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc46d02148fc6e8f-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.743 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.743 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.743 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.744 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:58:35.744 DEBUG:boto:Data: 2023-10-17T18:58:35.744 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.745 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.745 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.745 DEBUG:boto:Params: {} 2023-10-17T18:58:35.746 DEBUG:boto:Token: None 2023-10-17T18:58:35.746 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11 2023-10-17T18:58:35.747 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM= 2023-10-17T18:58:35.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM=', 'Content-Length': '0'} 2023-10-17T18:58:35.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73e5fce63122cef-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T18:58:35.750 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.750 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.751 DEBUG:boto:Method: GET 2023-10-17T18:58:35.751 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:58:35.752 DEBUG:boto:Data: 2023-10-17T18:58:35.752 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.752 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.753 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.753 DEBUG:boto:Params: {} 2023-10-17T18:58:35.753 DEBUG:boto:Token: None 2023-10-17T18:58:35.754 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11 2023-10-17T18:58:35.754 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k= 2023-10-17T18:58:35.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k=', 'Content-Length': '0'} 2023-10-17T18:58:35.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001381fd6928405102-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.757 DEBUG:boto:path=/hbmuog-59/obj11 2023-10-17T18:58:35.758 DEBUG:boto:auth_path=/hbmuog-59/obj11 2023-10-17T18:58:35.758 DEBUG:boto:Method: GET 2023-10-17T18:58:35.759 DEBUG:boto:Path: /hbmuog-59/obj11 2023-10-17T18:58:35.759 DEBUG:boto:Data: 2023-10-17T18:58:35.759 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.760 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.760 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.761 DEBUG:boto:Params: {} 2023-10-17T18:58:35.761 DEBUG:boto:Token: None 2023-10-17T18:58:35.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj11 2023-10-17T18:58:35.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k= 2023-10-17T18:58:35.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k=', 'Content-Length': '0'} 2023-10-17T18:58:35.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67889a90d4ca1eb-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.765 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.766 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.766 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.766 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.767 DEBUG:boto:Method: GET 2023-10-17T18:58:35.767 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.768 DEBUG:boto:Data: 2023-10-17T18:58:35.768 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.769 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.769 DEBUG:boto:Params: {} 2023-10-17T18:58:35.769 DEBUG:boto:Token: None 2023-10-17T18:58:35.770 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.771 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b607a18b85e1dcba-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.774 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.775 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.775 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.776 DEBUG:boto:path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.776 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.777 DEBUG:boto:Method: GET 2023-10-17T18:58:35.777 DEBUG:boto:Path: /hbmuog-59/?prefix=obj11 2023-10-17T18:58:35.778 DEBUG:boto:Data: 2023-10-17T18:58:35.778 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.778 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.779 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.779 DEBUG:boto:Params: {} 2023-10-17T18:58:35.780 DEBUG:boto:Token: None 2023-10-17T18:58:35.780 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059fc9fd148cfc9f5-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.784 DEBUG:boto:b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T18:58:35.785 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.786 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.786 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.787 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.787 DEBUG:boto:Method: GET 2023-10-17T18:58:35.788 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.788 DEBUG:boto:Data: 2023-10-17T18:58:35.788 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.789 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.789 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.790 DEBUG:boto:Params: {} 2023-10-17T18:58:35.790 DEBUG:boto:Token: None 2023-10-17T18:58:35.791 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.791 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T18:58:35.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T18:58:35.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb0950abd4fc22c-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.794 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.795 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.795 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.795 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.796 DEBUG:boto:Method: GET 2023-10-17T18:58:35.796 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.797 DEBUG:boto:Data: 2023-10-17T18:58:35.797 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.798 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.798 DEBUG:boto:Params: {} 2023-10-17T18:58:35.799 DEBUG:boto:Token: None 2023-10-17T18:58:35.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T18:58:35.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T18:58:35.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f399fa2935895881-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.803 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.803 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.804 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.804 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.805 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.805 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.805 DEBUG:boto:Data: 2023-10-17T18:58:35.806 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.807 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.807 DEBUG:boto:Params: {} 2023-10-17T18:58:35.808 DEBUG:boto:Token: None 2023-10-17T18:58:35.808 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8= 2023-10-17T18:58:35.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8=', 'Content-Length': '0'} 2023-10-17T18:58:35.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde8c36e014c97c8-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.812 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.813 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.813 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.813 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.814 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.814 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.815 DEBUG:boto:Data: 2023-10-17T18:58:35.815 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.815 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.816 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.816 DEBUG:boto:Params: {} 2023-10-17T18:58:35.817 DEBUG:boto:Token: None 2023-10-17T18:58:35.817 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.818 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8= 2023-10-17T18:58:35.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8=', 'Content-Length': '0'} 2023-10-17T18:58:35.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d09bd8b00cf9cac-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T18:58:35.821 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.821 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.822 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.822 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.823 DEBUG:boto:Method: GET 2023-10-17T18:58:35.823 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.824 DEBUG:boto:Data: 2023-10-17T18:58:35.824 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.824 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.825 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.825 DEBUG:boto:Params: {} 2023-10-17T18:58:35.826 DEBUG:boto:Token: None 2023-10-17T18:58:35.826 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.827 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T18:58:35.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T18:58:35.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f231be7b788f6e-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.830 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.831 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.831 DEBUG:boto:path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.832 DEBUG:boto:auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.832 DEBUG:boto:Method: GET 2023-10-17T18:58:35.832 DEBUG:boto:Path: /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.833 DEBUG:boto:Data: 2023-10-17T18:58:35.833 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.834 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.834 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.835 DEBUG:boto:Params: {} 2023-10-17T18:58:35.835 DEBUG:boto:Token: None 2023-10-17T18:58:35.836 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17?versionId=null 2023-10-17T18:58:35.836 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T18:58:35.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T18:58:35.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd77547394b93930-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.886 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.887 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.888 DEBUG:boto:path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.888 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.889 DEBUG:boto:Method: GET 2023-10-17T18:58:35.889 DEBUG:boto:Path: /hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.889 DEBUG:boto:Data: 2023-10-17T18:58:35.890 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.891 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.891 DEBUG:boto:Params: {} 2023-10-17T18:58:35.892 DEBUG:boto:Token: None 2023-10-17T18:58:35.892 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffbc3c573f4f9a2-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.896 DEBUG:boto:b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.897 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.897 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.898 DEBUG:boto:path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.898 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.899 DEBUG:boto:Method: GET 2023-10-17T18:58:35.899 DEBUG:boto:Path: /hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.900 DEBUG:boto:Data: 2023-10-17T18:58:35.900 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.901 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.901 DEBUG:boto:Params: {} 2023-10-17T18:58:35.902 DEBUG:boto:Token: None 2023-10-17T18:58:35.902 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.903 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b97fe2a60ffe7cc-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.907 DEBUG:boto:b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.908 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.908 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.908 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.909 DEBUG:boto:Path: /hbmuog-59/obj17 2023-10-17T18:58:35.909 DEBUG:boto:Data: 2023-10-17T18:58:35.910 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.910 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.911 DEBUG:boto:Params: {} 2023-10-17T18:58:35.911 DEBUG:boto:Token: None 2023-10-17T18:58:35.912 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17 2023-10-17T18:58:35.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM= 2023-10-17T18:58:35.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM=', 'Content-Length': '0'} 2023-10-17T18:58:35.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e97ca148471cce-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.915 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.915 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.916 DEBUG:boto:Method: HEAD 2023-10-17T18:58:35.916 DEBUG:boto:Path: /hbmuog-59/obj17 2023-10-17T18:58:35.916 DEBUG:boto:Data: 2023-10-17T18:58:35.917 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.917 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.918 DEBUG:boto:Params: {} 2023-10-17T18:58:35.918 DEBUG:boto:Token: None 2023-10-17T18:58:35.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17 2023-10-17T18:58:35.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM= 2023-10-17T18:58:35.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM=', 'Content-Length': '0'} 2023-10-17T18:58:35.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6910e0da035903-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T18:58:35.923 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.923 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.924 DEBUG:boto:Method: GET 2023-10-17T18:58:35.924 DEBUG:boto:Path: /hbmuog-59/obj17 2023-10-17T18:58:35.924 DEBUG:boto:Data: 2023-10-17T18:58:35.925 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.925 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.926 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.926 DEBUG:boto:Params: {} 2023-10-17T18:58:35.927 DEBUG:boto:Token: None 2023-10-17T18:58:35.927 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17 2023-10-17T18:58:35.928 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c= 2023-10-17T18:58:35.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001923d72c9b789572-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.970 DEBUG:boto:path=/hbmuog-59/obj17 2023-10-17T18:58:35.971 DEBUG:boto:auth_path=/hbmuog-59/obj17 2023-10-17T18:58:35.971 DEBUG:boto:Method: GET 2023-10-17T18:58:35.972 DEBUG:boto:Path: /hbmuog-59/obj17 2023-10-17T18:58:35.972 DEBUG:boto:Data: 2023-10-17T18:58:35.973 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.973 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.973 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.974 DEBUG:boto:Params: {} 2023-10-17T18:58:35.974 DEBUG:boto:Token: None 2023-10-17T18:58:35.975 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/obj17 2023-10-17T18:58:35.975 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c= 2023-10-17T18:58:35.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c=', 'Content-Length': '0'} 2023-10-17T18:58:35.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609f6d9b832deccd-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.979 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.979 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.979 DEBUG:boto:path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.980 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.980 DEBUG:boto:Method: GET 2023-10-17T18:58:35.980 DEBUG:boto:Path: /hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.981 DEBUG:boto:Data: 2023-10-17T18:58:35.981 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:35.982 DEBUG:boto:Port: 8001 2023-10-17T18:58:35.982 DEBUG:boto:Params: {} 2023-10-17T18:58:35.983 DEBUG:boto:Token: None 2023-10-17T18:58:35.983 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3dcd6bcbfbc3370-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.988 DEBUG:boto:b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.988 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:35.989 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:35.989 DEBUG:boto:path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.990 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.990 DEBUG:boto:Method: GET 2023-10-17T18:58:35.991 DEBUG:boto:Path: /hbmuog-59/?prefix=obj17 2023-10-17T18:58:35.991 DEBUG:boto:Data: 2023-10-17T18:58:35.991 DEBUG:boto:Headers: {} 2023-10-17T18:58:35.992 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:35.992 DEBUG:boto:Port: 8000 2023-10-17T18:58:35.993 DEBUG:boto:Params: {} 2023-10-17T18:58:35.993 DEBUG:boto:Token: None 2023-10-17T18:58:35.994 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:35 GMT /hbmuog-59/ 2023-10-17T18:58:35.994 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T18:58:35.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T18:58:35.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7b95cd2d64fb03-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:35.998 DEBUG:boto:b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:35.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:58:36.000 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.000 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.000 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.001 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.001 DEBUG:boto:Method: GET 2023-10-17T18:58:36.001 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.002 DEBUG:boto:Data: 2023-10-17T18:58:36.002 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.003 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.003 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.003 DEBUG:boto:Params: {} 2023-10-17T18:58:36.004 DEBUG:boto:Token: None 2023-10-17T18:58:36.004 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T18:58:36.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T18:58:36.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000695d450faf4e994c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.008 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.008 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.009 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.009 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.010 DEBUG:boto:Method: GET 2023-10-17T18:58:36.010 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.010 DEBUG:boto:Data: 2023-10-17T18:58:36.011 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.011 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.012 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.012 DEBUG:boto:Params: {} 2023-10-17T18:58:36.012 DEBUG:boto:Token: None 2023-10-17T18:58:36.013 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.013 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T18:58:36.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T18:58:36.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed07b73276fd915d-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.017 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.017 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.018 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.018 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.018 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.019 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.019 DEBUG:boto:Data: 2023-10-17T18:58:36.020 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.020 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.020 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.021 DEBUG:boto:Params: {} 2023-10-17T18:58:36.021 DEBUG:boto:Token: None 2023-10-17T18:58:36.022 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.022 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8= 2023-10-17T18:58:36.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8=', 'Content-Length': '0'} 2023-10-17T18:58:36.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b315fe3b8576b495-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.026 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.027 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.027 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.028 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.028 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.029 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.029 DEBUG:boto:Data: 2023-10-17T18:58:36.029 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.030 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.031 DEBUG:boto:Params: {} 2023-10-17T18:58:36.031 DEBUG:boto:Token: None 2023-10-17T18:58:36.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.032 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8= 2023-10-17T18:58:36.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8=', 'Content-Length': '0'} 2023-10-17T18:58:36.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01bd65d26c1564e-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:58:36.036 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.037 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.037 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.038 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.038 DEBUG:boto:Method: GET 2023-10-17T18:58:36.038 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.039 DEBUG:boto:Data: 2023-10-17T18:58:36.039 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.040 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.040 DEBUG:boto:Params: {} 2023-10-17T18:58:36.041 DEBUG:boto:Token: None 2023-10-17T18:58:36.041 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.042 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T18:58:36.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T18:58:36.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629c249767dd8609-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.086 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.087 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.087 DEBUG:boto:path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.088 DEBUG:boto:auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.088 DEBUG:boto:Method: GET 2023-10-17T18:58:36.088 DEBUG:boto:Path: /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.089 DEBUG:boto:Data: 2023-10-17T18:58:36.089 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.090 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.091 DEBUG:boto:Params: {} 2023-10-17T18:58:36.091 DEBUG:boto:Token: None 2023-10-17T18:58:36.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22?versionId=null 2023-10-17T18:58:36.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T18:58:36.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T18:58:36.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb30b59008fff8d2-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.095 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.095 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.096 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.096 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.096 DEBUG:boto:Method: GET 2023-10-17T18:58:36.097 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.097 DEBUG:boto:Data: 2023-10-17T18:58:36.098 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.098 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.099 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.099 DEBUG:boto:Params: {} 2023-10-17T18:58:36.099 DEBUG:boto:Token: None 2023-10-17T18:58:36.100 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7e975fc410b049-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.104 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.105 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.105 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.106 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.106 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.107 DEBUG:boto:Method: GET 2023-10-17T18:58:36.107 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.108 DEBUG:boto:Data: 2023-10-17T18:58:36.108 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.110 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.110 DEBUG:boto:Params: {} 2023-10-17T18:58:36.111 DEBUG:boto:Token: None 2023-10-17T18:58:36.112 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554a75144170b382-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.158 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.159 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.160 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.160 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.161 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:58:36.161 DEBUG:boto:Data: 2023-10-17T18:58:36.162 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.162 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.162 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.163 DEBUG:boto:Params: {} 2023-10-17T18:58:36.163 DEBUG:boto:Token: None 2023-10-17T18:58:36.164 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22 2023-10-17T18:58:36.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o= 2023-10-17T18:58:36.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o=', 'Content-Length': '0'} 2023-10-17T18:58:36.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e20013195dd4f01-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.167 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.168 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.168 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.169 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:58:36.169 DEBUG:boto:Data: 2023-10-17T18:58:36.169 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.170 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.171 DEBUG:boto:Params: {} 2023-10-17T18:58:36.171 DEBUG:boto:Token: None 2023-10-17T18:58:36.172 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22 2023-10-17T18:58:36.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o= 2023-10-17T18:58:36.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o=', 'Content-Length': '0'} 2023-10-17T18:58:36.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb4457affab5de1-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T18:58:36.178 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.179 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.179 DEBUG:boto:Method: GET 2023-10-17T18:58:36.179 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:58:36.180 DEBUG:boto:Data: 2023-10-17T18:58:36.180 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.181 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.181 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.181 DEBUG:boto:Params: {} 2023-10-17T18:58:36.182 DEBUG:boto:Token: None 2023-10-17T18:58:36.182 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22 2023-10-17T18:58:36.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo= 2023-10-17T18:58:36.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo=', 'Content-Length': '0'} 2023-10-17T18:58:36.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e5bbc4c1807e52-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.186 DEBUG:boto:path=/hbmuog-59/obj22 2023-10-17T18:58:36.186 DEBUG:boto:auth_path=/hbmuog-59/obj22 2023-10-17T18:58:36.187 DEBUG:boto:Method: GET 2023-10-17T18:58:36.187 DEBUG:boto:Path: /hbmuog-59/obj22 2023-10-17T18:58:36.188 DEBUG:boto:Data: 2023-10-17T18:58:36.188 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.189 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.189 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.189 DEBUG:boto:Params: {} 2023-10-17T18:58:36.190 DEBUG:boto:Token: None 2023-10-17T18:58:36.191 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj22 2023-10-17T18:58:36.191 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo= 2023-10-17T18:58:36.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo=', 'Content-Length': '0'} 2023-10-17T18:58:36.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60b19cbad6b4ed4-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.195 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.195 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.196 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.196 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.196 DEBUG:boto:Method: GET 2023-10-17T18:58:36.197 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.197 DEBUG:boto:Data: 2023-10-17T18:58:36.197 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.198 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.198 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.199 DEBUG:boto:Params: {} 2023-10-17T18:58:36.199 DEBUG:boto:Token: None 2023-10-17T18:58:36.200 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.200 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2789a8b2df48105-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.204 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.205 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.205 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.205 DEBUG:boto:path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.206 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.206 DEBUG:boto:Method: GET 2023-10-17T18:58:36.207 DEBUG:boto:Path: /hbmuog-59/?prefix=obj22 2023-10-17T18:58:36.207 DEBUG:boto:Data: 2023-10-17T18:58:36.208 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.209 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.209 DEBUG:boto:Params: {} 2023-10-17T18:58:36.209 DEBUG:boto:Token: None 2023-10-17T18:58:36.210 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497218669d78d30f-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.214 DEBUG:boto:b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:58:36.215 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.216 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.216 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.217 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.217 DEBUG:boto:Method: GET 2023-10-17T18:58:36.218 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.218 DEBUG:boto:Data: 2023-10-17T18:58:36.219 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.220 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.221 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.221 DEBUG:boto:Params: {} 2023-10-17T18:58:36.222 DEBUG:boto:Token: None 2023-10-17T18:58:36.222 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.223 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T18:58:36.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T18:58:36.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082198f4c2fb9a0f5-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.266 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.267 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.267 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.268 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.268 DEBUG:boto:Method: GET 2023-10-17T18:58:36.268 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.269 DEBUG:boto:Data: 2023-10-17T18:58:36.269 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.269 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.270 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.270 DEBUG:boto:Params: {} 2023-10-17T18:58:36.271 DEBUG:boto:Token: None 2023-10-17T18:58:36.271 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.272 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T18:58:36.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T18:58:36.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc8f99d67d725a6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.275 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.275 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.275 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.276 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.276 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.276 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.277 DEBUG:boto:Data: 2023-10-17T18:58:36.277 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.278 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.278 DEBUG:boto:Params: {} 2023-10-17T18:58:36.279 DEBUG:boto:Token: None 2023-10-17T18:58:36.279 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU= 2023-10-17T18:58:36.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU=', 'Content-Length': '0'} 2023-10-17T18:58:36.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1d96c3ceb7b66c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.283 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.283 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.284 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.284 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.284 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.285 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.285 DEBUG:boto:Data: 2023-10-17T18:58:36.285 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.286 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.286 DEBUG:boto:Params: {} 2023-10-17T18:58:36.287 DEBUG:boto:Token: None 2023-10-17T18:58:36.287 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU= 2023-10-17T18:58:36.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU=', 'Content-Length': '0'} 2023-10-17T18:58:36.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb9cec84289804b-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:58:36.291 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.292 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.292 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.293 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.293 DEBUG:boto:Method: GET 2023-10-17T18:58:36.293 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.294 DEBUG:boto:Data: 2023-10-17T18:58:36.294 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.295 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.295 DEBUG:boto:Params: {} 2023-10-17T18:58:36.296 DEBUG:boto:Token: None 2023-10-17T18:58:36.296 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T18:58:36.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T18:58:36.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a270ea18bfb8bf-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.300 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.300 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.301 DEBUG:boto:path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.301 DEBUG:boto:auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.301 DEBUG:boto:Method: GET 2023-10-17T18:58:36.302 DEBUG:boto:Path: /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.302 DEBUG:boto:Data: 2023-10-17T18:58:36.303 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.304 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.304 DEBUG:boto:Params: {} 2023-10-17T18:58:36.304 DEBUG:boto:Token: None 2023-10-17T18:58:36.305 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33?versionId=null 2023-10-17T18:58:36.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T18:58:36.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T18:58:36.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c792ab48be0ef9a7-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.350 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.351 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.351 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.352 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.352 DEBUG:boto:Method: GET 2023-10-17T18:58:36.352 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.353 DEBUG:boto:Data: 2023-10-17T18:58:36.353 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.354 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.354 DEBUG:boto:Params: {} 2023-10-17T18:58:36.355 DEBUG:boto:Token: None 2023-10-17T18:58:36.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.356 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a88d2438f2d0e4-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.359 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.360 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.361 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.361 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.361 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.362 DEBUG:boto:Method: GET 2023-10-17T18:58:36.362 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.363 DEBUG:boto:Data: 2023-10-17T18:58:36.363 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.364 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.364 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.365 DEBUG:boto:Params: {} 2023-10-17T18:58:36.365 DEBUG:boto:Token: None 2023-10-17T18:58:36.366 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008bb8be87eceb550-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.372 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.372 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.373 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.373 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.374 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:58:36.374 DEBUG:boto:Data: 2023-10-17T18:58:36.374 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.375 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.375 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.376 DEBUG:boto:Params: {} 2023-10-17T18:58:36.376 DEBUG:boto:Token: None 2023-10-17T18:58:36.376 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33 2023-10-17T18:58:36.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc= 2023-10-17T18:58:36.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc=', 'Content-Length': '0'} 2023-10-17T18:58:36.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a1daa99fae9957-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.380 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.380 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.381 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.381 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:58:36.382 DEBUG:boto:Data: 2023-10-17T18:58:36.382 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.383 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.383 DEBUG:boto:Params: {} 2023-10-17T18:58:36.384 DEBUG:boto:Token: None 2023-10-17T18:58:36.384 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33 2023-10-17T18:58:36.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc= 2023-10-17T18:58:36.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc=', 'Content-Length': '0'} 2023-10-17T18:58:36.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7f1ce3de875359-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T18:58:36.388 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.388 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.389 DEBUG:boto:Method: GET 2023-10-17T18:58:36.389 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:58:36.389 DEBUG:boto:Data: 2023-10-17T18:58:36.390 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.391 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.391 DEBUG:boto:Params: {} 2023-10-17T18:58:36.392 DEBUG:boto:Token: None 2023-10-17T18:58:36.392 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33 2023-10-17T18:58:36.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo= 2023-10-17T18:58:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo=', 'Content-Length': '0'} 2023-10-17T18:58:36.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f852028dbf2017c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.442 DEBUG:boto:path=/hbmuog-59/obj33 2023-10-17T18:58:36.443 DEBUG:boto:auth_path=/hbmuog-59/obj33 2023-10-17T18:58:36.443 DEBUG:boto:Method: GET 2023-10-17T18:58:36.444 DEBUG:boto:Path: /hbmuog-59/obj33 2023-10-17T18:58:36.444 DEBUG:boto:Data: 2023-10-17T18:58:36.445 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.446 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.446 DEBUG:boto:Params: {} 2023-10-17T18:58:36.447 DEBUG:boto:Token: None 2023-10-17T18:58:36.448 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj33 2023-10-17T18:58:36.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo= 2023-10-17T18:58:36.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo=', 'Content-Length': '0'} 2023-10-17T18:58:36.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813844f625902132-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.452 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.453 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.453 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.454 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.454 DEBUG:boto:Method: GET 2023-10-17T18:58:36.455 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.455 DEBUG:boto:Data: 2023-10-17T18:58:36.456 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.457 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.457 DEBUG:boto:Params: {} 2023-10-17T18:58:36.457 DEBUG:boto:Token: None 2023-10-17T18:58:36.458 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.458 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf2912c34681aae-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.462 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.463 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.463 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.464 DEBUG:boto:path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.464 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.465 DEBUG:boto:Method: GET 2023-10-17T18:58:36.465 DEBUG:boto:Path: /hbmuog-59/?prefix=obj33 2023-10-17T18:58:36.465 DEBUG:boto:Data: 2023-10-17T18:58:36.466 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.467 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.467 DEBUG:boto:Params: {} 2023-10-17T18:58:36.468 DEBUG:boto:Token: None 2023-10-17T18:58:36.468 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb41ac52bc2d1642-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.472 DEBUG:boto:b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T18:58:36.473 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.474 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.474 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.475 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.475 DEBUG:boto:Method: GET 2023-10-17T18:58:36.476 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.476 DEBUG:boto:Data: 2023-10-17T18:58:36.476 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.477 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.478 DEBUG:boto:Params: {} 2023-10-17T18:58:36.478 DEBUG:boto:Token: None 2023-10-17T18:58:36.479 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T18:58:36.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T18:58:36.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ad23ea537027cd-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.483 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.483 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.483 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.484 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.484 DEBUG:boto:Method: GET 2023-10-17T18:58:36.485 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.485 DEBUG:boto:Data: 2023-10-17T18:58:36.486 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.487 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.487 DEBUG:boto:Params: {} 2023-10-17T18:58:36.487 DEBUG:boto:Token: None 2023-10-17T18:58:36.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T18:58:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T18:58:36.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddc24d98881c9e6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.534 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.535 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.535 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.535 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.536 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.536 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.537 DEBUG:boto:Data: 2023-10-17T18:58:36.537 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.537 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.538 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.538 DEBUG:boto:Params: {} 2023-10-17T18:58:36.539 DEBUG:boto:Token: None 2023-10-17T18:58:36.539 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro= 2023-10-17T18:58:36.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro=', 'Content-Length': '0'} 2023-10-17T18:58:36.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006350c1730857725b-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.543 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.543 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.544 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.544 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.545 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.545 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.545 DEBUG:boto:Data: 2023-10-17T18:58:36.546 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.547 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.547 DEBUG:boto:Params: {} 2023-10-17T18:58:36.547 DEBUG:boto:Token: None 2023-10-17T18:58:36.548 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro= 2023-10-17T18:58:36.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro=', 'Content-Length': '0'} 2023-10-17T18:58:36.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c56dd7c6475069-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T18:58:36.552 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.552 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.553 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.553 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.553 DEBUG:boto:Method: GET 2023-10-17T18:58:36.554 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.554 DEBUG:boto:Data: 2023-10-17T18:58:36.555 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.555 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.555 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.556 DEBUG:boto:Params: {} 2023-10-17T18:58:36.556 DEBUG:boto:Token: None 2023-10-17T18:58:36.557 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.557 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T18:58:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T18:58:36.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d501b09f6fe4e9b-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.560 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.561 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.561 DEBUG:boto:path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.561 DEBUG:boto:auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.562 DEBUG:boto:Method: GET 2023-10-17T18:58:36.562 DEBUG:boto:Path: /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.563 DEBUG:boto:Data: 2023-10-17T18:58:36.563 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.563 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.564 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.564 DEBUG:boto:Params: {} 2023-10-17T18:58:36.565 DEBUG:boto:Token: None 2023-10-17T18:58:36.565 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34?versionId=null 2023-10-17T18:58:36.565 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T18:58:36.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T18:58:36.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073e3f2b7251ce27-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.568 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.569 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.569 DEBUG:boto:path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.569 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.570 DEBUG:boto:Method: GET 2023-10-17T18:58:36.570 DEBUG:boto:Path: /hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.571 DEBUG:boto:Data: 2023-10-17T18:58:36.571 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.571 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.572 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.572 DEBUG:boto:Params: {} 2023-10-17T18:58:36.573 DEBUG:boto:Token: None 2023-10-17T18:58:36.573 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.574 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac03cdd9ab78f43-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.578 DEBUG:boto:b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.578 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.579 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.579 DEBUG:boto:path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.580 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.580 DEBUG:boto:Method: GET 2023-10-17T18:58:36.580 DEBUG:boto:Path: /hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.581 DEBUG:boto:Data: 2023-10-17T18:58:36.581 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.582 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.583 DEBUG:boto:Params: {} 2023-10-17T18:58:36.583 DEBUG:boto:Token: None 2023-10-17T18:58:36.584 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063324a00e86953a2-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.588 DEBUG:boto:b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.589 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.589 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.590 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.590 DEBUG:boto:Path: /hbmuog-59/obj34 2023-10-17T18:58:36.591 DEBUG:boto:Data: 2023-10-17T18:58:36.591 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.591 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.592 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.592 DEBUG:boto:Params: {} 2023-10-17T18:58:36.593 DEBUG:boto:Token: None 2023-10-17T18:58:36.593 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34 2023-10-17T18:58:36.593 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac= 2023-10-17T18:58:36.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac=', 'Content-Length': '0'} 2023-10-17T18:58:36.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2f4146d211edd0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.596 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.597 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.597 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.597 DEBUG:boto:Path: /hbmuog-59/obj34 2023-10-17T18:58:36.598 DEBUG:boto:Data: 2023-10-17T18:58:36.598 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.599 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.599 DEBUG:boto:Params: {} 2023-10-17T18:58:36.600 DEBUG:boto:Token: None 2023-10-17T18:58:36.600 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34 2023-10-17T18:58:36.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac= 2023-10-17T18:58:36.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac=', 'Content-Length': '0'} 2023-10-17T18:58:36.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699f9022aee3060e-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T18:58:36.604 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.604 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.605 DEBUG:boto:Method: GET 2023-10-17T18:58:36.605 DEBUG:boto:Path: /hbmuog-59/obj34 2023-10-17T18:58:36.605 DEBUG:boto:Data: 2023-10-17T18:58:36.606 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.607 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.607 DEBUG:boto:Params: {} 2023-10-17T18:58:36.607 DEBUG:boto:Token: None 2023-10-17T18:58:36.608 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34 2023-10-17T18:58:36.608 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg= 2023-10-17T18:58:36.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg=', 'Content-Length': '0'} 2023-10-17T18:58:36.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39d56e22ef42ce0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.612 DEBUG:boto:path=/hbmuog-59/obj34 2023-10-17T18:58:36.612 DEBUG:boto:auth_path=/hbmuog-59/obj34 2023-10-17T18:58:36.612 DEBUG:boto:Method: GET 2023-10-17T18:58:36.613 DEBUG:boto:Path: /hbmuog-59/obj34 2023-10-17T18:58:36.613 DEBUG:boto:Data: 2023-10-17T18:58:36.614 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.614 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.614 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.615 DEBUG:boto:Params: {} 2023-10-17T18:58:36.615 DEBUG:boto:Token: None 2023-10-17T18:58:36.616 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj34 2023-10-17T18:58:36.616 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg= 2023-10-17T18:58:36.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg=', 'Content-Length': '0'} 2023-10-17T18:58:36.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea27ad39fd47bf4-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.662 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.663 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.663 DEBUG:boto:path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.664 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.664 DEBUG:boto:Method: GET 2023-10-17T18:58:36.664 DEBUG:boto:Path: /hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.665 DEBUG:boto:Data: 2023-10-17T18:58:36.665 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.666 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.666 DEBUG:boto:Params: {} 2023-10-17T18:58:36.667 DEBUG:boto:Token: None 2023-10-17T18:58:36.667 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.668 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043fe17162d443ab2-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.672 DEBUG:boto:b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.672 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.673 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.673 DEBUG:boto:path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.673 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.674 DEBUG:boto:Method: GET 2023-10-17T18:58:36.674 DEBUG:boto:Path: /hbmuog-59/?prefix=obj34 2023-10-17T18:58:36.675 DEBUG:boto:Data: 2023-10-17T18:58:36.675 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.676 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.676 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.676 DEBUG:boto:Params: {} 2023-10-17T18:58:36.677 DEBUG:boto:Token: None 2023-10-17T18:58:36.677 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.678 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae11823fc6b8e497-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.682 DEBUG:boto:b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:58:36.683 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.684 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.684 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.684 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.685 DEBUG:boto:Method: GET 2023-10-17T18:58:36.685 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.686 DEBUG:boto:Data: 2023-10-17T18:58:36.686 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.687 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.687 DEBUG:boto:Params: {} 2023-10-17T18:58:36.688 DEBUG:boto:Token: None 2023-10-17T18:58:36.688 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.689 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T18:58:36.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T18:58:36.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a092e68f9674c5e0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.734 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.735 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.736 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.736 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.737 DEBUG:boto:Method: GET 2023-10-17T18:58:36.737 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.737 DEBUG:boto:Data: 2023-10-17T18:58:36.738 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.739 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.739 DEBUG:boto:Params: {} 2023-10-17T18:58:36.740 DEBUG:boto:Token: None 2023-10-17T18:58:36.740 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T18:58:36.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T18:58:36.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6b09b6c6e9d3b9-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.744 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.745 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.745 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.746 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.746 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.746 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.747 DEBUG:boto:Data: 2023-10-17T18:58:36.747 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.748 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.748 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.749 DEBUG:boto:Params: {} 2023-10-17T18:58:36.749 DEBUG:boto:Token: None 2023-10-17T18:58:36.750 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.750 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw= 2023-10-17T18:58:36.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw=', 'Content-Length': '0'} 2023-10-17T18:58:36.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50b828e19251d87-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.753 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.753 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.754 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.754 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.755 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.755 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.756 DEBUG:boto:Data: 2023-10-17T18:58:36.756 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.757 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.757 DEBUG:boto:Params: {} 2023-10-17T18:58:36.757 DEBUG:boto:Token: None 2023-10-17T18:58:36.758 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw= 2023-10-17T18:58:36.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw=', 'Content-Length': '0'} 2023-10-17T18:58:36.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047367c69776b42a3-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:58:36.762 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.762 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.763 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.763 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.764 DEBUG:boto:Method: GET 2023-10-17T18:58:36.764 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.764 DEBUG:boto:Data: 2023-10-17T18:58:36.765 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.766 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.766 DEBUG:boto:Params: {} 2023-10-17T18:58:36.767 DEBUG:boto:Token: None 2023-10-17T18:58:36.767 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T18:58:36.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T18:58:36.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2a2b5822614e3f-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.771 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.771 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.772 DEBUG:boto:path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.772 DEBUG:boto:auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.772 DEBUG:boto:Method: GET 2023-10-17T18:58:36.773 DEBUG:boto:Path: /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.773 DEBUG:boto:Data: 2023-10-17T18:58:36.774 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.775 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.775 DEBUG:boto:Params: {} 2023-10-17T18:58:36.776 DEBUG:boto:Token: None 2023-10-17T18:58:36.776 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44?versionId=null 2023-10-17T18:58:36.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T18:58:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T18:58:36.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fff54a8483b2c6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.780 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.781 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.781 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.781 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.782 DEBUG:boto:Method: GET 2023-10-17T18:58:36.782 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.783 DEBUG:boto:Data: 2023-10-17T18:58:36.783 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.784 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.784 DEBUG:boto:Params: {} 2023-10-17T18:58:36.785 DEBUG:boto:Token: None 2023-10-17T18:58:36.785 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.786 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003326dd232c658ce1-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.790 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.790 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.791 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.791 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.792 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.792 DEBUG:boto:Method: GET 2023-10-17T18:58:36.792 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.793 DEBUG:boto:Data: 2023-10-17T18:58:36.793 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.794 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.794 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.794 DEBUG:boto:Params: {} 2023-10-17T18:58:36.795 DEBUG:boto:Token: None 2023-10-17T18:58:36.795 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.796 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d3d83556dff240b-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.800 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.800 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.801 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.801 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.801 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:58:36.802 DEBUG:boto:Data: 2023-10-17T18:58:36.802 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.803 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.803 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.803 DEBUG:boto:Params: {} 2023-10-17T18:58:36.804 DEBUG:boto:Token: None 2023-10-17T18:58:36.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44 2023-10-17T18:58:36.805 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg= 2023-10-17T18:58:36.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg=', 'Content-Length': '0'} 2023-10-17T18:58:36.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253e01ebfee8fbb7-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.808 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.808 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.809 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.809 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:58:36.809 DEBUG:boto:Data: 2023-10-17T18:58:36.810 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.811 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.811 DEBUG:boto:Params: {} 2023-10-17T18:58:36.811 DEBUG:boto:Token: None 2023-10-17T18:58:36.812 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44 2023-10-17T18:58:36.812 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg= 2023-10-17T18:58:36.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg=', 'Content-Length': '0'} 2023-10-17T18:58:36.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aae61fc328e4e92-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T18:58:36.815 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.816 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.816 DEBUG:boto:Method: GET 2023-10-17T18:58:36.817 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:58:36.817 DEBUG:boto:Data: 2023-10-17T18:58:36.818 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.818 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.819 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.819 DEBUG:boto:Params: {} 2023-10-17T18:58:36.820 DEBUG:boto:Token: None 2023-10-17T18:58:36.820 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44 2023-10-17T18:58:36.821 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw= 2023-10-17T18:58:36.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw=', 'Content-Length': '0'} 2023-10-17T18:58:36.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34191b03e83e185-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.823 DEBUG:boto:path=/hbmuog-59/obj44 2023-10-17T18:58:36.824 DEBUG:boto:auth_path=/hbmuog-59/obj44 2023-10-17T18:58:36.824 DEBUG:boto:Method: GET 2023-10-17T18:58:36.825 DEBUG:boto:Path: /hbmuog-59/obj44 2023-10-17T18:58:36.825 DEBUG:boto:Data: 2023-10-17T18:58:36.826 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.827 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.827 DEBUG:boto:Params: {} 2023-10-17T18:58:36.827 DEBUG:boto:Token: None 2023-10-17T18:58:36.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj44 2023-10-17T18:58:36.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw= 2023-10-17T18:58:36.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw=', 'Content-Length': '0'} 2023-10-17T18:58:36.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe1e4612f29f8fa-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.874 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.875 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.875 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.876 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.876 DEBUG:boto:Method: GET 2023-10-17T18:58:36.876 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.877 DEBUG:boto:Data: 2023-10-17T18:58:36.877 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.878 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.878 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.879 DEBUG:boto:Params: {} 2023-10-17T18:58:36.879 DEBUG:boto:Token: None 2023-10-17T18:58:36.880 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.880 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f602144aad000d28-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.884 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.884 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.885 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.885 DEBUG:boto:path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.886 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.886 DEBUG:boto:Method: GET 2023-10-17T18:58:36.886 DEBUG:boto:Path: /hbmuog-59/?prefix=obj44 2023-10-17T18:58:36.887 DEBUG:boto:Data: 2023-10-17T18:58:36.887 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.888 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.888 DEBUG:boto:Params: {} 2023-10-17T18:58:36.889 DEBUG:boto:Token: None 2023-10-17T18:58:36.889 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:36.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:36.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:36.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde02b8739160daf-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.894 DEBUG:boto:b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:36.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T18:58:36.895 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.895 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.895 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.896 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.896 DEBUG:boto:Method: GET 2023-10-17T18:58:36.897 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.897 DEBUG:boto:Data: 2023-10-17T18:58:36.897 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.898 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.898 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.899 DEBUG:boto:Params: {} 2023-10-17T18:58:36.899 DEBUG:boto:Token: None 2023-10-17T18:58:36.900 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.900 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T18:58:36.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T18:58:36.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9ebb56f5c17331-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.950 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.951 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.951 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.952 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.952 DEBUG:boto:Method: GET 2023-10-17T18:58:36.953 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.953 DEBUG:boto:Data: 2023-10-17T18:58:36.953 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.954 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.954 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.955 DEBUG:boto:Params: {} 2023-10-17T18:58:36.955 DEBUG:boto:Token: None 2023-10-17T18:58:36.956 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.956 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T18:58:36.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T18:58:36.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b3eadfaf35aab1-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.959 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.960 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.960 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.961 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.961 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.961 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.962 DEBUG:boto:Data: 2023-10-17T18:58:36.962 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.963 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.963 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.963 DEBUG:boto:Params: {} 2023-10-17T18:58:36.964 DEBUG:boto:Token: None 2023-10-17T18:58:36.964 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.965 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U= 2023-10-17T18:58:36.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U=', 'Content-Length': '0'} 2023-10-17T18:58:36.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad0e41ea51f31b6-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.968 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.968 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.969 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.969 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.969 DEBUG:boto:Method: HEAD 2023-10-17T18:58:36.970 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.970 DEBUG:boto:Data: 2023-10-17T18:58:36.971 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.971 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.971 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.972 DEBUG:boto:Params: {} 2023-10-17T18:58:36.972 DEBUG:boto:Token: None 2023-10-17T18:58:36.973 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U= 2023-10-17T18:58:36.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U=', 'Content-Length': '0'} 2023-10-17T18:58:36.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76644d838ce2ded-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T18:58:36.977 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.977 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.978 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.978 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.979 DEBUG:boto:Method: GET 2023-10-17T18:58:36.979 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.979 DEBUG:boto:Data: 2023-10-17T18:58:36.980 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.980 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.981 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.981 DEBUG:boto:Params: {} 2023-10-17T18:58:36.981 DEBUG:boto:Token: None 2023-10-17T18:58:36.982 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.982 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T18:58:36.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T18:58:36.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016531567f7f8c09c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.986 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:36.986 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:36.987 DEBUG:boto:path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.987 DEBUG:boto:auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.987 DEBUG:boto:Method: GET 2023-10-17T18:58:36.988 DEBUG:boto:Path: /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.988 DEBUG:boto:Data: 2023-10-17T18:58:36.988 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.989 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:36.989 DEBUG:boto:Port: 8000 2023-10-17T18:58:36.990 DEBUG:boto:Params: {} 2023-10-17T18:58:36.990 DEBUG:boto:Token: None 2023-10-17T18:58:36.991 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/obj51?versionId=null 2023-10-17T18:58:36.991 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T18:58:36.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T18:58:36.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0002c7bb6d3740a-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:36.994 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:36.995 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:36.995 DEBUG:boto:path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:36.996 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:36.996 DEBUG:boto:Method: GET 2023-10-17T18:58:36.996 DEBUG:boto:Path: /hbmuog-59/?prefix=obj51 2023-10-17T18:58:36.997 DEBUG:boto:Data: 2023-10-17T18:58:36.997 DEBUG:boto:Headers: {} 2023-10-17T18:58:36.998 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:36.998 DEBUG:boto:Port: 8001 2023-10-17T18:58:36.998 DEBUG:boto:Params: {} 2023-10-17T18:58:36.999 DEBUG:boto:Token: None 2023-10-17T18:58:36.999 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:36 GMT /hbmuog-59/ 2023-10-17T18:58:37.000 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T18:58:37.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T18:58:37.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ada77c8ee8686d-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.004 DEBUG:boto:b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.005 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.005 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.006 DEBUG:boto:path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.006 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.006 DEBUG:boto:Method: GET 2023-10-17T18:58:37.007 DEBUG:boto:Path: /hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.007 DEBUG:boto:Data: 2023-10-17T18:58:37.008 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.008 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.009 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.009 DEBUG:boto:Params: {} 2023-10-17T18:58:37.009 DEBUG:boto:Token: None 2023-10-17T18:58:37.010 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.010 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008623119eb8298872-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.014 DEBUG:boto:b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.015 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:37.015 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:37.016 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.016 DEBUG:boto:Path: /hbmuog-59/obj51 2023-10-17T18:58:37.017 DEBUG:boto:Data: 2023-10-17T18:58:37.017 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.018 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.018 DEBUG:boto:Params: {} 2023-10-17T18:58:37.019 DEBUG:boto:Token: None 2023-10-17T18:58:37.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj51 2023-10-17T18:58:37.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g= 2023-10-17T18:58:37.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g=', 'Content-Length': '0'} 2023-10-17T18:58:37.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c73fd298413a04cd-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.022 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:37.023 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:37.023 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.024 DEBUG:boto:Path: /hbmuog-59/obj51 2023-10-17T18:58:37.024 DEBUG:boto:Data: 2023-10-17T18:58:37.024 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.025 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.025 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.026 DEBUG:boto:Params: {} 2023-10-17T18:58:37.026 DEBUG:boto:Token: None 2023-10-17T18:58:37.027 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj51 2023-10-17T18:58:37.027 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g= 2023-10-17T18:58:37.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g=', 'Content-Length': '0'} 2023-10-17T18:58:37.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09c462ac517afe6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T18:58:37.031 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:37.031 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:37.031 DEBUG:boto:Method: GET 2023-10-17T18:58:37.032 DEBUG:boto:Path: /hbmuog-59/obj51 2023-10-17T18:58:37.032 DEBUG:boto:Data: 2023-10-17T18:58:37.032 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.033 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.033 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.034 DEBUG:boto:Params: {} 2023-10-17T18:58:37.034 DEBUG:boto:Token: None 2023-10-17T18:58:37.035 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj51 2023-10-17T18:58:37.035 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y= 2023-10-17T18:58:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y=', 'Content-Length': '0'} 2023-10-17T18:58:37.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef9f7f45e242b99-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.078 DEBUG:boto:path=/hbmuog-59/obj51 2023-10-17T18:58:37.079 DEBUG:boto:auth_path=/hbmuog-59/obj51 2023-10-17T18:58:37.079 DEBUG:boto:Method: GET 2023-10-17T18:58:37.080 DEBUG:boto:Path: /hbmuog-59/obj51 2023-10-17T18:58:37.080 DEBUG:boto:Data: 2023-10-17T18:58:37.081 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.082 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.082 DEBUG:boto:Params: {} 2023-10-17T18:58:37.082 DEBUG:boto:Token: None 2023-10-17T18:58:37.083 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj51 2023-10-17T18:58:37.083 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y= 2023-10-17T18:58:37.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y=', 'Content-Length': '0'} 2023-10-17T18:58:37.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000355b67403242942c-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.087 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.087 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.087 DEBUG:boto:path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.088 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.088 DEBUG:boto:Method: GET 2023-10-17T18:58:37.088 DEBUG:boto:Path: /hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.089 DEBUG:boto:Data: 2023-10-17T18:58:37.089 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.090 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.090 DEBUG:boto:Params: {} 2023-10-17T18:58:37.091 DEBUG:boto:Token: None 2023-10-17T18:58:37.091 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9316ed4540498ab-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.095 DEBUG:boto:b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.096 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.097 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.097 DEBUG:boto:path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.097 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.098 DEBUG:boto:Method: GET 2023-10-17T18:58:37.098 DEBUG:boto:Path: /hbmuog-59/?prefix=obj51 2023-10-17T18:58:37.099 DEBUG:boto:Data: 2023-10-17T18:58:37.099 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.100 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.101 DEBUG:boto:Params: {} 2023-10-17T18:58:37.101 DEBUG:boto:Token: None 2023-10-17T18:58:37.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000947bd5a24a204c52-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.106 DEBUG:boto:b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:58:37.107 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.107 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.108 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.108 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.109 DEBUG:boto:Method: GET 2023-10-17T18:58:37.109 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.109 DEBUG:boto:Data: 2023-10-17T18:58:37.110 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.111 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.111 DEBUG:boto:Params: {} 2023-10-17T18:58:37.111 DEBUG:boto:Token: None 2023-10-17T18:58:37.112 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T18:58:37.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T18:58:37.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd603a879bcf97e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.115 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.116 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.116 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.116 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.117 DEBUG:boto:Method: GET 2023-10-17T18:58:37.117 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.118 DEBUG:boto:Data: 2023-10-17T18:58:37.118 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.119 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.119 DEBUG:boto:Params: {} 2023-10-17T18:58:37.120 DEBUG:boto:Token: None 2023-10-17T18:58:37.120 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T18:58:37.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T18:58:37.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fda66bab7b427d9-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.124 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.124 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.125 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.125 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.125 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.126 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.126 DEBUG:boto:Data: 2023-10-17T18:58:37.127 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.127 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.128 DEBUG:boto:Params: {} 2023-10-17T18:58:37.128 DEBUG:boto:Token: None 2023-10-17T18:58:37.129 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI= 2023-10-17T18:58:37.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI=', 'Content-Length': '0'} 2023-10-17T18:58:37.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f86fac30f6eb1e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.132 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.133 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.133 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.133 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.134 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.134 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.135 DEBUG:boto:Data: 2023-10-17T18:58:37.135 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.136 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.136 DEBUG:boto:Params: {} 2023-10-17T18:58:37.137 DEBUG:boto:Token: None 2023-10-17T18:58:37.137 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.138 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI= 2023-10-17T18:58:37.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI=', 'Content-Length': '0'} 2023-10-17T18:58:37.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66fdb7419a52790-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:58:37.141 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.141 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.142 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.142 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.143 DEBUG:boto:Method: GET 2023-10-17T18:58:37.143 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.143 DEBUG:boto:Data: 2023-10-17T18:58:37.144 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.144 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.145 DEBUG:boto:Params: {} 2023-10-17T18:58:37.145 DEBUG:boto:Token: None 2023-10-17T18:58:37.146 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T18:58:37.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T18:58:37.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a364016d67cb55f-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.190 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.191 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.191 DEBUG:boto:path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.192 DEBUG:boto:auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.192 DEBUG:boto:Method: GET 2023-10-17T18:58:37.192 DEBUG:boto:Path: /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.193 DEBUG:boto:Data: 2023-10-17T18:58:37.193 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.194 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.194 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.195 DEBUG:boto:Params: {} 2023-10-17T18:58:37.195 DEBUG:boto:Token: None 2023-10-17T18:58:37.196 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55?versionId=null 2023-10-17T18:58:37.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T18:58:37.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T18:58:37.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32abd6e294dd75b-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.199 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.200 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.200 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.201 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.201 DEBUG:boto:Method: GET 2023-10-17T18:58:37.201 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.202 DEBUG:boto:Data: 2023-10-17T18:58:37.202 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.203 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.203 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.204 DEBUG:boto:Params: {} 2023-10-17T18:58:37.204 DEBUG:boto:Token: None 2023-10-17T18:58:37.205 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057c94f4c63c67241-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.209 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.209 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.210 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.210 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.211 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.211 DEBUG:boto:Method: GET 2023-10-17T18:58:37.211 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.212 DEBUG:boto:Data: 2023-10-17T18:58:37.212 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.213 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.213 DEBUG:boto:Params: {} 2023-10-17T18:58:37.214 DEBUG:boto:Token: None 2023-10-17T18:58:37.214 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca42267650ea019-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.218 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.219 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.219 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.220 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.220 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:58:37.221 DEBUG:boto:Data: 2023-10-17T18:58:37.221 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.221 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.222 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.222 DEBUG:boto:Params: {} 2023-10-17T18:58:37.223 DEBUG:boto:Token: None 2023-10-17T18:58:37.223 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55 2023-10-17T18:58:37.223 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo= 2023-10-17T18:58:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo=', 'Content-Length': '0'} 2023-10-17T18:58:37.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53ba684a9775b02-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.226 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.227 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.227 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.228 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:58:37.228 DEBUG:boto:Data: 2023-10-17T18:58:37.229 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.229 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.229 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.230 DEBUG:boto:Params: {} 2023-10-17T18:58:37.230 DEBUG:boto:Token: None 2023-10-17T18:58:37.231 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55 2023-10-17T18:58:37.231 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo= 2023-10-17T18:58:37.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo=', 'Content-Length': '0'} 2023-10-17T18:58:37.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f70f6ade4e3c0ca-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T18:58:37.236 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.236 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.237 DEBUG:boto:Method: GET 2023-10-17T18:58:37.237 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:58:37.238 DEBUG:boto:Data: 2023-10-17T18:58:37.238 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.238 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.239 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.239 DEBUG:boto:Params: {} 2023-10-17T18:58:37.240 DEBUG:boto:Token: None 2023-10-17T18:58:37.240 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55 2023-10-17T18:58:37.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As= 2023-10-17T18:58:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As=', 'Content-Length': '0'} 2023-10-17T18:58:37.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c07f295136e9ba5a-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.244 DEBUG:boto:path=/hbmuog-59/obj55 2023-10-17T18:58:37.244 DEBUG:boto:auth_path=/hbmuog-59/obj55 2023-10-17T18:58:37.245 DEBUG:boto:Method: GET 2023-10-17T18:58:37.245 DEBUG:boto:Path: /hbmuog-59/obj55 2023-10-17T18:58:37.246 DEBUG:boto:Data: 2023-10-17T18:58:37.246 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.247 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.248 DEBUG:boto:Params: {} 2023-10-17T18:58:37.248 DEBUG:boto:Token: None 2023-10-17T18:58:37.248 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj55 2023-10-17T18:58:37.249 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As= 2023-10-17T18:58:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As=', 'Content-Length': '0'} 2023-10-17T18:58:37.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a11e4db9149cc85-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.252 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.253 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.253 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.253 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.254 DEBUG:boto:Method: GET 2023-10-17T18:58:37.254 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.255 DEBUG:boto:Data: 2023-10-17T18:58:37.255 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.256 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.256 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.256 DEBUG:boto:Params: {} 2023-10-17T18:58:37.257 DEBUG:boto:Token: None 2023-10-17T18:58:37.257 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.258 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b24bb304467fd0e0-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.261 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.262 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.263 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.263 DEBUG:boto:path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.264 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.264 DEBUG:boto:Method: GET 2023-10-17T18:58:37.264 DEBUG:boto:Path: /hbmuog-59/?prefix=obj55 2023-10-17T18:58:37.265 DEBUG:boto:Data: 2023-10-17T18:58:37.265 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.266 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.266 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.267 DEBUG:boto:Params: {} 2023-10-17T18:58:37.267 DEBUG:boto:Token: None 2023-10-17T18:58:37.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.268 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000417ca94cb8ac5f61-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.318 DEBUG:boto:b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T18:58:37.379 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.380 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.380 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.381 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.382 DEBUG:boto:Method: GET 2023-10-17T18:58:37.382 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.383 DEBUG:boto:Data: 2023-10-17T18:58:37.384 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.385 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.386 DEBUG:boto:Params: {} 2023-10-17T18:58:37.387 DEBUG:boto:Token: None 2023-10-17T18:58:37.387 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.389 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T18:58:37.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T18:58:37.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0646ed6e1ccd18-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.392 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.393 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.393 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.394 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.394 DEBUG:boto:Method: GET 2023-10-17T18:58:37.394 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.395 DEBUG:boto:Data: 2023-10-17T18:58:37.395 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.396 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.396 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.397 DEBUG:boto:Params: {} 2023-10-17T18:58:37.397 DEBUG:boto:Token: None 2023-10-17T18:58:37.398 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.398 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T18:58:37.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T18:58:37.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd864c6b21f03b3b-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.402 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.402 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.403 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.403 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.403 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.404 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.404 DEBUG:boto:Data: 2023-10-17T18:58:37.405 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.405 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.406 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.406 DEBUG:boto:Params: {} 2023-10-17T18:58:37.407 DEBUG:boto:Token: None 2023-10-17T18:58:37.407 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.408 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs= 2023-10-17T18:58:37.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs=', 'Content-Length': '0'} 2023-10-17T18:58:37.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f22d0c850e85ba3-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.411 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.411 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.412 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.412 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.412 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.413 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.413 DEBUG:boto:Data: 2023-10-17T18:58:37.414 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.415 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.415 DEBUG:boto:Params: {} 2023-10-17T18:58:37.416 DEBUG:boto:Token: None 2023-10-17T18:58:37.416 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs= 2023-10-17T18:58:37.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs=', 'Content-Length': '0'} 2023-10-17T18:58:37.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d677eba75ce2fd-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T18:58:37.420 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.420 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.421 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.421 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.422 DEBUG:boto:Method: GET 2023-10-17T18:58:37.422 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.423 DEBUG:boto:Data: 2023-10-17T18:58:37.423 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.423 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.424 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.424 DEBUG:boto:Params: {} 2023-10-17T18:58:37.425 DEBUG:boto:Token: None 2023-10-17T18:58:37.425 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.426 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T18:58:37.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T18:58:37.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d3794dc634b50e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.470 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.471 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.471 DEBUG:boto:path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.472 DEBUG:boto:auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.472 DEBUG:boto:Method: GET 2023-10-17T18:58:37.473 DEBUG:boto:Path: /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.473 DEBUG:boto:Data: 2023-10-17T18:58:37.474 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.474 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.475 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.475 DEBUG:boto:Params: {} 2023-10-17T18:58:37.476 DEBUG:boto:Token: None 2023-10-17T18:58:37.476 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68?versionId=null 2023-10-17T18:58:37.477 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T18:58:37.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T18:58:37.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d654f2daecd7b0a4-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.480 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.481 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.481 DEBUG:boto:path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.481 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.482 DEBUG:boto:Method: GET 2023-10-17T18:58:37.482 DEBUG:boto:Path: /hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.483 DEBUG:boto:Data: 2023-10-17T18:58:37.483 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.484 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.484 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.485 DEBUG:boto:Params: {} 2023-10-17T18:58:37.485 DEBUG:boto:Token: None 2023-10-17T18:58:37.486 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2312f6f0d813d3c-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.490 DEBUG:boto:b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.491 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.492 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.492 DEBUG:boto:path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.493 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.493 DEBUG:boto:Method: GET 2023-10-17T18:58:37.494 DEBUG:boto:Path: /hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.494 DEBUG:boto:Data: 2023-10-17T18:58:37.494 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.495 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.496 DEBUG:boto:Params: {} 2023-10-17T18:58:37.496 DEBUG:boto:Token: None 2023-10-17T18:58:37.497 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003304d24c3702a73e-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.502 DEBUG:boto:b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.502 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.503 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.503 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.504 DEBUG:boto:Path: /hbmuog-59/obj68 2023-10-17T18:58:37.504 DEBUG:boto:Data: 2023-10-17T18:58:37.504 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.505 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.505 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.506 DEBUG:boto:Params: {} 2023-10-17T18:58:37.506 DEBUG:boto:Token: None 2023-10-17T18:58:37.507 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68 2023-10-17T18:58:37.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4= 2023-10-17T18:58:37.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4=', 'Content-Length': '0'} 2023-10-17T18:58:37.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eba0d64f6a2def3-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.510 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.511 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.511 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.512 DEBUG:boto:Path: /hbmuog-59/obj68 2023-10-17T18:58:37.512 DEBUG:boto:Data: 2023-10-17T18:58:37.513 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.513 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.513 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.514 DEBUG:boto:Params: {} 2023-10-17T18:58:37.514 DEBUG:boto:Token: None 2023-10-17T18:58:37.515 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68 2023-10-17T18:58:37.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4= 2023-10-17T18:58:37.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4=', 'Content-Length': '0'} 2023-10-17T18:58:37.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ac2a52296d7db6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T18:58:37.520 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.520 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.521 DEBUG:boto:Method: GET 2023-10-17T18:58:37.521 DEBUG:boto:Path: /hbmuog-59/obj68 2023-10-17T18:58:37.522 DEBUG:boto:Data: 2023-10-17T18:58:37.522 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.523 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.524 DEBUG:boto:Params: {} 2023-10-17T18:58:37.524 DEBUG:boto:Token: None 2023-10-17T18:58:37.525 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68 2023-10-17T18:58:37.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso= 2023-10-17T18:58:37.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso=', 'Content-Length': '0'} 2023-10-17T18:58:37.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a018f1e20aff42bc-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.529 DEBUG:boto:path=/hbmuog-59/obj68 2023-10-17T18:58:37.529 DEBUG:boto:auth_path=/hbmuog-59/obj68 2023-10-17T18:58:37.530 DEBUG:boto:Method: GET 2023-10-17T18:58:37.530 DEBUG:boto:Path: /hbmuog-59/obj68 2023-10-17T18:58:37.533 DEBUG:boto:Data: 2023-10-17T18:58:37.533 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.534 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.535 DEBUG:boto:Params: {} 2023-10-17T18:58:37.535 DEBUG:boto:Token: None 2023-10-17T18:58:37.536 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj68 2023-10-17T18:58:37.536 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso= 2023-10-17T18:58:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso=', 'Content-Length': '0'} 2023-10-17T18:58:37.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4768fc2d605bc7-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.540 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.540 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.541 DEBUG:boto:path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.541 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.542 DEBUG:boto:Method: GET 2023-10-17T18:58:37.542 DEBUG:boto:Path: /hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.543 DEBUG:boto:Data: 2023-10-17T18:58:37.543 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.544 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.544 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.545 DEBUG:boto:Params: {} 2023-10-17T18:58:37.545 DEBUG:boto:Token: None 2023-10-17T18:58:37.546 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.547 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d8d0e8a19382808-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.551 DEBUG:boto:b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.551 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.552 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.552 DEBUG:boto:path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.553 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.553 DEBUG:boto:Method: GET 2023-10-17T18:58:37.554 DEBUG:boto:Path: /hbmuog-59/?prefix=obj68 2023-10-17T18:58:37.554 DEBUG:boto:Data: 2023-10-17T18:58:37.555 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.555 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.556 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.556 DEBUG:boto:Params: {} 2023-10-17T18:58:37.557 DEBUG:boto:Token: None 2023-10-17T18:58:37.557 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.558 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688f43b0890e9603-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.602 DEBUG:boto:b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T18:58:37.604 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.604 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.604 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.605 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.605 DEBUG:boto:Method: GET 2023-10-17T18:58:37.606 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.606 DEBUG:boto:Data: 2023-10-17T18:58:37.607 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.607 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.608 DEBUG:boto:Params: {} 2023-10-17T18:58:37.608 DEBUG:boto:Token: None 2023-10-17T18:58:37.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T18:58:37.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T18:58:37.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e61f1b7a702d111-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.613 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.613 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.614 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.614 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.614 DEBUG:boto:Method: GET 2023-10-17T18:58:37.615 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.615 DEBUG:boto:Data: 2023-10-17T18:58:37.616 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.616 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.617 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.617 DEBUG:boto:Params: {} 2023-10-17T18:58:37.618 DEBUG:boto:Token: None 2023-10-17T18:58:37.618 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.619 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T18:58:37.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T18:58:37.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0a8024069bb1f12-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.622 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.623 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.623 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.624 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.624 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.624 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.625 DEBUG:boto:Data: 2023-10-17T18:58:37.625 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.626 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.627 DEBUG:boto:Params: {} 2023-10-17T18:58:37.627 DEBUG:boto:Token: None 2023-10-17T18:58:37.628 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw= 2023-10-17T18:58:37.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw=', 'Content-Length': '0'} 2023-10-17T18:58:37.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa8e2d212e3aa79-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.631 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.632 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.632 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.633 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.633 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.634 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.634 DEBUG:boto:Data: 2023-10-17T18:58:37.634 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.635 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.635 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.636 DEBUG:boto:Params: {} 2023-10-17T18:58:37.636 DEBUG:boto:Token: None 2023-10-17T18:58:37.637 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw= 2023-10-17T18:58:37.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw=', 'Content-Length': '0'} 2023-10-17T18:58:37.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130ec1770f442698-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T18:58:37.641 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.641 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.642 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.642 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.642 DEBUG:boto:Method: GET 2023-10-17T18:58:37.643 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.643 DEBUG:boto:Data: 2023-10-17T18:58:37.644 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.644 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.645 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.645 DEBUG:boto:Params: {} 2023-10-17T18:58:37.646 DEBUG:boto:Token: None 2023-10-17T18:58:37.646 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T18:58:37.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T18:58:37.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2c13bd08da2866-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.650 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.650 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.650 DEBUG:boto:path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.651 DEBUG:boto:auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.651 DEBUG:boto:Method: GET 2023-10-17T18:58:37.652 DEBUG:boto:Path: /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.652 DEBUG:boto:Data: 2023-10-17T18:58:37.653 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.653 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.653 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.654 DEBUG:boto:Params: {} 2023-10-17T18:58:37.654 DEBUG:boto:Token: None 2023-10-17T18:58:37.655 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85?versionId=null 2023-10-17T18:58:37.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T18:58:37.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T18:58:37.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf094c623afb6f6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.658 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.659 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.659 DEBUG:boto:path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.659 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.660 DEBUG:boto:Method: GET 2023-10-17T18:58:37.660 DEBUG:boto:Path: /hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.661 DEBUG:boto:Data: 2023-10-17T18:58:37.661 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.662 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.663 DEBUG:boto:Params: {} 2023-10-17T18:58:37.663 DEBUG:boto:Token: None 2023-10-17T18:58:37.664 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c29d7258d32cd10-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.668 DEBUG:boto:b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.668 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.669 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.669 DEBUG:boto:path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.670 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.670 DEBUG:boto:Method: GET 2023-10-17T18:58:37.671 DEBUG:boto:Path: /hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.671 DEBUG:boto:Data: 2023-10-17T18:58:37.671 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.672 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.672 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.673 DEBUG:boto:Params: {} 2023-10-17T18:58:37.673 DEBUG:boto:Token: None 2023-10-17T18:58:37.674 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.674 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003743595cf03327dd-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.678 DEBUG:boto:b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.679 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.679 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.680 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.680 DEBUG:boto:Path: /hbmuog-59/obj85 2023-10-17T18:58:37.680 DEBUG:boto:Data: 2023-10-17T18:58:37.681 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.681 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.682 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.682 DEBUG:boto:Params: {} 2023-10-17T18:58:37.683 DEBUG:boto:Token: None 2023-10-17T18:58:37.683 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85 2023-10-17T18:58:37.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI= 2023-10-17T18:58:37.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI=', 'Content-Length': '0'} 2023-10-17T18:58:37.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6a902e2896efc5-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.687 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.687 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.688 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.688 DEBUG:boto:Path: /hbmuog-59/obj85 2023-10-17T18:58:37.688 DEBUG:boto:Data: 2023-10-17T18:58:37.689 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.689 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.689 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.690 DEBUG:boto:Params: {} 2023-10-17T18:58:37.690 DEBUG:boto:Token: None 2023-10-17T18:58:37.691 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85 2023-10-17T18:58:37.691 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI= 2023-10-17T18:58:37.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI=', 'Content-Length': '0'} 2023-10-17T18:58:37.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3183cc32c322c0-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T18:58:37.695 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.695 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.696 DEBUG:boto:Method: GET 2023-10-17T18:58:37.696 DEBUG:boto:Path: /hbmuog-59/obj85 2023-10-17T18:58:37.697 DEBUG:boto:Data: 2023-10-17T18:58:37.697 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.698 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.698 DEBUG:boto:Params: {} 2023-10-17T18:58:37.699 DEBUG:boto:Token: None 2023-10-17T18:58:37.699 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85 2023-10-17T18:58:37.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM= 2023-10-17T18:58:37.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM=', 'Content-Length': '0'} 2023-10-17T18:58:37.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d873bda0cb81a20d-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.746 DEBUG:boto:path=/hbmuog-59/obj85 2023-10-17T18:58:37.747 DEBUG:boto:auth_path=/hbmuog-59/obj85 2023-10-17T18:58:37.747 DEBUG:boto:Method: GET 2023-10-17T18:58:37.748 DEBUG:boto:Path: /hbmuog-59/obj85 2023-10-17T18:58:37.748 DEBUG:boto:Data: 2023-10-17T18:58:37.749 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.750 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.750 DEBUG:boto:Params: {} 2023-10-17T18:58:37.751 DEBUG:boto:Token: None 2023-10-17T18:58:37.751 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj85 2023-10-17T18:58:37.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM= 2023-10-17T18:58:37.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM=', 'Content-Length': '0'} 2023-10-17T18:58:37.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090feeb1b5c0168b8-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.756 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.756 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.757 DEBUG:boto:path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.757 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.757 DEBUG:boto:Method: GET 2023-10-17T18:58:37.758 DEBUG:boto:Path: /hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.758 DEBUG:boto:Data: 2023-10-17T18:58:37.759 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T18:58:37.760 DEBUG:boto:Port: 8001 2023-10-17T18:58:37.760 DEBUG:boto:Params: {} 2023-10-17T18:58:37.761 DEBUG:boto:Token: None 2023-10-17T18:58:37.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194735063eb802f5-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.766 DEBUG:boto:b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.766 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.767 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.767 DEBUG:boto:path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.768 DEBUG:boto:auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.768 DEBUG:boto:Method: GET 2023-10-17T18:58:37.768 DEBUG:boto:Path: /hbmuog-59/?prefix=obj85 2023-10-17T18:58:37.769 DEBUG:boto:Data: 2023-10-17T18:58:37.769 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.770 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.770 DEBUG:boto:Params: {} 2023-10-17T18:58:37.771 DEBUG:boto:Token: None 2023-10-17T18:58:37.771 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T18:58:37.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T18:58:37.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259ea40f0dcfc338-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.776 DEBUG:boto:b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T18:58:37.776 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-59 zones={a2, a1} 2023-10-17T18:58:37.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:58:37.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T18:58:37.778 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T18:58:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:58:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:58:37.883 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T18:58:37.884 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-59 2023-10-17T18:58:37.889 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 10 2023-10-17T18:58:37.932 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.933 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.933 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.933 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:37.934 DEBUG:boto:Data: 2023-10-17T18:58:37.934 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.935 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.935 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.935 DEBUG:boto:Params: {} 2023-10-17T18:58:37.936 DEBUG:boto:Token: None 2023-10-17T18:58:37.936 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.937 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s= 2023-10-17T18:58:37.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s=', 'Content-Length': '0'} 2023-10-17T18:58:37.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005794cf73f72db3f6-00652ed95d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.941 DEBUG:boto:path=/hbmuog-59/obj19 2023-10-17T18:58:37.941 DEBUG:boto:auth_path=/hbmuog-59/obj19 2023-10-17T18:58:37.942 DEBUG:boto:Method: PUT 2023-10-17T18:58:37.942 DEBUG:boto:Path: /hbmuog-59/obj19 2023-10-17T18:58:37.943 DEBUG:boto:Data: 2023-10-17T18:58:37.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:58:37.944 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.944 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.944 DEBUG:boto:Params: {} 2023-10-17T18:58:37.945 DEBUG:boto:Token: None 2023-10-17T18:58:37.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj19 2023-10-17T18:58:37.946 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GqFtDwo2pWVMVnhr27t1E3r2mww= 2023-10-17T18:58:37.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqFtDwo2pWVMVnhr27t1E3r2mww='} 2023-10-17T18:58:37.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fff93f3e896d5de9-00652ed95d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.952 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:37.953 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:37.953 DEBUG:boto:Method: HEAD 2023-10-17T18:58:37.953 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:37.954 DEBUG:boto:Data: 2023-10-17T18:58:37.954 DEBUG:boto:Headers: {} 2023-10-17T18:58:37.955 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.955 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.956 DEBUG:boto:Params: {} 2023-10-17T18:58:37.958 DEBUG:boto:Token: None 2023-10-17T18:58:37.958 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/ 2023-10-17T18:58:37.959 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s= 2023-10-17T18:58:37.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s=', 'Content-Length': '0'} 2023-10-17T18:58:37.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064d66cd75faf2ad2-00652ed95d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:37.963 DEBUG:boto:path=/hbmuog-59/obj38 2023-10-17T18:58:37.963 DEBUG:boto:auth_path=/hbmuog-59/obj38 2023-10-17T18:58:37.964 DEBUG:boto:Method: PUT 2023-10-17T18:58:37.964 DEBUG:boto:Path: /hbmuog-59/obj38 2023-10-17T18:58:37.964 DEBUG:boto:Data: 2023-10-17T18:58:37.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:58:37.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:37.966 DEBUG:boto:Port: 8000 2023-10-17T18:58:37.967 DEBUG:boto:Params: {} 2023-10-17T18:58:37.967 DEBUG:boto:Token: None 2023-10-17T18:58:37.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:58:37 GMT /hbmuog-59/obj38 2023-10-17T18:58:37.968 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yq3mt6qSu4Ufjhw2FeACKgzV5Lk= 2023-10-17T18:58:37.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yq3mt6qSu4Ufjhw2FeACKgzV5Lk='} 2023-10-17T18:58:38.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ff964cd3cad62e2-00652ed95d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.015 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:38.016 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:38.016 DEBUG:boto:Method: HEAD 2023-10-17T18:58:38.016 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:38.017 DEBUG:boto:Data: 2023-10-17T18:58:38.017 DEBUG:boto:Headers: {} 2023-10-17T18:58:38.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.018 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.018 DEBUG:boto:Params: {} 2023-10-17T18:58:38.019 DEBUG:boto:Token: None 2023-10-17T18:58:38.019 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/ 2023-10-17T18:58:38.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T18:58:38.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T18:58:38.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bef770e4524e609f-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.023 DEBUG:boto:path=/hbmuog-59/obj57 2023-10-17T18:58:38.024 DEBUG:boto:auth_path=/hbmuog-59/obj57 2023-10-17T18:58:38.024 DEBUG:boto:Method: PUT 2023-10-17T18:58:38.024 DEBUG:boto:Path: /hbmuog-59/obj57 2023-10-17T18:58:38.025 DEBUG:boto:Data: 2023-10-17T18:58:38.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:58:38.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.026 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.026 DEBUG:boto:Params: {} 2023-10-17T18:58:38.027 DEBUG:boto:Token: None 2023-10-17T18:58:38.027 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/obj57 2023-10-17T18:58:38.028 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DprTkP43vvfJUHt+hsp7fRRUKFk= 2023-10-17T18:58:38.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DprTkP43vvfJUHt+hsp7fRRUKFk='} 2023-10-17T18:58:38.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5aa5d4a10fe8c82-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.032 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:38.033 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:38.033 DEBUG:boto:Method: HEAD 2023-10-17T18:58:38.034 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:38.034 DEBUG:boto:Data: 2023-10-17T18:58:38.034 DEBUG:boto:Headers: {} 2023-10-17T18:58:38.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.035 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.036 DEBUG:boto:Params: {} 2023-10-17T18:58:38.036 DEBUG:boto:Token: None 2023-10-17T18:58:38.036 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/ 2023-10-17T18:58:38.037 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T18:58:38.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T18:58:38.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f2095c90c138574-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.041 DEBUG:boto:path=/hbmuog-59/obj76 2023-10-17T18:58:38.041 DEBUG:boto:auth_path=/hbmuog-59/obj76 2023-10-17T18:58:38.041 DEBUG:boto:Method: PUT 2023-10-17T18:58:38.042 DEBUG:boto:Path: /hbmuog-59/obj76 2023-10-17T18:58:38.042 DEBUG:boto:Data: 2023-10-17T18:58:38.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:58:38.043 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.043 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.044 DEBUG:boto:Params: {} 2023-10-17T18:58:38.044 DEBUG:boto:Token: None 2023-10-17T18:58:38.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/obj76 2023-10-17T18:58:38.045 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PUdUDpv8obVBxSlfOK3pRd7/4ac= 2023-10-17T18:58:38.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PUdUDpv8obVBxSlfOK3pRd7/4ac='} 2023-10-17T18:58:38.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbde9a80f76c1e4e-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.091 DEBUG:boto:path=/hbmuog-59/ 2023-10-17T18:58:38.091 DEBUG:boto:auth_path=/hbmuog-59/ 2023-10-17T18:58:38.092 DEBUG:boto:Method: HEAD 2023-10-17T18:58:38.092 DEBUG:boto:Path: /hbmuog-59/ 2023-10-17T18:58:38.093 DEBUG:boto:Data: 2023-10-17T18:58:38.093 DEBUG:boto:Headers: {} 2023-10-17T18:58:38.093 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.094 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.094 DEBUG:boto:Params: {} 2023-10-17T18:58:38.095 DEBUG:boto:Token: None 2023-10-17T18:58:38.095 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/ 2023-10-17T18:58:38.096 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T18:58:38.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T18:58:38.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080867f0a79e52643-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.099 DEBUG:boto:path=/hbmuog-59/obj95 2023-10-17T18:58:38.100 DEBUG:boto:auth_path=/hbmuog-59/obj95 2023-10-17T18:58:38.100 DEBUG:boto:Method: PUT 2023-10-17T18:58:38.100 DEBUG:boto:Path: /hbmuog-59/obj95 2023-10-17T18:58:38.101 DEBUG:boto:Data: 2023-10-17T18:58:38.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T18:58:38.102 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T18:58:38.102 DEBUG:boto:Port: 8000 2023-10-17T18:58:38.103 DEBUG:boto:Params: {} 2023-10-17T18:58:38.103 DEBUG:boto:Token: None 2023-10-17T18:58:38.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 18:58:38 GMT /hbmuog-59/obj95 2023-10-17T18:58:38.104 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yLmfQchb06wLvB6XfViqXbd89EE= 2023-10-17T18:58:38.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yLmfQchb06wLvB6XfViqXbd89EE='} 2023-10-17T18:58:38.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a62d0fc3981a50a7-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T18:58:38.109 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T18:58:38.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:58:38.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T18:58:38.110 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T18:58:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T18:58:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T18:58:38.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:38.204+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:58:43.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:43.206+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:58:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:48.207+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:58:53.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:53.206+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T18:58:58.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:58:58.208+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:58:58.210 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-17T18:58:58.210 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:03.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:03.209+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:03.211 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:03.211 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:08.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:08.211+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:08.213 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:08.213 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:13.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:13.212+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:13.214 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:13.215 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:18.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:18.215+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:18.216 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:18.216 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:23.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:23.217+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:23.218 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:23.218 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:28.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:28.217+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:28.220 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:28.220 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:33.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:33.219+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:33.221 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:33.222 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:38.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:38.221+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:38.223 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:38.223 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:43.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:43.223+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:43.224 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:43.225 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:48.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:48.224+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:48.226 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:48.226 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:53.227+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:53.228 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:53.228 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T18:59:58.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T18:59:58.229+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T18:59:58.230 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T18:59:58.230 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:03.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:03.230+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:03.231 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:03.232 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:08.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:08.231+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:08.233 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:08.233 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:13.235 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:13.233+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:13.235 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:13.236 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:18.237 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:18.235+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:18.237 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:18.238 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:23.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:23.237+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:23.239 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:23.239 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:28.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:28.240+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:28.241 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:00:28.242 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:33.242 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:33.242+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:33.243 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:33.243 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:38.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:38.243+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:38.244 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:38.244 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:43.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:43.244+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:43.245 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:43.246 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:48.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:48.245+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:48.247 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:48.247 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:53.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:53.247+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:53.249 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:53.249 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:00:58.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:00:58.249+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:00:58.251 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:00:58.251 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:03.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:03.250+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:03.252 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:03.253 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:08.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:08.253+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:08.254 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:08.254 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:13.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:13.256+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:13.257 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:13.257 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:18.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:18.256+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:18.258 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:18.258 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:23.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:23.258+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:23.260 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:23.261 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:28.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:28.260+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:28.262 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:28.262 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:33.261+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:33.263 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:33.263 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:38.263+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:38.265 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:38.265 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:43.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:43.266+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:43.266 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:43.267 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:48.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:48.267+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:48.268 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:48.268 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:53.269 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:53.268+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:53.270 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:53.270 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:01:58.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:01:58.270+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:01:58.272 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:01:58.272 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:03.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:03.271+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:03.273 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:03.274 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:08.273+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:08.275 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:08.275 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:13.275+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:13.277 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:13.277 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:18.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:18.277+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:18.278 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:18.279 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:23.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:23.279+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:23.280 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:23.280 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:28.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:28.281+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:28.282 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:28.282 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:33.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:33.281+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:33.283 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:33.284 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:38.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:38.283+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:38.285 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:38.285 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:43.285+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:43.287 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:43.287 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:48.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:48.287+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:48.288 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:48.289 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:53.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:53.289+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:53.290 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:53.290 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:02:58.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:02:58.291+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:02:58.292 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:02:58.292 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:03.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:03.293+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:03.294 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:03.294 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:08.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:08.293+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:08.295 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:08.295 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:13.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:13.295+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:13.297 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:13.297 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:18.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:18.297+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:18.299 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:18.299 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:23.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:23.298+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:23.300 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:23.301 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:28.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:28.300+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:03:28.302 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:03:28.302 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:03:33.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:33.303+0000 7fc68f5ff780 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-10-17T19:03:33.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:33.303+0000 7fc68f5ff780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:03:33.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:33.303+0000 7fc68f5ff780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:03:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-17T19:03:33.327 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2023-10-17T19:03:33.328 INFO:rgw_multi.tests:create role zone=a1 name=roles-hbmuog-1 2023-10-17T19:03:33.329 DEBUG:boto:Method: POST 2023-10-17T19:03:33.329 DEBUG:boto:Path: / 2023-10-17T19:03:33.330 DEBUG:boto:Data: 2023-10-17T19:03:33.330 DEBUG:boto:Headers: {} 2023-10-17T19:03:33.330 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:33.331 DEBUG:boto:Port: 8000 2023-10-17T19:03:33.331 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-17T19:03:33.332 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:03:33.332 DEBUG:boto:Token: None 2023-10-17T19:03:33.333 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8000 x-amz-date:20231017T190333Z host;x-amz-date f584d35faf926d1bb200e5ad6be2279ad3344b7a730afec196774d55527d351a 2023-10-17T19:03:33.333 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190333Z 20231017/front/smithi049/aws4_request 837849515a32b82aaba6d0c1f083f55729725db70b90d0137d0413fd614cd6fb 2023-10-17T19:03:33.334 DEBUG:boto:Signature: 8657823896a77315223e9e056a246f154dc5d81c729aaae6f22e0ec64c246496 2023-10-17T19:03:33.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190333Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=8657823896a77315223e9e056a246f154dc5d81c729aaae6f22e0ec64c246496', 'Host': 'smithi049.front.sepia.ceph.com:8000'} 2023-10-17T19:03:33.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ac2ca9c89fd55e5-00652eda85-162c-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Tue, 17 Oct 2023 19:03:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:33.349 DEBUG:boto:b'eb4a24d6-0587-4cee-9811-1589e2a029c1roles-hbmuog-1/arn:aws:iam:::role/roles-hbmuog-12023-10-17T19:03:33.340Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000002ac2ca9c89fd55e5-00652eda85-162c-a1' 2023-10-17T19:03:33.350 INFO:rgw_multi.tests:create role zone=a2 name=roles-hbmuog-2 2023-10-17T19:03:33.350 DEBUG:boto:Method: POST 2023-10-17T19:03:33.351 DEBUG:boto:Path: / 2023-10-17T19:03:33.351 DEBUG:boto:Data: 2023-10-17T19:03:33.352 DEBUG:boto:Headers: {} 2023-10-17T19:03:33.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:33.352 DEBUG:boto:Port: 8001 2023-10-17T19:03:33.353 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-17T19:03:33.353 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:03:33.354 DEBUG:boto:Token: None 2023-10-17T19:03:33.354 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8001 x-amz-date:20231017T190333Z host;x-amz-date 7e5afeb8aec284de7ad1a4578d48bdf871bc41a473100431dbbac06fcee959c4 2023-10-17T19:03:33.355 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190333Z 20231017/front/smithi049/aws4_request 3441e28044faf21bc4423add593cdf69200967733ec50a8ebea31aa2a36d76fc 2023-10-17T19:03:33.355 DEBUG:boto:Signature: f6c4547aeae9fc0ec16584b68624b2abff968da043fe142eab3de358d081d47f 2023-10-17T19:03:33.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190333Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=f6c4547aeae9fc0ec16584b68624b2abff968da043fe142eab3de358d081d47f', 'Host': 'smithi049.front.sepia.ceph.com:8001'} 2023-10-17T19:03:33.374 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073fc2df7126c6d9c-00652eda85-1b46-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Tue, 17 Oct 2023 19:03:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:33.375 DEBUG:boto:b'dbb4bfb7-b182-4c9f-ba4e-e662cc63ac7aroles-hbmuog-2/arn:aws:iam:::role/roles-hbmuog-22023-10-17T19:03:33.369Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000073fc2df7126c6d9c-00652eda85-1b46-a2' 2023-10-17T19:03:33.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:33.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:33.377 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:03:33.432 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:33.433 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:33.466 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:03:33.479 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:03:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569117.894453_750.1", 2023-10-17T19:03:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:58:37.894453Z" 2023-10-17T19:03:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:03:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:03:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:03:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:03:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:03:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:03:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.483 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:03:33.484 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697569117.894453_750.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:33.485 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:03:33.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:33.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:33.486 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:03:33.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:33.538 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:33.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:33.568+0000 7f9b6b8a9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:33.569 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:33.569+0000 7f9b6b8a9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:33.583 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:03:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:03:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:03:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:03:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:03:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:03:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:03:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:03:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569117.894453_750.1", 2023-10-17T19:03:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:03:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:58:37.894453Z", 2023-10-17T19:03:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:03:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.360593_762.1", 2023-10-17T19:03:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.360593Z", 2023-10-17T19:03:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:03:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:03:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:03:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:03:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:03:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:03:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:03:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:03:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:03:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:03:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:03:33Z" 2023-10-17T19:03:33.596 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:03:33.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569117.894453_750.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:58:37.894453Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.360593_762.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.360593Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:03:33Z" } 2023-10-17T19:03:33.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569117.894453_750.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:58:37.894453Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.360593_762.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.360593Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:03:33.598 DEBUG:rgw_multi.tests:log_status={0: '1_1697569117.894453_750.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:33.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569117.894453_750.1', 1: '1_1697569413.360593_762.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:33.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697569413.363964_763.1 target=1_1697569413.360593_762.1 2023-10-17T19:03:38.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.601 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:03:38.660 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:38.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:38.695+0000 7f7bf743d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:38.697+0000 7f7bf743d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:38.711 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:03:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:03:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:03:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:03:38.712 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:03:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:03:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.713 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:03:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:03:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569117.894453_750.1", 2023-10-17T19:03:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:03:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:38.715 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:58:37.894453Z", 2023-10-17T19:03:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:03:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:03:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:03:38.718 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:03:38.719 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:03:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:38.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:38.721 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:03:38.721 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.722 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:03:38.722 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:38.722 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:38.723 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:03:38.723 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:38.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:38.723 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:38.724 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:03:38.724 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.724 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:03:38.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.725 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:03:38.725 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:03:38.725 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:03:38.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.726 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:03:38Z" 2023-10-17T19:03:38.726 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:03:38.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569117.894453_750.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T18:58:37.894453Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:03:38Z" } 2023-10-17T19:03:38.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569117.894453_750.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:58:37.894453Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:03:38.728 DEBUG:rgw_multi.tests:log_status={0: '1_1697569117.894453_750.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:38.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569117.894453_750.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:38.729 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:03:38.729 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hbmuog-1 zones={a2, a1} 2023-10-17T19:03:38.730 DEBUG:boto:Method: POST 2023-10-17T19:03:38.730 DEBUG:boto:Path: / 2023-10-17T19:03:38.731 DEBUG:boto:Data: 2023-10-17T19:03:38.731 DEBUG:boto:Headers: {} 2023-10-17T19:03:38.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:38.732 DEBUG:boto:Port: 8001 2023-10-17T19:03:38.733 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-17T19:03:38.733 DEBUG:boto:Token: None 2023-10-17T19:03:38.734 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8001 x-amz-date:20231017T190338Z host;x-amz-date e35c6ff2a2174506d233e799988dc1c431a5e9698984c4ef0591d688b0f66552 2023-10-17T19:03:38.734 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190338Z 20231017/front/smithi049/aws4_request 1be60c158a47c81e8be4cd30ad7b181927b7fbe1f6bae56dfa7ea5949645c071 2023-10-17T19:03:38.735 DEBUG:boto:Signature: 20926f531ea5368995ce689eaf27472fd24ca0744464f3d80e85f548d7d771f5 2023-10-17T19:03:38.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190338Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=20926f531ea5368995ce689eaf27472fd24ca0744464f3d80e85f548d7d771f5', 'Host': 'smithi049.front.sepia.ceph.com:8001'} 2023-10-17T19:03:38.737 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000320d04a821260732-00652eda8a-1b46-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Tue, 17 Oct 2023 19:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:38.738 DEBUG:boto:b'tx00000320d04a821260732-00652eda8a-1b46-a2eb4a24d6-0587-4cee-9811-1589e2a029c1roles-hbmuog-1/arn:aws:iam:::role/roles-hbmuog-12023-10-17T19:03:33.562Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-17T19:03:38.739 DEBUG:boto:Method: POST 2023-10-17T19:03:38.739 DEBUG:boto:Path: / 2023-10-17T19:03:38.740 DEBUG:boto:Data: 2023-10-17T19:03:38.740 DEBUG:boto:Headers: {} 2023-10-17T19:03:38.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:38.741 DEBUG:boto:Port: 8000 2023-10-17T19:03:38.741 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-17T19:03:38.742 DEBUG:boto:Token: None 2023-10-17T19:03:38.742 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8000 x-amz-date:20231017T190338Z host;x-amz-date e35c6ff2a2174506d233e799988dc1c431a5e9698984c4ef0591d688b0f66552 2023-10-17T19:03:38.743 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190338Z 20231017/front/smithi049/aws4_request 7dd205ad6f2a9789d8a0fa5225d3d721fecbcefd59530ffb47ac44761e1d7bc1 2023-10-17T19:03:38.743 DEBUG:boto:Signature: b3b9247f9462616d870e525f70bcee3395746294155f6ae88b26bc137dc13b7b 2023-10-17T19:03:38.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190338Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=b3b9247f9462616d870e525f70bcee3395746294155f6ae88b26bc137dc13b7b', 'Host': 'smithi049.front.sepia.ceph.com:8000'} 2023-10-17T19:03:38.745 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed16ed1a43f89cd7-00652eda8a-162c-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Tue, 17 Oct 2023 19:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:38.746 DEBUG:boto:b'tx00000ed16ed1a43f89cd7-00652eda8a-162c-a1eb4a24d6-0587-4cee-9811-1589e2a029c1roles-hbmuog-1/arn:aws:iam:::role/roles-hbmuog-12023-10-17T19:03:33.340Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-17T19:03:38.747 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hbmuog-1 2023-10-17T19:03:38.747 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hbmuog-1 zones={a2, a1} 2023-10-17T19:03:38.747 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hbmuog-2 zones={a1, a2} 2023-10-17T19:03:38.748 DEBUG:boto:Method: POST 2023-10-17T19:03:38.748 DEBUG:boto:Path: / 2023-10-17T19:03:38.749 DEBUG:boto:Data: 2023-10-17T19:03:38.749 DEBUG:boto:Headers: {} 2023-10-17T19:03:38.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:38.750 DEBUG:boto:Port: 8000 2023-10-17T19:03:38.750 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-17T19:03:38.751 DEBUG:boto:Token: None 2023-10-17T19:03:38.751 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8000 x-amz-date:20231017T190338Z host;x-amz-date b1959c8dd732e5d023722c8c73e0da3837f01749b0138b4bc843c0303599219a 2023-10-17T19:03:38.752 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190338Z 20231017/front/smithi049/aws4_request 8d13c1a49b50aec2d5185d69d528e710e89de284ee8364834c1a4a85089eca3b 2023-10-17T19:03:38.752 DEBUG:boto:Signature: 4716f33ff2a22680ebd275bec79d178aa1a9011dc12660fc2c0949d9b856244c 2023-10-17T19:03:38.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190338Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=4716f33ff2a22680ebd275bec79d178aa1a9011dc12660fc2c0949d9b856244c', 'Host': 'smithi049.front.sepia.ceph.com:8000'} 2023-10-17T19:03:38.755 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3dbd19f79d1fc48-00652eda8a-162c-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Tue, 17 Oct 2023 19:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:38.755 DEBUG:boto:b'tx00000b3dbd19f79d1fc48-00652eda8a-162c-a1dbb4bfb7-b182-4c9f-ba4e-e662cc63ac7aroles-hbmuog-2/arn:aws:iam:::role/roles-hbmuog-22023-10-17T19:03:33.360Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-17T19:03:38.758 DEBUG:boto:Method: POST 2023-10-17T19:03:38.758 DEBUG:boto:Path: / 2023-10-17T19:03:38.759 DEBUG:boto:Data: 2023-10-17T19:03:38.759 DEBUG:boto:Headers: {} 2023-10-17T19:03:38.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com 2023-10-17T19:03:38.760 DEBUG:boto:Port: 8001 2023-10-17T19:03:38.760 DEBUG:boto:Params: {'RoleName': 'roles-hbmuog-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-17T19:03:38.761 DEBUG:boto:Token: None 2023-10-17T19:03:38.761 DEBUG:boto:CanonicalRequest: POST / host:smithi049.front.sepia.ceph.com:8001 x-amz-date:20231017T190338Z host;x-amz-date b1959c8dd732e5d023722c8c73e0da3837f01749b0138b4bc843c0303599219a 2023-10-17T19:03:38.762 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231017T190338Z 20231017/front/smithi049/aws4_request 90c78a591866950e0e94546f0299476b736ded7b679f9b38f4e5bf992998e42f 2023-10-17T19:03:38.762 DEBUG:boto:Signature: ce729fae68a684d28d88ed580ed804037cc3c736daf564e925b944bd2a2b8613 2023-10-17T19:03:38.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231017T190338Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=EERKR7CLBH40TV9B4PL4/20231017/front/smithi049/aws4_request,SignedHeaders=host;x-amz-date,Signature=ce729fae68a684d28d88ed580ed804037cc3c736daf564e925b944bd2a2b8613', 'Host': 'smithi049.front.sepia.ceph.com:8001'} 2023-10-17T19:03:38.765 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000163cd25e8d3b8346-00652eda8a-1b46-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Tue, 17 Oct 2023 19:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:38.765 DEBUG:boto:b'tx00000163cd25e8d3b8346-00652eda8a-1b46-a2dbb4bfb7-b182-4c9f-ba4e-e662cc63ac7aroles-hbmuog-2/arn:aws:iam:::role/roles-hbmuog-22023-10-17T19:03:33.360Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-17T19:03:38.766 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hbmuog-2 2023-10-17T19:03:38.767 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hbmuog-2 zones={a1, a2} 2023-10-17T19:03:38.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-10-17T19:03:38.768 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:38.768 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:38.769 DEBUG:boto:Method: PUT 2023-10-17T19:03:38.769 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:38.770 DEBUG:boto:Data: 2023-10-17T19:03:38.770 DEBUG:boto:Headers: {} 2023-10-17T19:03:38.770 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:38.773 DEBUG:boto:Port: 8000 2023-10-17T19:03:38.774 DEBUG:boto:Params: {} 2023-10-17T19:03:38.775 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:03:38.775 DEBUG:boto:Token: None 2023-10-17T19:03:38.776 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:03:38 GMT /hbmuog-60/ 2023-10-17T19:03:38.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s6sMenbGsuyeman6se6UVrDek1g= 2023-10-17T19:03:38.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s6sMenbGsuyeman6se6UVrDek1g=', 'Content-Length': '0'} 2023-10-17T19:03:38.794 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000031e41383a1eaa4fb-00652eda8a-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:38.794 DEBUG:rgw_multi.tests:created bucket=hbmuog-60 2023-10-17T19:03:38.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.796 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:03:38.846 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:38.847 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:38.877 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:03:38.890 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:03:38.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.891 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569418.792263_767.1", 2023-10-17T19:03:38.891 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:38.792263Z" 2023-10-17T19:03:38.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:03:38.892 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:03:38.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:03:38.893 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:03:38.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:03:38.893 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:03:38.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:38.894 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:03:38.895 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697569418.792263_767.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:38.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:03:38.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:03:38.897 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:03:38.949 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:38.949 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:38.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:38.982+0000 7fa078e76780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:38.983 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:38.983+0000 7fa078e76780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:03:38.996 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:03:38.996 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:03:38.996 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:03:38.997 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:03:38.997 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:03:38.997 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:03:38.997 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:38.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:38.998 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:03:38.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:38.998 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:03:38.998 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:38.998 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:38.999 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569418.792263_767.1", 2023-10-17T19:03:38.999 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:38.999 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:03:38.999 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:39.000 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:38.792263Z", 2023-10-17T19:03:39.000 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:39.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:39.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:39.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:39.001 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:03:39.001 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:39.001 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:39.002 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:03:39.002 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:39.002 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:39.002 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:39.002 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:03:39.003 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:39.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:39.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:39.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:39.003 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:03:39.004 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:39.004 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:39.004 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:03:39.004 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:39.005 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:39.005 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:39.005 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:03:39.005 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:39.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:03:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:03:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:03:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:03:39.007 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:03:39.007 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:03:39.007 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:03:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:03:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:03:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:03:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:03:39.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:03:39.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:39.009 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:03:39.009 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:03:39.010 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:03:39.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:03:39.010 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:03:38Z" 2023-10-17T19:03:39.010 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:03:39.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569418.792263_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:03:38.792263Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:03:38Z" } 2023-10-17T19:03:39.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569418.792263_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:03:38.792263Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:03:39.012 DEBUG:rgw_multi.tests:log_status={0: '1_1697569418.792263_767.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:39.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569418.792263_767.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:03:39.013 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:03:39.014 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T19:03:39.014 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:03:39.015 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:03:39.014+0000 7fbd6134a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 160334) UID: 0 2023-10-17T19:03:39.015 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:03:39.014+0000 7fbd655b37c0 -1 shutting down 2023-10-17T19:03:44.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:03:45.116 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T19:03:45.117 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:45.117 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:45.118 DEBUG:boto:Method: HEAD 2023-10-17T19:03:45.118 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:45.119 DEBUG:boto:Data: 2023-10-17T19:03:45.119 DEBUG:boto:Headers: {} 2023-10-17T19:03:45.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.120 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.120 DEBUG:boto:Params: {} 2023-10-17T19:03:45.121 DEBUG:boto:Token: None 2023-10-17T19:03:45.121 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/ 2023-10-17T19:03:45.122 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ= 2023-10-17T19:03:45.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ=', 'Content-Length': '0'} 2023-10-17T19:03:45.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059ad47e117e04c9e-00652eda91-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.126 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:03:45.127 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:03:45.127 DEBUG:boto:Method: PUT 2023-10-17T19:03:45.130 DEBUG:boto:Path: /hbmuog-60/obj11 2023-10-17T19:03:45.130 DEBUG:boto:Data: 2023-10-17T19:03:45.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:03:45.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.131 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.132 DEBUG:boto:Params: {} 2023-10-17T19:03:45.132 DEBUG:boto:Token: None 2023-10-17T19:03:45.133 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/obj11 2023-10-17T19:03:45.133 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pUXd/PgWwiF4WmXqLPd+7Y56SXY= 2023-10-17T19:03:45.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pUXd/PgWwiF4WmXqLPd+7Y56SXY='} 2023-10-17T19:03:45.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca1dd84bb2528821-00652eda91-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.142 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:45.142 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:45.142 DEBUG:boto:Method: HEAD 2023-10-17T19:03:45.143 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:45.143 DEBUG:boto:Data: 2023-10-17T19:03:45.144 DEBUG:boto:Headers: {} 2023-10-17T19:03:45.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.144 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.145 DEBUG:boto:Params: {} 2023-10-17T19:03:45.145 DEBUG:boto:Token: None 2023-10-17T19:03:45.146 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/ 2023-10-17T19:03:45.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ= 2023-10-17T19:03:45.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ=', 'Content-Length': '0'} 2023-10-17T19:03:45.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014978b65002350f7-00652eda91-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.150 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:03:45.150 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:03:45.151 DEBUG:boto:Method: PUT 2023-10-17T19:03:45.151 DEBUG:boto:Path: /hbmuog-60/obj22 2023-10-17T19:03:45.151 DEBUG:boto:Data: 2023-10-17T19:03:45.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:03:45.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.152 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.153 DEBUG:boto:Params: {} 2023-10-17T19:03:45.153 DEBUG:boto:Token: None 2023-10-17T19:03:45.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/obj22 2023-10-17T19:03:45.154 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/6Kgi/doBtxfzxyIhJhFBEVUd1w= 2023-10-17T19:03:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/6Kgi/doBtxfzxyIhJhFBEVUd1w='} 2023-10-17T19:03:45.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002f53b92585f32e2-00652eda91-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.159 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:45.160 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:45.160 DEBUG:boto:Method: HEAD 2023-10-17T19:03:45.161 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:45.161 DEBUG:boto:Data: 2023-10-17T19:03:45.161 DEBUG:boto:Headers: {} 2023-10-17T19:03:45.162 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.162 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.162 DEBUG:boto:Params: {} 2023-10-17T19:03:45.163 DEBUG:boto:Token: None 2023-10-17T19:03:45.163 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/ 2023-10-17T19:03:45.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ= 2023-10-17T19:03:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ=', 'Content-Length': '0'} 2023-10-17T19:03:45.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024bf304f74fd4e78-00652eda91-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.167 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:03:45.168 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:03:45.168 DEBUG:boto:Method: PUT 2023-10-17T19:03:45.169 DEBUG:boto:Path: /hbmuog-60/obj33 2023-10-17T19:03:45.169 DEBUG:boto:Data: 2023-10-17T19:03:45.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:03:45.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.170 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.171 DEBUG:boto:Params: {} 2023-10-17T19:03:45.171 DEBUG:boto:Token: None 2023-10-17T19:03:45.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/obj33 2023-10-17T19:03:45.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lwn9XoTBJVr87ZRGjxcLmpG5Yes= 2023-10-17T19:03:45.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lwn9XoTBJVr87ZRGjxcLmpG5Yes='} 2023-10-17T19:03:45.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da57db3992973b92-00652eda91-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.215 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:45.216 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:45.216 DEBUG:boto:Method: HEAD 2023-10-17T19:03:45.217 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:45.217 DEBUG:boto:Data: 2023-10-17T19:03:45.217 DEBUG:boto:Headers: {} 2023-10-17T19:03:45.218 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.218 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.219 DEBUG:boto:Params: {} 2023-10-17T19:03:45.219 DEBUG:boto:Token: None 2023-10-17T19:03:45.219 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/ 2023-10-17T19:03:45.220 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ= 2023-10-17T19:03:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ=', 'Content-Length': '0'} 2023-10-17T19:03:45.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000867f33e5f85607fc-00652eda91-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.224 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:03:45.225 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:03:45.225 DEBUG:boto:Method: PUT 2023-10-17T19:03:45.226 DEBUG:boto:Path: /hbmuog-60/obj44 2023-10-17T19:03:45.226 DEBUG:boto:Data: 2023-10-17T19:03:45.226 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:03:45.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.227 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.227 DEBUG:boto:Params: {} 2023-10-17T19:03:45.228 DEBUG:boto:Token: None 2023-10-17T19:03:45.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/obj44 2023-10-17T19:03:45.229 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yu0+dx/EQMwaS7cPT7qsuRzUVUw= 2023-10-17T19:03:45.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yu0+dx/EQMwaS7cPT7qsuRzUVUw='} 2023-10-17T19:03:45.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000044406800afda706-00652eda91-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.236 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:03:45.237 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:03:45.237 DEBUG:boto:Method: HEAD 2023-10-17T19:03:45.237 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:03:45.238 DEBUG:boto:Data: 2023-10-17T19:03:45.238 DEBUG:boto:Headers: {} 2023-10-17T19:03:45.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.239 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.240 DEBUG:boto:Params: {} 2023-10-17T19:03:45.240 DEBUG:boto:Token: None 2023-10-17T19:03:45.240 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/ 2023-10-17T19:03:45.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ= 2023-10-17T19:03:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9NL1d5XRBo3q4rceZpgt4WXqmPQ=', 'Content-Length': '0'} 2023-10-17T19:03:45.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b851254ecdb2639-00652eda91-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.245 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:03:45.245 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:03:45.246 DEBUG:boto:Method: PUT 2023-10-17T19:03:45.246 DEBUG:boto:Path: /hbmuog-60/obj55 2023-10-17T19:03:45.246 DEBUG:boto:Data: 2023-10-17T19:03:45.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:03:45.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:03:45.248 DEBUG:boto:Port: 8000 2023-10-17T19:03:45.248 DEBUG:boto:Params: {} 2023-10-17T19:03:45.249 DEBUG:boto:Token: None 2023-10-17T19:03:45.249 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:03:45 GMT /hbmuog-60/obj55 2023-10-17T19:03:45.250 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P/m6RIx1I9uv5kUmhTgCx4Hv080= 2023-10-17T19:03:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:03:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P/m6RIx1I9uv5kUmhTgCx4Hv080='} 2023-10-17T19:03:45.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b75b702ccadde18-00652eda91-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:03:45.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:03:45.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:03:45.302 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:03:45.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:45.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:45.397+0000 7f8947fd7780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-17T19:03:45.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:45.398+0000 7f8947fd7780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-17T19:03:45.424 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T19:03:45.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T19:03:45.427 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T19:03:45.428 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:03:45.450 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:03:45.450 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:03:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T19:03:45.454 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T19:03:46.455 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:03:46.479 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:03:46.479 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:03:46.485 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2023-10-17T19:03:46.486 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T19:03:46.486 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-60 2023-10-17T19:03:46.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:03:46.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:03:46.488 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:03:46.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:03:46.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:03:46.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:46.620+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:03:50.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T19:03:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:51.621+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:03:56.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:03:56.621+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:01.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:01.622+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:06.623+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:11.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:11.624+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:16.625 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:16.624+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:21.626 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:21.624+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:26.626 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:26.625+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:31.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:31.625+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:36.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:36.626+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:41.626+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:46.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:46.628+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:51.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:51.629+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:04:56.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:04:56.628+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:01.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:01.629+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:06.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:06.630+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:11.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:11.630+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:16.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:16.631+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:21.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:21.632+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:26.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:26.633+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:31.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:31.632+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:36.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:36.633+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:41.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:41.634+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:46.634+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:51.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:51.635+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:05:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:05:56.637+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:06:01.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:01.637+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:06:06.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:06.638+0000 7fb8aef32780 1 waiting to reach incremental sync.. 2023-10-17T19:06:11.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:11.639+0000 7fb8aef32780 1 bucket sync caught up with source: 2023-10-17T19:06:11.641 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.52.6, 00000000002.67.6, 00000000002.50.6] 2023-10-17T19:06:11.641 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.52.6, 00000000002.67.6, 00000000002.50.6] 2023-10-17T19:06:11.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:11.639+0000 7fb8aef32780 0 bucket checkpoint complete 2023-10-17T19:06:11.659 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-60 2023-10-17T19:06:11.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:06:11.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:06:11.661 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:06:11.727 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:11.728 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:11.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:11.765+0000 7fb0525c7780 1 bucket sync caught up with source: 2023-10-17T19:06:11.767 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.291.6, 00000000002.235.6, 00000000002.294.6] 2023-10-17T19:06:11.768 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.291.6, 00000000002.235.6, 00000000002.294.6] 2023-10-17T19:06:11.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:11.765+0000 7fb0525c7780 0 bucket checkpoint complete 2023-10-17T19:06:11.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-60 zones={a2, a1} 2023-10-17T19:06:11.782 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.783 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.783 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.784 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:06:11.784 DEBUG:boto:Data: 2023-10-17T19:06:11.784 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.785 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.786 DEBUG:boto:Params: {} 2023-10-17T19:06:11.786 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:06:11.787 DEBUG:boto:Token: None 2023-10-17T19:06:11.788 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/ 2023-10-17T19:06:11.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MAknu4EpAIqQ+hVKqKuaxv/AQoo= 2023-10-17T19:06:11.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MAknu4EpAIqQ+hVKqKuaxv/AQoo=', 'Content-Length': '0'} 2023-10-17T19:06:11.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fb1c8c3032129b5-00652edb23-1c6e-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.794 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.795 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.795 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.796 DEBUG:boto:Path: /hbmuog-60/ 2023-10-17T19:06:11.796 DEBUG:boto:Data: 2023-10-17T19:06:11.797 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.797 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.798 DEBUG:boto:Params: {} 2023-10-17T19:06:11.798 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:06:11.799 DEBUG:boto:Token: None 2023-10-17T19:06:11.799 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/ 2023-10-17T19:06:11.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MAknu4EpAIqQ+hVKqKuaxv/AQoo= 2023-10-17T19:06:11.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MAknu4EpAIqQ+hVKqKuaxv/AQoo=', 'Content-Length': '0'} 2023-10-17T19:06:11.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71f3732e848dd5c-00652edb23-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.804 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T19:06:11.805 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.805 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.806 DEBUG:boto:path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.806 DEBUG:boto:auth_path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.807 DEBUG:boto:Method: GET 2023-10-17T19:06:11.807 DEBUG:boto:Path: /hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.807 DEBUG:boto:Data: 2023-10-17T19:06:11.808 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.808 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.809 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.809 DEBUG:boto:Params: {} 2023-10-17T19:06:11.810 DEBUG:boto:Token: None 2023-10-17T19:06:11.810 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/?versions 2023-10-17T19:06:11.811 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM= 2023-10-17T19:06:11.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM=', 'Content-Length': '0'} 2023-10-17T19:06:11.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029e7b6622f4f8c9b-00652edb23-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.814 DEBUG:boto:b'hbmuog-60999falseobj11nulltrue2023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:06:11.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:06:11.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:06:11.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:06:11.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:06:11.817 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T19:06:11.818 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.818 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.818 DEBUG:boto:path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.819 DEBUG:boto:auth_path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.819 DEBUG:boto:Method: GET 2023-10-17T19:06:11.820 DEBUG:boto:Path: /hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.820 DEBUG:boto:Data: 2023-10-17T19:06:11.820 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.821 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.822 DEBUG:boto:Params: {} 2023-10-17T19:06:11.822 DEBUG:boto:Token: None 2023-10-17T19:06:11.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/?versions 2023-10-17T19:06:11.823 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM= 2023-10-17T19:06:11.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM=', 'Content-Length': '0'} 2023-10-17T19:06:11.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005325d5aae547a050-00652edb23-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.827 DEBUG:boto:b'hbmuog-60999falseobj11nulltrue2023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:06:11.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:06:11.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:06:11.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:06:11.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:06:11.830 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.830 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.831 DEBUG:boto:path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.831 DEBUG:boto:auth_path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.832 DEBUG:boto:Method: GET 2023-10-17T19:06:11.832 DEBUG:boto:Path: /hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.833 DEBUG:boto:Data: 2023-10-17T19:06:11.833 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.834 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.834 DEBUG:boto:Params: {} 2023-10-17T19:06:11.835 DEBUG:boto:Token: None 2023-10-17T19:06:11.835 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/?versions 2023-10-17T19:06:11.836 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM= 2023-10-17T19:06:11.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM=', 'Content-Length': '0'} 2023-10-17T19:06:11.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b8f73505fa57ce-00652edb23-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.839 DEBUG:boto:b'hbmuog-60999falseobj11nulltrue2023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.840 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.840 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.841 DEBUG:boto:path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.841 DEBUG:boto:auth_path=/hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.842 DEBUG:boto:Method: GET 2023-10-17T19:06:11.842 DEBUG:boto:Path: /hbmuog-60/?versions&max-keys=999 2023-10-17T19:06:11.842 DEBUG:boto:Data: 2023-10-17T19:06:11.843 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.843 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.844 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.844 DEBUG:boto:Params: {} 2023-10-17T19:06:11.845 DEBUG:boto:Token: None 2023-10-17T19:06:11.845 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/?versions 2023-10-17T19:06:11.846 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM= 2023-10-17T19:06:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UeyHxsvJyZLpRYK4nVwMS4azRCM=', 'Content-Length': '0'} 2023-10-17T19:06:11.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac9da5d29197241-00652edb23-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.849 DEBUG:boto:b'hbmuog-60999falseobj11nulltrue2023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:06:11.850 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.851 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.851 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.852 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.852 DEBUG:boto:Method: GET 2023-10-17T19:06:11.853 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.853 DEBUG:boto:Data: 2023-10-17T19:06:11.853 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.854 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.854 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.855 DEBUG:boto:Params: {} 2023-10-17T19:06:11.855 DEBUG:boto:Token: None 2023-10-17T19:06:11.856 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM= 2023-10-17T19:06:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM=', 'Content-Length': '0'} 2023-10-17T19:06:11.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f90fd34897b4fe10-00652edb23-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.860 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.860 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.860 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.861 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.861 DEBUG:boto:Method: GET 2023-10-17T19:06:11.862 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.862 DEBUG:boto:Data: 2023-10-17T19:06:11.863 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.863 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.864 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.864 DEBUG:boto:Params: {} 2023-10-17T19:06:11.864 DEBUG:boto:Token: None 2023-10-17T19:06:11.865 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM= 2023-10-17T19:06:11.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM=', 'Content-Length': '0'} 2023-10-17T19:06:11.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d4daf28b9180327-00652edb23-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.868 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.869 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.869 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.869 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.870 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.870 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.871 DEBUG:boto:Data: 2023-10-17T19:06:11.871 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.872 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.872 DEBUG:boto:Params: {} 2023-10-17T19:06:11.873 DEBUG:boto:Token: None 2023-10-17T19:06:11.873 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lIolcW1oTvKLjWOIXxNIFiqYCR4= 2023-10-17T19:06:11.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lIolcW1oTvKLjWOIXxNIFiqYCR4=', 'Content-Length': '0'} 2023-10-17T19:06:11.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f51851824ad18e4-00652edb23-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.877 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.877 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.877 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.878 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.878 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.879 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.879 DEBUG:boto:Data: 2023-10-17T19:06:11.880 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.880 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.881 DEBUG:boto:Params: {} 2023-10-17T19:06:11.881 DEBUG:boto:Token: None 2023-10-17T19:06:11.882 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lIolcW1oTvKLjWOIXxNIFiqYCR4= 2023-10-17T19:06:11.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lIolcW1oTvKLjWOIXxNIFiqYCR4=', 'Content-Length': '0'} 2023-10-17T19:06:11.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb15d1651e78f977-00652edb23-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:06:11.885 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.886 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.886 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.887 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.887 DEBUG:boto:Method: GET 2023-10-17T19:06:11.887 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.888 DEBUG:boto:Data: 2023-10-17T19:06:11.888 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.889 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.889 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.890 DEBUG:boto:Params: {} 2023-10-17T19:06:11.890 DEBUG:boto:Token: None 2023-10-17T19:06:11.891 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.891 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM= 2023-10-17T19:06:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM=', 'Content-Length': '0'} 2023-10-17T19:06:11.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4dbfcd8da0edbd-00652edb23-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.894 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.894 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.895 DEBUG:boto:path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.895 DEBUG:boto:auth_path=/hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.896 DEBUG:boto:Method: GET 2023-10-17T19:06:11.896 DEBUG:boto:Path: /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.896 DEBUG:boto:Data: 2023-10-17T19:06:11.897 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.897 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.898 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.898 DEBUG:boto:Params: {} 2023-10-17T19:06:11.899 DEBUG:boto:Token: None 2023-10-17T19:06:11.899 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11?versionId=null 2023-10-17T19:06:11.900 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM= 2023-10-17T19:06:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0H+q83RcH8L3+Pz9ZRtL9shfWXM=', 'Content-Length': '0'} 2023-10-17T19:06:11.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1b17f9bb1bf8f3-00652edb23-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.902 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.903 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.903 DEBUG:boto:path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.904 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.904 DEBUG:boto:Method: GET 2023-10-17T19:06:11.905 DEBUG:boto:Path: /hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.905 DEBUG:boto:Data: 2023-10-17T19:06:11.905 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.906 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.907 DEBUG:boto:Params: {} 2023-10-17T19:06:11.907 DEBUG:boto:Token: None 2023-10-17T19:06:11.907 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/ 2023-10-17T19:06:11.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wprktCYpxGPhHFL8RT9PriI7aV0= 2023-10-17T19:06:11.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wprktCYpxGPhHFL8RT9PriI7aV0=', 'Content-Length': '0'} 2023-10-17T19:06:11.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af2668274871892-00652edb23-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.911 DEBUG:boto:b'hbmuog-60obj111000falseobj112023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.912 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.912 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:11.913 DEBUG:boto:path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.913 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.914 DEBUG:boto:Method: GET 2023-10-17T19:06:11.914 DEBUG:boto:Path: /hbmuog-60/?prefix=obj11 2023-10-17T19:06:11.915 DEBUG:boto:Data: 2023-10-17T19:06:11.915 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.915 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.916 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.916 DEBUG:boto:Params: {} 2023-10-17T19:06:11.917 DEBUG:boto:Token: None 2023-10-17T19:06:11.917 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/ 2023-10-17T19:06:11.918 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wprktCYpxGPhHFL8RT9PriI7aV0= 2023-10-17T19:06:11.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wprktCYpxGPhHFL8RT9PriI7aV0=', 'Content-Length': '0'} 2023-10-17T19:06:11.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027131fbd94c5d0a7-00652edb23-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.921 DEBUG:boto:b'hbmuog-60obj111000falseobj112023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:11.922 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.922 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.923 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.923 DEBUG:boto:Path: /hbmuog-60/obj11 2023-10-17T19:06:11.923 DEBUG:boto:Data: 2023-10-17T19:06:11.924 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.924 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.925 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.925 DEBUG:boto:Params: {} 2023-10-17T19:06:11.926 DEBUG:boto:Token: None 2023-10-17T19:06:11.926 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11 2023-10-17T19:06:11.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2kldI4dlFfJjHAvd3Yru5UlZh9U= 2023-10-17T19:06:11.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2kldI4dlFfJjHAvd3Yru5UlZh9U=', 'Content-Length': '0'} 2023-10-17T19:06:11.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd157f092858a04-00652edb23-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.929 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.930 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.930 DEBUG:boto:Method: HEAD 2023-10-17T19:06:11.931 DEBUG:boto:Path: /hbmuog-60/obj11 2023-10-17T19:06:11.931 DEBUG:boto:Data: 2023-10-17T19:06:11.932 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.932 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.932 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.933 DEBUG:boto:Params: {} 2023-10-17T19:06:11.933 DEBUG:boto:Token: None 2023-10-17T19:06:11.934 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11 2023-10-17T19:06:11.934 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2kldI4dlFfJjHAvd3Yru5UlZh9U= 2023-10-17T19:06:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2kldI4dlFfJjHAvd3Yru5UlZh9U=', 'Content-Length': '0'} 2023-10-17T19:06:11.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304999ee9f0c3f44-00652edb23-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:06:11.937 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.938 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.938 DEBUG:boto:Method: GET 2023-10-17T19:06:11.939 DEBUG:boto:Path: /hbmuog-60/obj11 2023-10-17T19:06:11.939 DEBUG:boto:Data: 2023-10-17T19:06:11.939 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.940 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:11.940 DEBUG:boto:Port: 8001 2023-10-17T19:06:11.941 DEBUG:boto:Params: {} 2023-10-17T19:06:11.941 DEBUG:boto:Token: None 2023-10-17T19:06:11.942 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11 2023-10-17T19:06:11.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tIgDcdyBdtf6zu6J62MeCB39niQ= 2023-10-17T19:06:11.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tIgDcdyBdtf6zu6J62MeCB39niQ=', 'Content-Length': '0'} 2023-10-17T19:06:11.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d38bbd57a8c9d4-00652edb23-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.990 DEBUG:boto:path=/hbmuog-60/obj11 2023-10-17T19:06:11.991 DEBUG:boto:auth_path=/hbmuog-60/obj11 2023-10-17T19:06:11.991 DEBUG:boto:Method: GET 2023-10-17T19:06:11.992 DEBUG:boto:Path: /hbmuog-60/obj11 2023-10-17T19:06:11.992 DEBUG:boto:Data: 2023-10-17T19:06:11.993 DEBUG:boto:Headers: {} 2023-10-17T19:06:11.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:11.994 DEBUG:boto:Port: 8000 2023-10-17T19:06:11.994 DEBUG:boto:Params: {} 2023-10-17T19:06:11.995 DEBUG:boto:Token: None 2023-10-17T19:06:11.995 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:11 GMT /hbmuog-60/obj11 2023-10-17T19:06:11.996 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tIgDcdyBdtf6zu6J62MeCB39niQ= 2023-10-17T19:06:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tIgDcdyBdtf6zu6J62MeCB39niQ=', 'Content-Length': '0'} 2023-10-17T19:06:11.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002720843b73115721-00652edb23-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:11.999 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:11.999 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.000 DEBUG:boto:path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.000 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.001 DEBUG:boto:Method: GET 2023-10-17T19:06:12.001 DEBUG:boto:Path: /hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.001 DEBUG:boto:Data: 2023-10-17T19:06:12.002 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.003 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.003 DEBUG:boto:Params: {} 2023-10-17T19:06:12.004 DEBUG:boto:Token: None 2023-10-17T19:06:12.004 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.005 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f952a3515d4bf0-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.008 DEBUG:boto:b'hbmuog-60obj111000falseobj112023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.009 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.009 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.010 DEBUG:boto:path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.010 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.011 DEBUG:boto:Method: GET 2023-10-17T19:06:12.011 DEBUG:boto:Path: /hbmuog-60/?prefix=obj11 2023-10-17T19:06:12.011 DEBUG:boto:Data: 2023-10-17T19:06:12.012 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.012 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.013 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.013 DEBUG:boto:Params: {} 2023-10-17T19:06:12.014 DEBUG:boto:Token: None 2023-10-17T19:06:12.014 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.015 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726c33e1f1d562f0-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.018 DEBUG:boto:b'hbmuog-60obj111000falseobj112023-10-17T19:03:45.136Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:06:12.019 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.020 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.020 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.021 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.021 DEBUG:boto:Method: GET 2023-10-17T19:06:12.021 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.022 DEBUG:boto:Data: 2023-10-17T19:06:12.022 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.023 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.024 DEBUG:boto:Params: {} 2023-10-17T19:06:12.024 DEBUG:boto:Token: None 2023-10-17T19:06:12.025 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg= 2023-10-17T19:06:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg=', 'Content-Length': '0'} 2023-10-17T19:06:12.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0eef4456c3917b-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.028 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.029 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.029 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.030 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.030 DEBUG:boto:Method: GET 2023-10-17T19:06:12.030 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.031 DEBUG:boto:Data: 2023-10-17T19:06:12.031 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.032 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.032 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.032 DEBUG:boto:Params: {} 2023-10-17T19:06:12.033 DEBUG:boto:Token: None 2023-10-17T19:06:12.033 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.034 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg= 2023-10-17T19:06:12.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg=', 'Content-Length': '0'} 2023-10-17T19:06:12.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a03b8e934b2f2109-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.037 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.037 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.038 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.038 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.039 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.039 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.039 DEBUG:boto:Data: 2023-10-17T19:06:12.040 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.041 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.041 DEBUG:boto:Params: {} 2023-10-17T19:06:12.042 DEBUG:boto:Token: None 2023-10-17T19:06:12.042 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.043 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:apTDVHlG7i85Ln6wpOY97Vkmjzg= 2023-10-17T19:06:12.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:apTDVHlG7i85Ln6wpOY97Vkmjzg=', 'Content-Length': '0'} 2023-10-17T19:06:12.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2b4312bdd6e9ab-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.046 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.046 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.046 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.047 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.047 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.048 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.048 DEBUG:boto:Data: 2023-10-17T19:06:12.048 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.049 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.049 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.050 DEBUG:boto:Params: {} 2023-10-17T19:06:12.050 DEBUG:boto:Token: None 2023-10-17T19:06:12.051 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.051 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:apTDVHlG7i85Ln6wpOY97Vkmjzg= 2023-10-17T19:06:12.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:apTDVHlG7i85Ln6wpOY97Vkmjzg=', 'Content-Length': '0'} 2023-10-17T19:06:12.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca322ad01e95bb0-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:06:12.054 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.055 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.055 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.056 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.056 DEBUG:boto:Method: GET 2023-10-17T19:06:12.057 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.057 DEBUG:boto:Data: 2023-10-17T19:06:12.057 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.058 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.058 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.059 DEBUG:boto:Params: {} 2023-10-17T19:06:12.059 DEBUG:boto:Token: None 2023-10-17T19:06:12.060 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.060 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg= 2023-10-17T19:06:12.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg=', 'Content-Length': '0'} 2023-10-17T19:06:12.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b75f09dea39178c-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.063 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.064 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.064 DEBUG:boto:path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.064 DEBUG:boto:auth_path=/hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.065 DEBUG:boto:Method: GET 2023-10-17T19:06:12.065 DEBUG:boto:Path: /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.066 DEBUG:boto:Data: 2023-10-17T19:06:12.066 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.067 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.067 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.068 DEBUG:boto:Params: {} 2023-10-17T19:06:12.068 DEBUG:boto:Token: None 2023-10-17T19:06:12.069 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22?versionId=null 2023-10-17T19:06:12.069 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg= 2023-10-17T19:06:12.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:i8dwCM6yrWx7hMFfpH27g5/PdUg=', 'Content-Length': '0'} 2023-10-17T19:06:12.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e6bd723fa6bbcc-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.073 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.073 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.074 DEBUG:boto:path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.074 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.074 DEBUG:boto:Method: GET 2023-10-17T19:06:12.075 DEBUG:boto:Path: /hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.075 DEBUG:boto:Data: 2023-10-17T19:06:12.076 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.077 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.077 DEBUG:boto:Params: {} 2023-10-17T19:06:12.078 DEBUG:boto:Token: None 2023-10-17T19:06:12.078 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148ea301eea57b50-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.122 DEBUG:boto:b'hbmuog-60obj221000falseobj222023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.123 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.124 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.124 DEBUG:boto:path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.125 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.125 DEBUG:boto:Method: GET 2023-10-17T19:06:12.126 DEBUG:boto:Path: /hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.126 DEBUG:boto:Data: 2023-10-17T19:06:12.127 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.127 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.128 DEBUG:boto:Params: {} 2023-10-17T19:06:12.128 DEBUG:boto:Token: None 2023-10-17T19:06:12.129 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006048886672c54a7c-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.133 DEBUG:boto:b'hbmuog-60obj221000falseobj222023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.134 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.134 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.135 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.135 DEBUG:boto:Path: /hbmuog-60/obj22 2023-10-17T19:06:12.136 DEBUG:boto:Data: 2023-10-17T19:06:12.136 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.137 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.137 DEBUG:boto:Params: {} 2023-10-17T19:06:12.138 DEBUG:boto:Token: None 2023-10-17T19:06:12.138 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22 2023-10-17T19:06:12.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nAXKu/jcH6OuGVDcvQxisv6RYoE= 2023-10-17T19:06:12.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nAXKu/jcH6OuGVDcvQxisv6RYoE=', 'Content-Length': '0'} 2023-10-17T19:06:12.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cc0643b42ffa42-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.142 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.142 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.143 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.143 DEBUG:boto:Path: /hbmuog-60/obj22 2023-10-17T19:06:12.143 DEBUG:boto:Data: 2023-10-17T19:06:12.144 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.145 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.145 DEBUG:boto:Params: {} 2023-10-17T19:06:12.145 DEBUG:boto:Token: None 2023-10-17T19:06:12.146 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22 2023-10-17T19:06:12.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nAXKu/jcH6OuGVDcvQxisv6RYoE= 2023-10-17T19:06:12.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nAXKu/jcH6OuGVDcvQxisv6RYoE=', 'Content-Length': '0'} 2023-10-17T19:06:12.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb24cb3c8cb0996-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:06:12.150 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.150 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.151 DEBUG:boto:Method: GET 2023-10-17T19:06:12.151 DEBUG:boto:Path: /hbmuog-60/obj22 2023-10-17T19:06:12.151 DEBUG:boto:Data: 2023-10-17T19:06:12.152 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.152 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.153 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.153 DEBUG:boto:Params: {} 2023-10-17T19:06:12.154 DEBUG:boto:Token: None 2023-10-17T19:06:12.154 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22 2023-10-17T19:06:12.155 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gMbt3A8qyoVMxPizOCdMwEktY7o= 2023-10-17T19:06:12.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gMbt3A8qyoVMxPizOCdMwEktY7o=', 'Content-Length': '0'} 2023-10-17T19:06:12.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f1c6934482525a-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.158 DEBUG:boto:path=/hbmuog-60/obj22 2023-10-17T19:06:12.158 DEBUG:boto:auth_path=/hbmuog-60/obj22 2023-10-17T19:06:12.158 DEBUG:boto:Method: GET 2023-10-17T19:06:12.159 DEBUG:boto:Path: /hbmuog-60/obj22 2023-10-17T19:06:12.159 DEBUG:boto:Data: 2023-10-17T19:06:12.160 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.161 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.161 DEBUG:boto:Params: {} 2023-10-17T19:06:12.161 DEBUG:boto:Token: None 2023-10-17T19:06:12.162 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj22 2023-10-17T19:06:12.162 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gMbt3A8qyoVMxPizOCdMwEktY7o= 2023-10-17T19:06:12.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gMbt3A8qyoVMxPizOCdMwEktY7o=', 'Content-Length': '0'} 2023-10-17T19:06:12.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0cdefd0dd41b5e6-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.167 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.167 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.168 DEBUG:boto:path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.168 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.168 DEBUG:boto:Method: GET 2023-10-17T19:06:12.169 DEBUG:boto:Path: /hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.169 DEBUG:boto:Data: 2023-10-17T19:06:12.170 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.170 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.171 DEBUG:boto:Params: {} 2023-10-17T19:06:12.171 DEBUG:boto:Token: None 2023-10-17T19:06:12.172 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003646cfbb6cc5b9ab-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.176 DEBUG:boto:b'hbmuog-60obj221000falseobj222023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.176 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.177 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.177 DEBUG:boto:path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.178 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.178 DEBUG:boto:Method: GET 2023-10-17T19:06:12.179 DEBUG:boto:Path: /hbmuog-60/?prefix=obj22 2023-10-17T19:06:12.179 DEBUG:boto:Data: 2023-10-17T19:06:12.179 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.180 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.180 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.181 DEBUG:boto:Params: {} 2023-10-17T19:06:12.181 DEBUG:boto:Token: None 2023-10-17T19:06:12.182 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.182 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af2e55cdd94ff86d-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.186 DEBUG:boto:b'hbmuog-60obj221000falseobj222023-10-17T19:03:45.156Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:06:12.187 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.187 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.188 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.188 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.188 DEBUG:boto:Method: GET 2023-10-17T19:06:12.189 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.189 DEBUG:boto:Data: 2023-10-17T19:06:12.190 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.191 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.191 DEBUG:boto:Params: {} 2023-10-17T19:06:12.192 DEBUG:boto:Token: None 2023-10-17T19:06:12.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA= 2023-10-17T19:06:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA=', 'Content-Length': '0'} 2023-10-17T19:06:12.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e8c2fe9710bad29-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.195 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.196 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.196 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.197 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.197 DEBUG:boto:Method: GET 2023-10-17T19:06:12.198 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.198 DEBUG:boto:Data: 2023-10-17T19:06:12.198 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.199 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.200 DEBUG:boto:Params: {} 2023-10-17T19:06:12.200 DEBUG:boto:Token: None 2023-10-17T19:06:12.201 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA= 2023-10-17T19:06:12.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA=', 'Content-Length': '0'} 2023-10-17T19:06:12.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009643207ac908ba3a-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.204 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.205 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.205 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.205 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.206 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.206 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.206 DEBUG:boto:Data: 2023-10-17T19:06:12.207 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.207 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.208 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.208 DEBUG:boto:Params: {} 2023-10-17T19:06:12.208 DEBUG:boto:Token: None 2023-10-17T19:06:12.209 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PreENa7tMYHCmiRHR4bbi819nw8= 2023-10-17T19:06:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PreENa7tMYHCmiRHR4bbi819nw8=', 'Content-Length': '0'} 2023-10-17T19:06:12.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5fb8adc52c8c7b0-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.212 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.213 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.213 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.213 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.214 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.214 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.214 DEBUG:boto:Data: 2023-10-17T19:06:12.215 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.215 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.216 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.216 DEBUG:boto:Params: {} 2023-10-17T19:06:12.216 DEBUG:boto:Token: None 2023-10-17T19:06:12.217 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.217 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PreENa7tMYHCmiRHR4bbi819nw8= 2023-10-17T19:06:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PreENa7tMYHCmiRHR4bbi819nw8=', 'Content-Length': '0'} 2023-10-17T19:06:12.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ff32e9fcd291553-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:06:12.220 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.221 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.221 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.222 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.222 DEBUG:boto:Method: GET 2023-10-17T19:06:12.222 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.223 DEBUG:boto:Data: 2023-10-17T19:06:12.223 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.223 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.224 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.224 DEBUG:boto:Params: {} 2023-10-17T19:06:12.225 DEBUG:boto:Token: None 2023-10-17T19:06:12.225 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.226 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA= 2023-10-17T19:06:12.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA=', 'Content-Length': '0'} 2023-10-17T19:06:12.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc0dcb700b4e4d2-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.270 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.271 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.271 DEBUG:boto:path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.272 DEBUG:boto:auth_path=/hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.272 DEBUG:boto:Method: GET 2023-10-17T19:06:12.273 DEBUG:boto:Path: /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.273 DEBUG:boto:Data: 2023-10-17T19:06:12.273 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.274 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.274 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.275 DEBUG:boto:Params: {} 2023-10-17T19:06:12.275 DEBUG:boto:Token: None 2023-10-17T19:06:12.276 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33?versionId=null 2023-10-17T19:06:12.276 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA= 2023-10-17T19:06:12.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LRTi1zK3n/PGRXUfF1EVFo6WegA=', 'Content-Length': '0'} 2023-10-17T19:06:12.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c8f4f7377166703-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.279 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.280 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.280 DEBUG:boto:path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.281 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.281 DEBUG:boto:Method: GET 2023-10-17T19:06:12.281 DEBUG:boto:Path: /hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.282 DEBUG:boto:Data: 2023-10-17T19:06:12.282 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.283 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.283 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.284 DEBUG:boto:Params: {} 2023-10-17T19:06:12.284 DEBUG:boto:Token: None 2023-10-17T19:06:12.284 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.285 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b233d92e0b262fa-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.289 DEBUG:boto:b'hbmuog-60obj331000falseobj332023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.289 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.290 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.290 DEBUG:boto:path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.291 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.291 DEBUG:boto:Method: GET 2023-10-17T19:06:12.292 DEBUG:boto:Path: /hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.292 DEBUG:boto:Data: 2023-10-17T19:06:12.292 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.293 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.293 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.294 DEBUG:boto:Params: {} 2023-10-17T19:06:12.294 DEBUG:boto:Token: None 2023-10-17T19:06:12.295 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011a639f629b5f6dc-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.299 DEBUG:boto:b'hbmuog-60obj331000falseobj332023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.299 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.300 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.300 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.301 DEBUG:boto:Path: /hbmuog-60/obj33 2023-10-17T19:06:12.301 DEBUG:boto:Data: 2023-10-17T19:06:12.301 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.302 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.303 DEBUG:boto:Params: {} 2023-10-17T19:06:12.303 DEBUG:boto:Token: None 2023-10-17T19:06:12.304 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33 2023-10-17T19:06:12.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7FotcazxX3yiQHDrDvbQzhOi2M0= 2023-10-17T19:06:12.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7FotcazxX3yiQHDrDvbQzhOi2M0=', 'Content-Length': '0'} 2023-10-17T19:06:12.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d296215f327caaf4-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.307 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.307 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.308 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.308 DEBUG:boto:Path: /hbmuog-60/obj33 2023-10-17T19:06:12.308 DEBUG:boto:Data: 2023-10-17T19:06:12.309 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.309 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.310 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.310 DEBUG:boto:Params: {} 2023-10-17T19:06:12.311 DEBUG:boto:Token: None 2023-10-17T19:06:12.311 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33 2023-10-17T19:06:12.311 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7FotcazxX3yiQHDrDvbQzhOi2M0= 2023-10-17T19:06:12.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7FotcazxX3yiQHDrDvbQzhOi2M0=', 'Content-Length': '0'} 2023-10-17T19:06:12.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a67b81eb53372b2-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:06:12.314 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.315 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.315 DEBUG:boto:Method: GET 2023-10-17T19:06:12.316 DEBUG:boto:Path: /hbmuog-60/obj33 2023-10-17T19:06:12.316 DEBUG:boto:Data: 2023-10-17T19:06:12.317 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.317 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.317 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.318 DEBUG:boto:Params: {} 2023-10-17T19:06:12.318 DEBUG:boto:Token: None 2023-10-17T19:06:12.319 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33 2023-10-17T19:06:12.319 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zXTYQUS3uLziqL9+gsyWDTZ0jc= 2023-10-17T19:06:12.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zXTYQUS3uLziqL9+gsyWDTZ0jc=', 'Content-Length': '0'} 2023-10-17T19:06:12.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6387f181560cd09-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.322 DEBUG:boto:path=/hbmuog-60/obj33 2023-10-17T19:06:12.322 DEBUG:boto:auth_path=/hbmuog-60/obj33 2023-10-17T19:06:12.323 DEBUG:boto:Method: GET 2023-10-17T19:06:12.323 DEBUG:boto:Path: /hbmuog-60/obj33 2023-10-17T19:06:12.324 DEBUG:boto:Data: 2023-10-17T19:06:12.324 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.324 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.325 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.325 DEBUG:boto:Params: {} 2023-10-17T19:06:12.326 DEBUG:boto:Token: None 2023-10-17T19:06:12.326 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj33 2023-10-17T19:06:12.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4zXTYQUS3uLziqL9+gsyWDTZ0jc= 2023-10-17T19:06:12.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4zXTYQUS3uLziqL9+gsyWDTZ0jc=', 'Content-Length': '0'} 2023-10-17T19:06:12.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0fbe2314db9dd2-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.370 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.371 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.372 DEBUG:boto:path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.372 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.372 DEBUG:boto:Method: GET 2023-10-17T19:06:12.373 DEBUG:boto:Path: /hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.373 DEBUG:boto:Data: 2023-10-17T19:06:12.374 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.375 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.375 DEBUG:boto:Params: {} 2023-10-17T19:06:12.376 DEBUG:boto:Token: None 2023-10-17T19:06:12.376 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.377 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d25de63cdd8b45-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.380 DEBUG:boto:b'hbmuog-60obj331000falseobj332023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.381 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.381 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.382 DEBUG:boto:path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.382 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.383 DEBUG:boto:Method: GET 2023-10-17T19:06:12.383 DEBUG:boto:Path: /hbmuog-60/?prefix=obj33 2023-10-17T19:06:12.384 DEBUG:boto:Data: 2023-10-17T19:06:12.384 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.385 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.385 DEBUG:boto:Params: {} 2023-10-17T19:06:12.386 DEBUG:boto:Token: None 2023-10-17T19:06:12.386 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e9599dd450cd4e-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.391 DEBUG:boto:b'hbmuog-60obj331000falseobj332023-10-17T19:03:45.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:06:12.392 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.392 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.393 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.393 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.393 DEBUG:boto:Method: GET 2023-10-17T19:06:12.394 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.394 DEBUG:boto:Data: 2023-10-17T19:06:12.395 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.395 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.396 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.396 DEBUG:boto:Params: {} 2023-10-17T19:06:12.397 DEBUG:boto:Token: None 2023-10-17T19:06:12.397 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.398 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s= 2023-10-17T19:06:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s=', 'Content-Length': '0'} 2023-10-17T19:06:12.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef61cf583f63bf1d-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.446 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.447 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.447 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.448 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.448 DEBUG:boto:Method: GET 2023-10-17T19:06:12.449 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.449 DEBUG:boto:Data: 2023-10-17T19:06:12.450 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.450 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.451 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.451 DEBUG:boto:Params: {} 2023-10-17T19:06:12.452 DEBUG:boto:Token: None 2023-10-17T19:06:12.452 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s= 2023-10-17T19:06:12.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s=', 'Content-Length': '0'} 2023-10-17T19:06:12.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4de0ea749b26792-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.456 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.456 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.457 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.457 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.458 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.458 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.458 DEBUG:boto:Data: 2023-10-17T19:06:12.459 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.459 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.460 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.460 DEBUG:boto:Params: {} 2023-10-17T19:06:12.461 DEBUG:boto:Token: None 2023-10-17T19:06:12.461 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.462 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rO6fbMkUtkFvA8srsePKifsVKTA= 2023-10-17T19:06:12.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rO6fbMkUtkFvA8srsePKifsVKTA=', 'Content-Length': '0'} 2023-10-17T19:06:12.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da00f2138029643d-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.465 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.465 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.466 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.466 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.467 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.467 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.467 DEBUG:boto:Data: 2023-10-17T19:06:12.468 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.469 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.469 DEBUG:boto:Params: {} 2023-10-17T19:06:12.470 DEBUG:boto:Token: None 2023-10-17T19:06:12.470 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rO6fbMkUtkFvA8srsePKifsVKTA= 2023-10-17T19:06:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rO6fbMkUtkFvA8srsePKifsVKTA=', 'Content-Length': '0'} 2023-10-17T19:06:12.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a62df03d8b53886-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:06:12.474 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.474 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.475 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.475 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.476 DEBUG:boto:Method: GET 2023-10-17T19:06:12.476 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.477 DEBUG:boto:Data: 2023-10-17T19:06:12.477 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.478 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.478 DEBUG:boto:Params: {} 2023-10-17T19:06:12.479 DEBUG:boto:Token: None 2023-10-17T19:06:12.479 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s= 2023-10-17T19:06:12.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s=', 'Content-Length': '0'} 2023-10-17T19:06:12.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031a4997d942111f-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.483 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.483 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.484 DEBUG:boto:path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.484 DEBUG:boto:auth_path=/hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.484 DEBUG:boto:Method: GET 2023-10-17T19:06:12.485 DEBUG:boto:Path: /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.485 DEBUG:boto:Data: 2023-10-17T19:06:12.486 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.487 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.487 DEBUG:boto:Params: {} 2023-10-17T19:06:12.488 DEBUG:boto:Token: None 2023-10-17T19:06:12.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44?versionId=null 2023-10-17T19:06:12.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s= 2023-10-17T19:06:12.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p06KPMLdNG7PfqG3eb3WJ4eB71s=', 'Content-Length': '0'} 2023-10-17T19:06:12.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cf78dee2d47d83-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.491 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.492 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.492 DEBUG:boto:path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.493 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.493 DEBUG:boto:Method: GET 2023-10-17T19:06:12.494 DEBUG:boto:Path: /hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.494 DEBUG:boto:Data: 2023-10-17T19:06:12.494 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.495 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.496 DEBUG:boto:Params: {} 2023-10-17T19:06:12.496 DEBUG:boto:Token: None 2023-10-17T19:06:12.496 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005067a9b149ce3ae1-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.500 DEBUG:boto:b'hbmuog-60obj441000falseobj442023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.501 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.501 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.502 DEBUG:boto:path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.502 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.502 DEBUG:boto:Method: GET 2023-10-17T19:06:12.503 DEBUG:boto:Path: /hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.503 DEBUG:boto:Data: 2023-10-17T19:06:12.504 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.505 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.505 DEBUG:boto:Params: {} 2023-10-17T19:06:12.505 DEBUG:boto:Token: None 2023-10-17T19:06:12.506 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.506 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cce18896a222580-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.550 DEBUG:boto:b'hbmuog-60obj441000falseobj442023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.551 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.551 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.552 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.552 DEBUG:boto:Path: /hbmuog-60/obj44 2023-10-17T19:06:12.553 DEBUG:boto:Data: 2023-10-17T19:06:12.553 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.554 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.554 DEBUG:boto:Params: {} 2023-10-17T19:06:12.555 DEBUG:boto:Token: None 2023-10-17T19:06:12.555 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44 2023-10-17T19:06:12.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QOqSIjqdV/lcJtfV12+lOC9KcrU= 2023-10-17T19:06:12.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QOqSIjqdV/lcJtfV12+lOC9KcrU=', 'Content-Length': '0'} 2023-10-17T19:06:12.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c07cf5ed9604f55-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.559 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.559 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.560 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.560 DEBUG:boto:Path: /hbmuog-60/obj44 2023-10-17T19:06:12.560 DEBUG:boto:Data: 2023-10-17T19:06:12.561 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.562 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.562 DEBUG:boto:Params: {} 2023-10-17T19:06:12.562 DEBUG:boto:Token: None 2023-10-17T19:06:12.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44 2023-10-17T19:06:12.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QOqSIjqdV/lcJtfV12+lOC9KcrU= 2023-10-17T19:06:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QOqSIjqdV/lcJtfV12+lOC9KcrU=', 'Content-Length': '0'} 2023-10-17T19:06:12.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aaedc36113b9637-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:06:12.567 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.567 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.568 DEBUG:boto:Method: GET 2023-10-17T19:06:12.568 DEBUG:boto:Path: /hbmuog-60/obj44 2023-10-17T19:06:12.568 DEBUG:boto:Data: 2023-10-17T19:06:12.569 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.570 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.570 DEBUG:boto:Params: {} 2023-10-17T19:06:12.571 DEBUG:boto:Token: None 2023-10-17T19:06:12.571 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44 2023-10-17T19:06:12.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aox6SCpFU1H/FSnd3y2DvO5ZJWA= 2023-10-17T19:06:12.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aox6SCpFU1H/FSnd3y2DvO5ZJWA=', 'Content-Length': '0'} 2023-10-17T19:06:12.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74b0313be459d84-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.575 DEBUG:boto:path=/hbmuog-60/obj44 2023-10-17T19:06:12.575 DEBUG:boto:auth_path=/hbmuog-60/obj44 2023-10-17T19:06:12.575 DEBUG:boto:Method: GET 2023-10-17T19:06:12.576 DEBUG:boto:Path: /hbmuog-60/obj44 2023-10-17T19:06:12.576 DEBUG:boto:Data: 2023-10-17T19:06:12.577 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.577 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.578 DEBUG:boto:Params: {} 2023-10-17T19:06:12.578 DEBUG:boto:Token: None 2023-10-17T19:06:12.579 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj44 2023-10-17T19:06:12.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aox6SCpFU1H/FSnd3y2DvO5ZJWA= 2023-10-17T19:06:12.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aox6SCpFU1H/FSnd3y2DvO5ZJWA=', 'Content-Length': '0'} 2023-10-17T19:06:12.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075744c6c76bbe0e-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.582 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.583 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.583 DEBUG:boto:path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.584 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.584 DEBUG:boto:Method: GET 2023-10-17T19:06:12.584 DEBUG:boto:Path: /hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.585 DEBUG:boto:Data: 2023-10-17T19:06:12.585 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.586 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.586 DEBUG:boto:Params: {} 2023-10-17T19:06:12.587 DEBUG:boto:Token: None 2023-10-17T19:06:12.587 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c55c7b156305da-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.591 DEBUG:boto:b'hbmuog-60obj441000falseobj442023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.592 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.592 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.593 DEBUG:boto:path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.593 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.594 DEBUG:boto:Method: GET 2023-10-17T19:06:12.594 DEBUG:boto:Path: /hbmuog-60/?prefix=obj44 2023-10-17T19:06:12.594 DEBUG:boto:Data: 2023-10-17T19:06:12.595 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.596 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.596 DEBUG:boto:Params: {} 2023-10-17T19:06:12.596 DEBUG:boto:Token: None 2023-10-17T19:06:12.597 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000439ed72bd406c107-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.601 DEBUG:boto:b'hbmuog-60obj441000falseobj442023-10-17T19:03:45.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:06:12.602 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.602 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.603 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.603 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.603 DEBUG:boto:Method: GET 2023-10-17T19:06:12.604 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.604 DEBUG:boto:Data: 2023-10-17T19:06:12.605 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.605 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.605 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.606 DEBUG:boto:Params: {} 2023-10-17T19:06:12.606 DEBUG:boto:Token: None 2023-10-17T19:06:12.607 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.607 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc= 2023-10-17T19:06:12.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc=', 'Content-Length': '0'} 2023-10-17T19:06:12.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba0d352c8a400c3-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.650 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.651 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.651 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.652 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.652 DEBUG:boto:Method: GET 2023-10-17T19:06:12.653 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.653 DEBUG:boto:Data: 2023-10-17T19:06:12.653 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.654 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.654 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.655 DEBUG:boto:Params: {} 2023-10-17T19:06:12.655 DEBUG:boto:Token: None 2023-10-17T19:06:12.656 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.656 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc= 2023-10-17T19:06:12.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc=', 'Content-Length': '0'} 2023-10-17T19:06:12.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8983dfa83d87c3-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.659 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.660 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.660 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.660 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.661 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.661 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.662 DEBUG:boto:Data: 2023-10-17T19:06:12.662 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.663 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.663 DEBUG:boto:Params: {} 2023-10-17T19:06:12.664 DEBUG:boto:Token: None 2023-10-17T19:06:12.664 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/sLoB4HdN1FvPoHlv4byBLUylZI= 2023-10-17T19:06:12.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/sLoB4HdN1FvPoHlv4byBLUylZI=', 'Content-Length': '0'} 2023-10-17T19:06:12.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6299f9094b3b54-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.667 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.668 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.668 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.669 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.669 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.669 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.670 DEBUG:boto:Data: 2023-10-17T19:06:12.670 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.671 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.671 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.671 DEBUG:boto:Params: {} 2023-10-17T19:06:12.672 DEBUG:boto:Token: None 2023-10-17T19:06:12.672 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/sLoB4HdN1FvPoHlv4byBLUylZI= 2023-10-17T19:06:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/sLoB4HdN1FvPoHlv4byBLUylZI=', 'Content-Length': '0'} 2023-10-17T19:06:12.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9216561ba8b917a-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:06:12.676 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.676 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.677 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.677 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.678 DEBUG:boto:Method: GET 2023-10-17T19:06:12.678 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.679 DEBUG:boto:Data: 2023-10-17T19:06:12.679 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.680 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.680 DEBUG:boto:Params: {} 2023-10-17T19:06:12.681 DEBUG:boto:Token: None 2023-10-17T19:06:12.681 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc= 2023-10-17T19:06:12.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc=', 'Content-Length': '0'} 2023-10-17T19:06:12.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ca112b5b34a25d-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.685 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.685 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.685 DEBUG:boto:path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.686 DEBUG:boto:auth_path=/hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.686 DEBUG:boto:Method: GET 2023-10-17T19:06:12.687 DEBUG:boto:Path: /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.687 DEBUG:boto:Data: 2023-10-17T19:06:12.688 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.688 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.688 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.689 DEBUG:boto:Params: {} 2023-10-17T19:06:12.689 DEBUG:boto:Token: None 2023-10-17T19:06:12.690 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55?versionId=null 2023-10-17T19:06:12.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc= 2023-10-17T19:06:12.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EhfDl1Xi6EU+Z8sqWBBJARy5aBc=', 'Content-Length': '0'} 2023-10-17T19:06:12.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4a0bdf877d9a67-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.734 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.735 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.735 DEBUG:boto:path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.736 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.736 DEBUG:boto:Method: GET 2023-10-17T19:06:12.737 DEBUG:boto:Path: /hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.737 DEBUG:boto:Data: 2023-10-17T19:06:12.737 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.738 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.739 DEBUG:boto:Params: {} 2023-10-17T19:06:12.739 DEBUG:boto:Token: None 2023-10-17T19:06:12.740 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.740 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a31b09667ba2df9f-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.744 DEBUG:boto:b'hbmuog-60obj551000falseobj552023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.744 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.745 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.745 DEBUG:boto:path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.746 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.746 DEBUG:boto:Method: GET 2023-10-17T19:06:12.747 DEBUG:boto:Path: /hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.747 DEBUG:boto:Data: 2023-10-17T19:06:12.747 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.748 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.748 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.749 DEBUG:boto:Params: {} 2023-10-17T19:06:12.749 DEBUG:boto:Token: None 2023-10-17T19:06:12.750 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.750 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879b580230f2cd37-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.753 DEBUG:boto:b'hbmuog-60obj551000falseobj552023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.754 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.755 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.755 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.755 DEBUG:boto:Path: /hbmuog-60/obj55 2023-10-17T19:06:12.756 DEBUG:boto:Data: 2023-10-17T19:06:12.756 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.757 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.757 DEBUG:boto:Params: {} 2023-10-17T19:06:12.758 DEBUG:boto:Token: None 2023-10-17T19:06:12.758 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55 2023-10-17T19:06:12.759 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WkIa0ubJiXuj/6JnhS9ulg2CSSQ= 2023-10-17T19:06:12.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WkIa0ubJiXuj/6JnhS9ulg2CSSQ=', 'Content-Length': '0'} 2023-10-17T19:06:12.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22b1db9b0afdbc6-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.761 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.762 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.762 DEBUG:boto:Method: HEAD 2023-10-17T19:06:12.763 DEBUG:boto:Path: /hbmuog-60/obj55 2023-10-17T19:06:12.763 DEBUG:boto:Data: 2023-10-17T19:06:12.763 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.764 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.764 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.765 DEBUG:boto:Params: {} 2023-10-17T19:06:12.765 DEBUG:boto:Token: None 2023-10-17T19:06:12.765 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55 2023-10-17T19:06:12.766 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WkIa0ubJiXuj/6JnhS9ulg2CSSQ= 2023-10-17T19:06:12.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WkIa0ubJiXuj/6JnhS9ulg2CSSQ=', 'Content-Length': '0'} 2023-10-17T19:06:12.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009314310558ee5e7f-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:06:12.769 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.770 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.770 DEBUG:boto:Method: GET 2023-10-17T19:06:12.771 DEBUG:boto:Path: /hbmuog-60/obj55 2023-10-17T19:06:12.771 DEBUG:boto:Data: 2023-10-17T19:06:12.771 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.772 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.772 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.773 DEBUG:boto:Params: {} 2023-10-17T19:06:12.773 DEBUG:boto:Token: None 2023-10-17T19:06:12.774 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55 2023-10-17T19:06:12.774 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NhMosIE0MyZH9SlsSt1AJ7B/N+E= 2023-10-17T19:06:12.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NhMosIE0MyZH9SlsSt1AJ7B/N+E=', 'Content-Length': '0'} 2023-10-17T19:06:12.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-60:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164318cac515642a-00652edb24-1c6e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.826 DEBUG:boto:path=/hbmuog-60/obj55 2023-10-17T19:06:12.827 DEBUG:boto:auth_path=/hbmuog-60/obj55 2023-10-17T19:06:12.828 DEBUG:boto:Method: GET 2023-10-17T19:06:12.828 DEBUG:boto:Path: /hbmuog-60/obj55 2023-10-17T19:06:12.828 DEBUG:boto:Data: 2023-10-17T19:06:12.829 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.829 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.830 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.830 DEBUG:boto:Params: {} 2023-10-17T19:06:12.831 DEBUG:boto:Token: None 2023-10-17T19:06:12.831 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/obj55 2023-10-17T19:06:12.832 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NhMosIE0MyZH9SlsSt1AJ7B/N+E= 2023-10-17T19:06:12.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NhMosIE0MyZH9SlsSt1AJ7B/N+E=', 'Content-Length': '0'} 2023-10-17T19:06:12.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:03:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e227b5e0318fecdc-00652edb24-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.835 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.835 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.836 DEBUG:boto:path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.836 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.836 DEBUG:boto:Method: GET 2023-10-17T19:06:12.837 DEBUG:boto:Path: /hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.837 DEBUG:boto:Data: 2023-10-17T19:06:12.838 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.838 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:12.838 DEBUG:boto:Port: 8001 2023-10-17T19:06:12.839 DEBUG:boto:Params: {} 2023-10-17T19:06:12.839 DEBUG:boto:Token: None 2023-10-17T19:06:12.840 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.840 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3381ee42626215-00652edb24-1c6e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.843 DEBUG:boto:b'hbmuog-60obj551000falseobj552023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.844 DEBUG:boto:path=/hbmuog-60/ 2023-10-17T19:06:12.844 DEBUG:boto:auth_path=/hbmuog-60/ 2023-10-17T19:06:12.845 DEBUG:boto:path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.845 DEBUG:boto:auth_path=/hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.845 DEBUG:boto:Method: GET 2023-10-17T19:06:12.846 DEBUG:boto:Path: /hbmuog-60/?prefix=obj55 2023-10-17T19:06:12.846 DEBUG:boto:Data: 2023-10-17T19:06:12.847 DEBUG:boto:Headers: {} 2023-10-17T19:06:12.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:12.848 DEBUG:boto:Port: 8000 2023-10-17T19:06:12.848 DEBUG:boto:Params: {} 2023-10-17T19:06:12.849 DEBUG:boto:Token: None 2023-10-17T19:06:12.849 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:06:12 GMT /hbmuog-60/ 2023-10-17T19:06:12.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc= 2023-10-17T19:06:12.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zn1ut+WouGNvTlJjRC++sKrOkAc=', 'Content-Length': '0'} 2023-10-17T19:06:12.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2a24f79f1a07aa-00652edb24-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:06:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:12.853 DEBUG:boto:b'hbmuog-60obj551000falseobj552023-10-17T19:03:45.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:06:12.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-60 zones={a2, a1} 2023-10-17T19:06:12.854 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T19:06:12.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:12.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:12.855 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:12.905 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:12.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:12.954 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:06:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000022678", 2023-10-17T19:06:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:57.276315Z" 2023-10-17T19:06:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000021160", 2023-10-17T19:06:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:57.277922Z" 2023-10-17T19:06:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000015351", 2023-10-17T19:06:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:58:49.273195Z" 2023-10-17T19:06:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000025924", 2023-10-17T19:06:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:45.234929Z" 2023-10-17T19:06:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:12.958 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:06:12.959 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000022678', 1: '00000000000000000000:00000000000000021160', 2: '00000000000000000000:00000000000000015351', 3: '00000000000000000000:00000000000000025924'} 2023-10-17T19:06:12.959 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:06:12.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:12.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:12.961 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:13.058 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:06:13.059 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:06:13.059 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:06:13.059 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:06:13.059 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:06:13.059 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 5220910224406309578 2023-10-17T19:06:13.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.060 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:06:13.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.060 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:06:13.060 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.061 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.061 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000022678", 2023-10-17T19:06:13.061 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.061 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.062 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:57.276315Z" 2023-10-17T19:06:13.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000021160", 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.063 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:57.277922Z" 2023-10-17T19:06:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.065 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:06:13.065 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.065 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.065 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000015351", 2023-10-17T19:06:13.065 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.066 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:06:13.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.066 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:58:49.273195Z" 2023-10-17T19:06:13.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.067 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:06:13.067 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.067 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.068 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000025924", 2023-10-17T19:06:13.068 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.068 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 51, 2023-10-17T19:06:13.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.068 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:45.234929Z" 2023-10-17T19:06:13.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.069 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:06:13.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.069 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:06:13.070 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 60, 2023-10-17T19:06:13.070 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 60 2023-10-17T19:06:13.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.070 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:06:13Z" 2023-10-17T19:06:13.070 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:06:13.072 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5220910224406309578 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022678", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:57.276315Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021160", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:57.277922Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015351", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T18:58:49.273195Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025924", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-10-17T19:03:45.234929Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-10-17T19:06:13Z" } 2023-10-17T19:06:13.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022678', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:57.276315Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021160', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:57.277922Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015351', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T18:58:49.273195Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025924', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-10-17T19:03:45.234929Z'}}] 2023-10-17T19:06:13.072 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000022678', 1: '00000000000000000000:00000000000000021160', 2: '00000000000000000000:00000000000000015351', 3: '00000000000000000000:00000000000000025924'} 2023-10-17T19:06:13.073 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000022678', 1: '00000000000000000000:00000000000000021160', 2: '00000000000000000000:00000000000000015351', 3: '00000000000000000000:00000000000000025924'} 2023-10-17T19:06:13.073 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:06:13.074 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T19:06:13.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.075 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:13.173 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:06:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008968", 2023-10-17T19:06:13.173 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:04:51.539431Z" 2023-10-17T19:06:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000007856", 2023-10-17T19:06:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:05:19.923568Z" 2023-10-17T19:06:13.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004313", 2023-10-17T19:06:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T18:59:32.063173Z" 2023-10-17T19:06:13.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012063", 2023-10-17T19:06:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:04:49.299828Z" 2023-10-17T19:06:13.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.176 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:06:13.177 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000008968', 1: '00000000000000000000:00000000000000007856', 2: '00000000000000000000:00000000000000004313', 3: '00000000000000000000:00000000000000012063'} 2023-10-17T19:06:13.178 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:06:13.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.179 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:13.232 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:13.232 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:13.281 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:06:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:06:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:06:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:06:13.282 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:06:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T19:06:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:06:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.283 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:06:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008968", 2023-10-17T19:06:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.284 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:04:51.539431Z" 2023-10-17T19:06:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:06:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000007856", 2023-10-17T19:06:13.286 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:05:19.923568Z" 2023-10-17T19:06:13.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:06:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.288 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004313", 2023-10-17T19:06:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.289 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T18:59:32.063173Z" 2023-10-17T19:06:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.290 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:06:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:06:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012063", 2023-10-17T19:06:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:04:49.299828Z" 2023-10-17T19:06:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:06:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:06:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T19:06:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T19:06:13.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.294 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:06:13Z" 2023-10-17T19:06:13.294 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:06:13.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008968", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:04:51.539431Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007856", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:05:19.923568Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004313", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T18:59:32.063173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:04:49.299828Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T19:06:13Z" } 2023-10-17T19:06:13.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008968', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:04:51.539431Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007856', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:05:19.923568Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004313', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:59:32.063173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:04:49.299828Z'}}] 2023-10-17T19:06:13.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008968', 1: '00000000000000000000:00000000000000007856', 2: '00000000000000000000:00000000000000004313', 3: '00000000000000000000:00000000000000012063'} 2023-10-17T19:06:13.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008968', 1: '00000000000000000000:00000000000000007856', 2: '00000000000000000000:00000000000000004313', 3: '00000000000000000000:00000000000000012063'} 2023-10-17T19:06:13.297 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:06:13.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-10-17T19:06:13.298 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:13.299 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:13.299 DEBUG:boto:Method: PUT 2023-10-17T19:06:13.299 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:13.300 DEBUG:boto:Data: 2023-10-17T19:06:13.300 DEBUG:boto:Headers: {} 2023-10-17T19:06:13.300 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:13.301 DEBUG:boto:Port: 8000 2023-10-17T19:06:13.301 DEBUG:boto:Params: {} 2023-10-17T19:06:13.302 DEBUG:boto:Token: None 2023-10-17T19:06:13.302 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:06:13 GMT /hbmuog-61/ 2023-10-17T19:06:13.303 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O7kVf+YWHHuR+5tiifffjyRylZ8= 2023-10-17T19:06:13.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:13 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O7kVf+YWHHuR+5tiifffjyRylZ8=', 'Content-Length': '0'} 2023-10-17T19:06:13.318 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bb4ec90bf5aa0ec9-00652edb25-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:13.318 DEBUG:rgw_multi.tests:created bucket=hbmuog-61 2023-10-17T19:06:13.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.320 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:13.369 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:13.369 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:13.400 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:06:13.414 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:06:13.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.414 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569573.316509_776.1", 2023-10-17T19:06:13.414 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:13.316509Z" 2023-10-17T19:06:13.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.415 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:06:13.415 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:06:13.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:06:13.416 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:06:13.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.417 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:06:13.417 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:06:13.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.418 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:06:13.419 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697569573.316509_776.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:06:13.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:06:13.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:06:13.421 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:06:13.474 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:13.474 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:13.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:13.505+0000 7fe028db7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:06:13.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:13.506+0000 7fe028db7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:06:13.520 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:06:13.521 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:06:13.521 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:06:13.521 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:06:13.521 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:06:13.522 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:06:13.522 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:06:13.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.522 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:06:13.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.523 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:06:13.523 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.523 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:06:13.523 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569573.316509_776.1", 2023-10-17T19:06:13.524 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.524 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:06:13.524 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.524 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:13.316509Z", 2023-10-17T19:06:13.524 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:06:13.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.525 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:06:13.525 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.526 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:06:13.526 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:06:13.526 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.526 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.527 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.527 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:06:13.527 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:06:13.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.528 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:06:13.528 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.528 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:06:13.529 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:06:13.529 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.529 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.529 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.530 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:06:13.530 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:06:13.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:06:13.531 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:06:13.531 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:06:13.531 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:06:13.532 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:06:13.532 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:06:13.532 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:06:13.532 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:06:13.533 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:06:13.533 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:06:13.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:06:13.533 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:06:13.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.534 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:06:13.534 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:06:13.534 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:06:13.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:06:13.535 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:06:13Z" 2023-10-17T19:06:13.535 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:06:13.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569573.316509_776.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:06:13.316509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:06:13Z" } 2023-10-17T19:06:13.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569573.316509_776.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:06:13.316509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:06:13.537 DEBUG:rgw_multi.tests:log_status={0: '1_1697569573.316509_776.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:06:13.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569573.316509_776.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:06:13.538 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:06:13.539 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T19:06:13.539 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:06:13.540 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:06:13.538+0000 7ff35f1c0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 165076) UID: 0 2023-10-17T19:06:13.540 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:06:13.538+0000 7ff3634297c0 -1 shutting down 2023-10-17T19:06:18.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:06:19.640 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T19:06:19.641 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.641 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.642 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.642 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.643 DEBUG:boto:Data: 2023-10-17T19:06:19.643 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.644 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.644 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.644 DEBUG:boto:Params: {} 2023-10-17T19:06:19.645 DEBUG:boto:Token: None 2023-10-17T19:06:19.645 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.646 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fcbbeab93779465-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.650 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:06:19.650 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:06:19.651 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.651 DEBUG:boto:Path: /hbmuog-61/obj11 2023-10-17T19:06:19.652 DEBUG:boto:Data: 2023-10-17T19:06:19.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.652 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.653 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.653 DEBUG:boto:Params: {} 2023-10-17T19:06:19.654 DEBUG:boto:Token: None 2023-10-17T19:06:19.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj11 2023-10-17T19:06:19.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V2jZZHx1G293vVOH2n3Eqtv+wZE= 2023-10-17T19:06:19.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V2jZZHx1G293vVOH2n3Eqtv+wZE='} 2023-10-17T19:06:19.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057a3ebe3b7193c66-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.664 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.664 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.664 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.665 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.665 DEBUG:boto:Data: 2023-10-17T19:06:19.666 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.666 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.667 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.667 DEBUG:boto:Params: {} 2023-10-17T19:06:19.667 DEBUG:boto:Token: None 2023-10-17T19:06:19.668 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.668 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016147b015cddab3b-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.672 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:06:19.672 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:06:19.673 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.673 DEBUG:boto:Path: /hbmuog-61/obj22 2023-10-17T19:06:19.673 DEBUG:boto:Data: 2023-10-17T19:06:19.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.675 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.675 DEBUG:boto:Params: {} 2023-10-17T19:06:19.675 DEBUG:boto:Token: None 2023-10-17T19:06:19.676 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj22 2023-10-17T19:06:19.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pKHuJOuKZ9z1hqWO0T9BkVTsLw4= 2023-10-17T19:06:19.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pKHuJOuKZ9z1hqWO0T9BkVTsLw4='} 2023-10-17T19:06:19.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a07224f8c9ec8b18-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.723 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.724 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.724 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.724 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.725 DEBUG:boto:Data: 2023-10-17T19:06:19.725 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.726 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.726 DEBUG:boto:Params: {} 2023-10-17T19:06:19.727 DEBUG:boto:Token: None 2023-10-17T19:06:19.727 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a50b171be20a76ed-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.732 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:06:19.732 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:06:19.732 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.733 DEBUG:boto:Path: /hbmuog-61/obj33 2023-10-17T19:06:19.733 DEBUG:boto:Data: 2023-10-17T19:06:19.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.734 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.734 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.737 DEBUG:boto:Params: {} 2023-10-17T19:06:19.738 DEBUG:boto:Token: None 2023-10-17T19:06:19.738 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj33 2023-10-17T19:06:19.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QlmMu34CN06xzXD52UPFyBDPNeY= 2023-10-17T19:06:19.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QlmMu34CN06xzXD52UPFyBDPNeY='} 2023-10-17T19:06:19.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afda17c7ef38f961-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.747 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.747 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.748 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.748 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.749 DEBUG:boto:Data: 2023-10-17T19:06:19.749 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.750 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.750 DEBUG:boto:Params: {} 2023-10-17T19:06:19.751 DEBUG:boto:Token: None 2023-10-17T19:06:19.751 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4846d37922e529b-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.756 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:06:19.756 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:06:19.757 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.757 DEBUG:boto:Path: /hbmuog-61/obj44 2023-10-17T19:06:19.757 DEBUG:boto:Data: 2023-10-17T19:06:19.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.759 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.759 DEBUG:boto:Params: {} 2023-10-17T19:06:19.760 DEBUG:boto:Token: None 2023-10-17T19:06:19.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj44 2023-10-17T19:06:19.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BMoKbr3njCRQdBo+lffPLcQ5mk4= 2023-10-17T19:06:19.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BMoKbr3njCRQdBo+lffPLcQ5mk4='} 2023-10-17T19:06:19.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab026ae82b6a5ba9-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.808 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.808 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.809 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.809 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.810 DEBUG:boto:Data: 2023-10-17T19:06:19.810 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.811 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.811 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.812 DEBUG:boto:Params: {} 2023-10-17T19:06:19.812 DEBUG:boto:Token: None 2023-10-17T19:06:19.813 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.813 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5daacff17bfda25-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.817 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:06:19.817 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:06:19.818 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.818 DEBUG:boto:Path: /hbmuog-61/obj55 2023-10-17T19:06:19.819 DEBUG:boto:Data: 2023-10-17T19:06:19.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.820 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.820 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.820 DEBUG:boto:Params: {} 2023-10-17T19:06:19.821 DEBUG:boto:Token: None 2023-10-17T19:06:19.821 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj55 2023-10-17T19:06:19.822 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Gubo+8B4NvGPZL/z3Ycanh3HJ4k= 2023-10-17T19:06:19.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Gubo+8B4NvGPZL/z3Ycanh3HJ4k='} 2023-10-17T19:06:19.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025563da46c79a8a5-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:19.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:19.828 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:19.888 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:19.889 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:19.934 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:19.934 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:19.938 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 5 2023-10-17T19:06:19.974 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.975 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.975 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.976 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.976 DEBUG:boto:Data: 2023-10-17T19:06:19.977 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.977 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.978 DEBUG:boto:Params: {} 2023-10-17T19:06:19.978 DEBUG:boto:Token: None 2023-10-17T19:06:19.979 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.979 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:19.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001654a4634496c7d1-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.983 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:06:19.983 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:06:19.984 DEBUG:boto:Method: PUT 2023-10-17T19:06:19.984 DEBUG:boto:Path: /hbmuog-61/obj17 2023-10-17T19:06:19.985 DEBUG:boto:Data: 2023-10-17T19:06:19.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:19.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.986 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.987 DEBUG:boto:Params: {} 2023-10-17T19:06:19.987 DEBUG:boto:Token: None 2023-10-17T19:06:19.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/obj17 2023-10-17T19:06:19.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pNI1NMS+2FJsPNMgchqngVLlRJ8= 2023-10-17T19:06:19.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pNI1NMS+2FJsPNMgchqngVLlRJ8='} 2023-10-17T19:06:19.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003722c24539767471-00652edb2b-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:19.995 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:19.995 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:19.995 DEBUG:boto:Method: HEAD 2023-10-17T19:06:19.996 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:19.996 DEBUG:boto:Data: 2023-10-17T19:06:19.996 DEBUG:boto:Headers: {} 2023-10-17T19:06:19.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:19.997 DEBUG:boto:Port: 8000 2023-10-17T19:06:19.997 DEBUG:boto:Params: {} 2023-10-17T19:06:19.998 DEBUG:boto:Token: None 2023-10-17T19:06:19.998 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:19 GMT /hbmuog-61/ 2023-10-17T19:06:19.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY= 2023-10-17T19:06:19.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tUQ4wEAi9qFoFs5OKZk/4VYPNFY=', 'Content-Length': '0'} 2023-10-17T19:06:20.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bed66c5e63727fc1-00652edb2b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.003 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:06:20.003 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:06:20.003 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.004 DEBUG:boto:Path: /hbmuog-61/obj34 2023-10-17T19:06:20.004 DEBUG:boto:Data: 2023-10-17T19:06:20.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.005 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.006 DEBUG:boto:Params: {} 2023-10-17T19:06:20.006 DEBUG:boto:Token: None 2023-10-17T19:06:20.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj34 2023-10-17T19:06:20.007 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BoaRbEktJ9235e5uLiKOP1MQuno= 2023-10-17T19:06:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BoaRbEktJ9235e5uLiKOP1MQuno='} 2023-10-17T19:06:20.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4f798ca22c4e2ea-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.052 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.052 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.052 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.053 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.053 DEBUG:boto:Data: 2023-10-17T19:06:20.054 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.054 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.055 DEBUG:boto:Params: {} 2023-10-17T19:06:20.055 DEBUG:boto:Token: None 2023-10-17T19:06:20.056 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a71556b8378ffef-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.060 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:06:20.060 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:06:20.061 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.061 DEBUG:boto:Path: /hbmuog-61/obj51 2023-10-17T19:06:20.062 DEBUG:boto:Data: 2023-10-17T19:06:20.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.062 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.063 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.063 DEBUG:boto:Params: {} 2023-10-17T19:06:20.064 DEBUG:boto:Token: None 2023-10-17T19:06:20.064 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj51 2023-10-17T19:06:20.065 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:crgE5tI0XeiZLTwzUT1zKt5xs8Y= 2023-10-17T19:06:20.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:crgE5tI0XeiZLTwzUT1zKt5xs8Y='} 2023-10-17T19:06:20.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dc4debe18e4a139-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.071 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.071 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.072 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.072 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.073 DEBUG:boto:Data: 2023-10-17T19:06:20.073 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.074 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.074 DEBUG:boto:Params: {} 2023-10-17T19:06:20.075 DEBUG:boto:Token: None 2023-10-17T19:06:20.075 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3509abcd47edae0-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.079 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:06:20.080 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:06:20.080 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.081 DEBUG:boto:Path: /hbmuog-61/obj68 2023-10-17T19:06:20.081 DEBUG:boto:Data: 2023-10-17T19:06:20.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.083 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.083 DEBUG:boto:Params: {} 2023-10-17T19:06:20.084 DEBUG:boto:Token: None 2023-10-17T19:06:20.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj68 2023-10-17T19:06:20.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CgoGRk6J88CAA+648HZ+YvBc9u4= 2023-10-17T19:06:20.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CgoGRk6J88CAA+648HZ+YvBc9u4='} 2023-10-17T19:06:20.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000681439cdef6ed28c-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.131 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.132 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.132 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.132 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.133 DEBUG:boto:Data: 2023-10-17T19:06:20.133 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.134 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.134 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.134 DEBUG:boto:Params: {} 2023-10-17T19:06:20.135 DEBUG:boto:Token: None 2023-10-17T19:06:20.135 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bd31e66d3974607-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.139 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:06:20.140 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:06:20.140 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.141 DEBUG:boto:Path: /hbmuog-61/obj85 2023-10-17T19:06:20.141 DEBUG:boto:Data: 2023-10-17T19:06:20.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.142 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.143 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.143 DEBUG:boto:Params: {} 2023-10-17T19:06:20.143 DEBUG:boto:Token: None 2023-10-17T19:06:20.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj85 2023-10-17T19:06:20.144 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XGWhn3vyT+5AIKd1t+PCjZ7+GFM= 2023-10-17T19:06:20.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XGWhn3vyT+5AIKd1t+PCjZ7+GFM='} 2023-10-17T19:06:20.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6f2ec35911a6066-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.152 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:20.207 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:20.208 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:20.252 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:20.252 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:20.258 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 10 2023-10-17T19:06:20.300 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.300 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.301 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.301 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.302 DEBUG:boto:Data: 2023-10-17T19:06:20.302 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.303 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.303 DEBUG:boto:Params: {} 2023-10-17T19:06:20.304 DEBUG:boto:Token: None 2023-10-17T19:06:20.304 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f651b8a1051d9055-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.309 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:06:20.310 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:06:20.310 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.311 DEBUG:boto:Path: /hbmuog-61/obj19 2023-10-17T19:06:20.311 DEBUG:boto:Data: 2023-10-17T19:06:20.312 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.312 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.312 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.313 DEBUG:boto:Params: {} 2023-10-17T19:06:20.313 DEBUG:boto:Token: None 2023-10-17T19:06:20.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj19 2023-10-17T19:06:20.314 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yhDVfO+SWd3Dlr4yH3mjVGFeZLo= 2023-10-17T19:06:20.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yhDVfO+SWd3Dlr4yH3mjVGFeZLo='} 2023-10-17T19:06:20.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e1479598c063da6-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.320 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.321 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.321 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.322 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.322 DEBUG:boto:Data: 2023-10-17T19:06:20.323 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.323 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.324 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.324 DEBUG:boto:Params: {} 2023-10-17T19:06:20.324 DEBUG:boto:Token: None 2023-10-17T19:06:20.325 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.325 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de84a428326e5a4-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.329 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:06:20.329 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:06:20.330 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.330 DEBUG:boto:Path: /hbmuog-61/obj38 2023-10-17T19:06:20.331 DEBUG:boto:Data: 2023-10-17T19:06:20.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.331 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.332 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.332 DEBUG:boto:Params: {} 2023-10-17T19:06:20.333 DEBUG:boto:Token: None 2023-10-17T19:06:20.333 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj38 2023-10-17T19:06:20.334 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qk7i4TfrTQ9hTCwbGjYWs5Vse00= 2023-10-17T19:06:20.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qk7i4TfrTQ9hTCwbGjYWs5Vse00='} 2023-10-17T19:06:20.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093335c7ad52e0f16-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.379 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.380 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.380 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.381 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.381 DEBUG:boto:Data: 2023-10-17T19:06:20.382 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.382 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.382 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.383 DEBUG:boto:Params: {} 2023-10-17T19:06:20.383 DEBUG:boto:Token: None 2023-10-17T19:06:20.384 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.384 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001954379c6acd845e-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.388 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:06:20.389 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:06:20.389 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.389 DEBUG:boto:Path: /hbmuog-61/obj57 2023-10-17T19:06:20.390 DEBUG:boto:Data: 2023-10-17T19:06:20.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.391 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.391 DEBUG:boto:Params: {} 2023-10-17T19:06:20.392 DEBUG:boto:Token: None 2023-10-17T19:06:20.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj57 2023-10-17T19:06:20.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I/feWV7ObDGsvNJHwhUyj8ppFew= 2023-10-17T19:06:20.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I/feWV7ObDGsvNJHwhUyj8ppFew='} 2023-10-17T19:06:20.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000417960d009a1b5e1-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.397 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.398 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.398 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.398 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.399 DEBUG:boto:Data: 2023-10-17T19:06:20.399 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.400 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.400 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.400 DEBUG:boto:Params: {} 2023-10-17T19:06:20.401 DEBUG:boto:Token: None 2023-10-17T19:06:20.401 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b346b82902b285d-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.406 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:06:20.406 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:06:20.407 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.407 DEBUG:boto:Path: /hbmuog-61/obj76 2023-10-17T19:06:20.407 DEBUG:boto:Data: 2023-10-17T19:06:20.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.409 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.409 DEBUG:boto:Params: {} 2023-10-17T19:06:20.409 DEBUG:boto:Token: None 2023-10-17T19:06:20.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj76 2023-10-17T19:06:20.410 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lnivzqx3li+qSemyR+84mtGVZ3Q= 2023-10-17T19:06:20.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnivzqx3li+qSemyR+84mtGVZ3Q='} 2023-10-17T19:06:20.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004bee72e0f7ba3d0-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.456 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.456 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.456 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.457 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.457 DEBUG:boto:Data: 2023-10-17T19:06:20.458 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.458 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.459 DEBUG:boto:Params: {} 2023-10-17T19:06:20.459 DEBUG:boto:Token: None 2023-10-17T19:06:20.460 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073fd6b1a35114461-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.464 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:06:20.464 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:06:20.465 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.465 DEBUG:boto:Path: /hbmuog-61/obj95 2023-10-17T19:06:20.466 DEBUG:boto:Data: 2023-10-17T19:06:20.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.466 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.467 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.467 DEBUG:boto:Params: {} 2023-10-17T19:06:20.468 DEBUG:boto:Token: None 2023-10-17T19:06:20.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj95 2023-10-17T19:06:20.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pmg3vhz88K0UEYkSlGJ5tngGR64= 2023-10-17T19:06:20.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pmg3vhz88K0UEYkSlGJ5tngGR64='} 2023-10-17T19:06:20.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f1a6d39638956db-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.474 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:20.577 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:20.577 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:20.584 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 15 2023-10-17T19:06:20.629 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.630 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.630 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.630 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.631 DEBUG:boto:Data: 2023-10-17T19:06:20.631 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.632 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.632 DEBUG:boto:Params: {} 2023-10-17T19:06:20.633 DEBUG:boto:Token: None 2023-10-17T19:06:20.633 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a9773548fcdeda0-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.638 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:06:20.638 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:06:20.639 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.639 DEBUG:boto:Path: /hbmuog-61/obj23 2023-10-17T19:06:20.639 DEBUG:boto:Data: 2023-10-17T19:06:20.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.640 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.641 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.641 DEBUG:boto:Params: {} 2023-10-17T19:06:20.641 DEBUG:boto:Token: None 2023-10-17T19:06:20.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj23 2023-10-17T19:06:20.642 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QdQ9W+wlX2NN/T7S7DA5n2cEYnk= 2023-10-17T19:06:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QdQ9W+wlX2NN/T7S7DA5n2cEYnk='} 2023-10-17T19:06:20.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f50826307fe9a8fe-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.648 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.649 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.649 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.650 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.650 DEBUG:boto:Data: 2023-10-17T19:06:20.650 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.651 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.651 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.652 DEBUG:boto:Params: {} 2023-10-17T19:06:20.652 DEBUG:boto:Token: None 2023-10-17T19:06:20.653 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.653 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001950aa58187ca10d-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.657 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:06:20.657 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:06:20.658 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.658 DEBUG:boto:Path: /hbmuog-61/obj46 2023-10-17T19:06:20.658 DEBUG:boto:Data: 2023-10-17T19:06:20.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.659 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.660 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.660 DEBUG:boto:Params: {} 2023-10-17T19:06:20.661 DEBUG:boto:Token: None 2023-10-17T19:06:20.661 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj46 2023-10-17T19:06:20.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wqQhNsfa4iCv54wiW62/oz+Uigw= 2023-10-17T19:06:20.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wqQhNsfa4iCv54wiW62/oz+Uigw='} 2023-10-17T19:06:20.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002da8f62d28d0e223-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.712 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.713 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.713 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.714 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.714 DEBUG:boto:Data: 2023-10-17T19:06:20.714 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.715 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.716 DEBUG:boto:Params: {} 2023-10-17T19:06:20.716 DEBUG:boto:Token: None 2023-10-17T19:06:20.717 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a6e2aac83affeb7-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.721 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:06:20.721 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:06:20.722 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.722 DEBUG:boto:Path: /hbmuog-61/obj69 2023-10-17T19:06:20.723 DEBUG:boto:Data: 2023-10-17T19:06:20.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.724 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.724 DEBUG:boto:Params: {} 2023-10-17T19:06:20.725 DEBUG:boto:Token: None 2023-10-17T19:06:20.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj69 2023-10-17T19:06:20.726 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GK9kNRS3VhwV4AUtfjlSu1qQfcY= 2023-10-17T19:06:20.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GK9kNRS3VhwV4AUtfjlSu1qQfcY='} 2023-10-17T19:06:20.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005696c9b630d011ab-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.732 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.733 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.733 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.733 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.734 DEBUG:boto:Data: 2023-10-17T19:06:20.734 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.735 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.735 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.736 DEBUG:boto:Params: {} 2023-10-17T19:06:20.736 DEBUG:boto:Token: None 2023-10-17T19:06:20.736 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.737 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b42a7d81528451ed-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.741 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:06:20.741 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:06:20.742 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.742 DEBUG:boto:Path: /hbmuog-61/obj92 2023-10-17T19:06:20.743 DEBUG:boto:Data: 2023-10-17T19:06:20.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.744 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.744 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.744 DEBUG:boto:Params: {} 2023-10-17T19:06:20.745 DEBUG:boto:Token: None 2023-10-17T19:06:20.745 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj92 2023-10-17T19:06:20.746 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tGmK3W+2RkWPe9WV5mvNXzSDHo4= 2023-10-17T19:06:20.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tGmK3W+2RkWPe9WV5mvNXzSDHo4='} 2023-10-17T19:06:20.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000247af751781adac6-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.795 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.796 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.796 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.796 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.797 DEBUG:boto:Data: 2023-10-17T19:06:20.797 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.798 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.798 DEBUG:boto:Params: {} 2023-10-17T19:06:20.799 DEBUG:boto:Token: None 2023-10-17T19:06:20.799 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3b77a62f6ad62dc-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.804 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:06:20.804 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:06:20.805 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.805 DEBUG:boto:Path: /hbmuog-61/obj115 2023-10-17T19:06:20.805 DEBUG:boto:Data: 2023-10-17T19:06:20.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.806 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.807 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.807 DEBUG:boto:Params: {} 2023-10-17T19:06:20.808 DEBUG:boto:Token: None 2023-10-17T19:06:20.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj115 2023-10-17T19:06:20.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:R0Xz571vi/lPeapeUWtyfu5MRp8= 2023-10-17T19:06:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:R0Xz571vi/lPeapeUWtyfu5MRp8='} 2023-10-17T19:06:20.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064382b3d365e65ac-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:20.816 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:20.869 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:20.870 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:20.915 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:20.916 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:20.924 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 20 2023-10-17T19:06:20.974 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:20.975 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:20.975 DEBUG:boto:Method: HEAD 2023-10-17T19:06:20.976 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:20.976 DEBUG:boto:Data: 2023-10-17T19:06:20.976 DEBUG:boto:Headers: {} 2023-10-17T19:06:20.977 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.977 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.977 DEBUG:boto:Params: {} 2023-10-17T19:06:20.978 DEBUG:boto:Token: None 2023-10-17T19:06:20.978 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/ 2023-10-17T19:06:20.979 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs= 2023-10-17T19:06:20.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GOCtvjkdpFA7TbxhqLaHsNLnkqs=', 'Content-Length': '0'} 2023-10-17T19:06:20.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f1da8abebe49059-00652edb2c-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:20.983 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:06:20.984 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:06:20.984 DEBUG:boto:Method: PUT 2023-10-17T19:06:20.984 DEBUG:boto:Path: /hbmuog-61/obj29 2023-10-17T19:06:20.985 DEBUG:boto:Data: 2023-10-17T19:06:20.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:20.986 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:20.986 DEBUG:boto:Port: 8000 2023-10-17T19:06:20.986 DEBUG:boto:Params: {} 2023-10-17T19:06:20.987 DEBUG:boto:Token: None 2023-10-17T19:06:20.987 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:20 GMT /hbmuog-61/obj29 2023-10-17T19:06:20.988 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A4wFxPrnsFtvg+yOuwPVAGVxzSE= 2023-10-17T19:06:20.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A4wFxPrnsFtvg+yOuwPVAGVxzSE='} 2023-10-17T19:06:21.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed5e6bb1ee86f0b-00652edb2c-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.031 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.031 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.032 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.032 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.032 DEBUG:boto:Data: 2023-10-17T19:06:21.033 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.033 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.033 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.034 DEBUG:boto:Params: {} 2023-10-17T19:06:21.034 DEBUG:boto:Token: None 2023-10-17T19:06:21.035 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.035 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f34e239e50d43305-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.039 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:06:21.040 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:06:21.040 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.040 DEBUG:boto:Path: /hbmuog-61/obj58 2023-10-17T19:06:21.041 DEBUG:boto:Data: 2023-10-17T19:06:21.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.042 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.043 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.043 DEBUG:boto:Params: {} 2023-10-17T19:06:21.044 DEBUG:boto:Token: None 2023-10-17T19:06:21.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj58 2023-10-17T19:06:21.045 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y2616UjuK4X1wC8upJLsVH5eR3o= 2023-10-17T19:06:21.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y2616UjuK4X1wC8upJLsVH5eR3o='} 2023-10-17T19:06:21.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002622578698b7caca-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.050 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.051 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.051 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.051 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.052 DEBUG:boto:Data: 2023-10-17T19:06:21.052 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.052 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.053 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.053 DEBUG:boto:Params: {} 2023-10-17T19:06:21.054 DEBUG:boto:Token: None 2023-10-17T19:06:21.054 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.054 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1ec3f4372ace9c4-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.059 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:06:21.059 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:06:21.059 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.060 DEBUG:boto:Path: /hbmuog-61/obj87 2023-10-17T19:06:21.060 DEBUG:boto:Data: 2023-10-17T19:06:21.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.061 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.061 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.062 DEBUG:boto:Params: {} 2023-10-17T19:06:21.062 DEBUG:boto:Token: None 2023-10-17T19:06:21.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj87 2023-10-17T19:06:21.063 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+O+rwaiCyxgm4N7viAGBSOWLQsg= 2023-10-17T19:06:21.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+O+rwaiCyxgm4N7viAGBSOWLQsg='} 2023-10-17T19:06:21.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ede5e26b6779ac6-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.107 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.107 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.108 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.108 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.108 DEBUG:boto:Data: 2023-10-17T19:06:21.109 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.110 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.110 DEBUG:boto:Params: {} 2023-10-17T19:06:21.111 DEBUG:boto:Token: None 2023-10-17T19:06:21.111 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.112 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006712ef65fbfc1774-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.116 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:06:21.116 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:06:21.117 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.117 DEBUG:boto:Path: /hbmuog-61/obj116 2023-10-17T19:06:21.117 DEBUG:boto:Data: 2023-10-17T19:06:21.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.118 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.119 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.119 DEBUG:boto:Params: {} 2023-10-17T19:06:21.120 DEBUG:boto:Token: None 2023-10-17T19:06:21.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj116 2023-10-17T19:06:21.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bNuupCXVXIJXu+nSOFrAqnbIqFY= 2023-10-17T19:06:21.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bNuupCXVXIJXu+nSOFrAqnbIqFY='} 2023-10-17T19:06:21.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cebabed551b90a6-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.126 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.127 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.127 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.128 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.128 DEBUG:boto:Data: 2023-10-17T19:06:21.128 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.129 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.129 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.130 DEBUG:boto:Params: {} 2023-10-17T19:06:21.130 DEBUG:boto:Token: None 2023-10-17T19:06:21.130 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011fcfad2a77a8c72-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.135 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:06:21.135 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:06:21.136 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.136 DEBUG:boto:Path: /hbmuog-61/obj145 2023-10-17T19:06:21.136 DEBUG:boto:Data: 2023-10-17T19:06:21.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.138 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.138 DEBUG:boto:Params: {} 2023-10-17T19:06:21.138 DEBUG:boto:Token: None 2023-10-17T19:06:21.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj145 2023-10-17T19:06:21.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rYdXu2fB4rf9DMwoNMC5KI8ychM= 2023-10-17T19:06:21.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rYdXu2fB4rf9DMwoNMC5KI8ychM='} 2023-10-17T19:06:21.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55ee1f7db11aedb-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.188 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:21.242 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:21.242 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:21.297 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:21.297 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:21.308 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 25 2023-10-17T19:06:21.365 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.365 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.366 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.366 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.367 DEBUG:boto:Data: 2023-10-17T19:06:21.367 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.367 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.369 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.370 DEBUG:boto:Params: {} 2023-10-17T19:06:21.370 DEBUG:boto:Token: None 2023-10-17T19:06:21.371 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.371 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24506b7da66dd3c-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.376 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:06:21.376 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:06:21.376 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.377 DEBUG:boto:Path: /hbmuog-61/obj31 2023-10-17T19:06:21.377 DEBUG:boto:Data: 2023-10-17T19:06:21.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.378 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.378 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.379 DEBUG:boto:Params: {} 2023-10-17T19:06:21.379 DEBUG:boto:Token: None 2023-10-17T19:06:21.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj31 2023-10-17T19:06:21.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fhmyowNL/cTUF6S8sBUA1/vOxDg= 2023-10-17T19:06:21.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fhmyowNL/cTUF6S8sBUA1/vOxDg='} 2023-10-17T19:06:21.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054686f90d748d017-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.386 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.386 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.387 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.387 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.388 DEBUG:boto:Data: 2023-10-17T19:06:21.388 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.388 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.389 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.389 DEBUG:boto:Params: {} 2023-10-17T19:06:21.389 DEBUG:boto:Token: None 2023-10-17T19:06:21.390 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.390 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000273b07be8c07641a-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.395 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:06:21.395 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:06:21.395 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.396 DEBUG:boto:Path: /hbmuog-61/obj62 2023-10-17T19:06:21.396 DEBUG:boto:Data: 2023-10-17T19:06:21.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.397 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.397 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.398 DEBUG:boto:Params: {} 2023-10-17T19:06:21.398 DEBUG:boto:Token: None 2023-10-17T19:06:21.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj62 2023-10-17T19:06:21.399 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DRJEG33rYi+DKZTxlDQZpgjfH28= 2023-10-17T19:06:21.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DRJEG33rYi+DKZTxlDQZpgjfH28='} 2023-10-17T19:06:21.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003411fd6d41e302dc-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.405 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.405 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.405 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.406 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.406 DEBUG:boto:Data: 2023-10-17T19:06:21.407 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.407 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.408 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.408 DEBUG:boto:Params: {} 2023-10-17T19:06:21.408 DEBUG:boto:Token: None 2023-10-17T19:06:21.409 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.409 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2605c063ef56612-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.413 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:06:21.414 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:06:21.414 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.415 DEBUG:boto:Path: /hbmuog-61/obj93 2023-10-17T19:06:21.415 DEBUG:boto:Data: 2023-10-17T19:06:21.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.416 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.416 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.416 DEBUG:boto:Params: {} 2023-10-17T19:06:21.417 DEBUG:boto:Token: None 2023-10-17T19:06:21.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj93 2023-10-17T19:06:21.418 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OQFjahNxO4eEJyB5NLCJqr8HLvs= 2023-10-17T19:06:21.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OQFjahNxO4eEJyB5NLCJqr8HLvs='} 2023-10-17T19:06:21.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2ac0ec3f14df93d-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.463 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.463 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.464 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.464 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.464 DEBUG:boto:Data: 2023-10-17T19:06:21.465 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.465 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.466 DEBUG:boto:Params: {} 2023-10-17T19:06:21.466 DEBUG:boto:Token: None 2023-10-17T19:06:21.467 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.467 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd6adbba6f56c2e4-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.472 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:06:21.472 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:06:21.472 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.473 DEBUG:boto:Path: /hbmuog-61/obj124 2023-10-17T19:06:21.473 DEBUG:boto:Data: 2023-10-17T19:06:21.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.474 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.474 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.475 DEBUG:boto:Params: {} 2023-10-17T19:06:21.475 DEBUG:boto:Token: None 2023-10-17T19:06:21.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj124 2023-10-17T19:06:21.476 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E+yzW/U7KgNC24fNiCZCeqizreQ= 2023-10-17T19:06:21.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E+yzW/U7KgNC24fNiCZCeqizreQ='} 2023-10-17T19:06:21.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f28f64677dd7c75b-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.482 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.482 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.482 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.483 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.483 DEBUG:boto:Data: 2023-10-17T19:06:21.484 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.484 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.484 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.485 DEBUG:boto:Params: {} 2023-10-17T19:06:21.485 DEBUG:boto:Token: None 2023-10-17T19:06:21.485 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.486 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9810f180a0b6696-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.490 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:06:21.490 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:06:21.491 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.491 DEBUG:boto:Path: /hbmuog-61/obj155 2023-10-17T19:06:21.492 DEBUG:boto:Data: 2023-10-17T19:06:21.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.493 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.493 DEBUG:boto:Params: {} 2023-10-17T19:06:21.493 DEBUG:boto:Token: None 2023-10-17T19:06:21.494 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj155 2023-10-17T19:06:21.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ExRAKNy+q371Q7udffbW3fMqF1E= 2023-10-17T19:06:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ExRAKNy+q371Q7udffbW3fMqF1E='} 2023-10-17T19:06:21.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025eda3af9174c96e-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.540 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:21.593 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:21.594 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:21.646 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:06:21.646 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-61 2023-10-17T19:06:21.658 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 30 2023-10-17T19:06:21.719 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.720 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.720 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.721 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.721 DEBUG:boto:Data: 2023-10-17T19:06:21.722 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.723 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.724 DEBUG:boto:Params: {} 2023-10-17T19:06:21.724 DEBUG:boto:Token: None 2023-10-17T19:06:21.725 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f66625c7fee012b0-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.730 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:06:21.730 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:06:21.731 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.731 DEBUG:boto:Path: /hbmuog-61/obj37 2023-10-17T19:06:21.732 DEBUG:boto:Data: 2023-10-17T19:06:21.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.733 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.733 DEBUG:boto:Params: {} 2023-10-17T19:06:21.734 DEBUG:boto:Token: None 2023-10-17T19:06:21.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj37 2023-10-17T19:06:21.735 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HSlgxi0Mv9EcsKdp/28j1TNDZ9I= 2023-10-17T19:06:21.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HSlgxi0Mv9EcsKdp/28j1TNDZ9I='} 2023-10-17T19:06:21.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014186abbcdad649b-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.741 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.741 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.741 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.742 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.742 DEBUG:boto:Data: 2023-10-17T19:06:21.743 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.744 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.744 DEBUG:boto:Params: {} 2023-10-17T19:06:21.744 DEBUG:boto:Token: None 2023-10-17T19:06:21.745 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b554441a894f5c5-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.750 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:06:21.750 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:06:21.751 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.751 DEBUG:boto:Path: /hbmuog-61/obj74 2023-10-17T19:06:21.751 DEBUG:boto:Data: 2023-10-17T19:06:21.752 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.752 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.753 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.753 DEBUG:boto:Params: {} 2023-10-17T19:06:21.754 DEBUG:boto:Token: None 2023-10-17T19:06:21.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj74 2023-10-17T19:06:21.755 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:14yQUY5HoXwCnotWpbwusZYttVg= 2023-10-17T19:06:21.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:14yQUY5HoXwCnotWpbwusZYttVg='} 2023-10-17T19:06:21.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044b6c8886574dc63-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.761 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.761 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.762 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.762 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.762 DEBUG:boto:Data: 2023-10-17T19:06:21.763 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.763 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.764 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.764 DEBUG:boto:Params: {} 2023-10-17T19:06:21.764 DEBUG:boto:Token: None 2023-10-17T19:06:21.765 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.765 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008147480458772d4d-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.770 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:06:21.770 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:06:21.771 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.771 DEBUG:boto:Path: /hbmuog-61/obj111 2023-10-17T19:06:21.772 DEBUG:boto:Data: 2023-10-17T19:06:21.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.773 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.773 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.773 DEBUG:boto:Params: {} 2023-10-17T19:06:21.774 DEBUG:boto:Token: None 2023-10-17T19:06:21.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj111 2023-10-17T19:06:21.775 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FLKwQ2RxF7lOI2wN1EzxW/IVf+8= 2023-10-17T19:06:21.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FLKwQ2RxF7lOI2wN1EzxW/IVf+8='} 2023-10-17T19:06:21.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fddf96c6ccd5342-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.783 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.783 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.784 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.784 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.784 DEBUG:boto:Data: 2023-10-17T19:06:21.785 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.786 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.786 DEBUG:boto:Params: {} 2023-10-17T19:06:21.787 DEBUG:boto:Token: None 2023-10-17T19:06:21.787 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000101431967b36ab25-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.793 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:06:21.793 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:06:21.794 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.794 DEBUG:boto:Path: /hbmuog-61/obj148 2023-10-17T19:06:21.794 DEBUG:boto:Data: 2023-10-17T19:06:21.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.795 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.796 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.796 DEBUG:boto:Params: {} 2023-10-17T19:06:21.796 DEBUG:boto:Token: None 2023-10-17T19:06:21.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj148 2023-10-17T19:06:21.797 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L/MO//LLlkr8FuM8teCBVY7k+as= 2023-10-17T19:06:21.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L/MO//LLlkr8FuM8teCBVY7k+as='} 2023-10-17T19:06:21.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d5154ba37df6423-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.843 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:06:21.844 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:06:21.844 DEBUG:boto:Method: HEAD 2023-10-17T19:06:21.844 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:06:21.845 DEBUG:boto:Data: 2023-10-17T19:06:21.845 DEBUG:boto:Headers: {} 2023-10-17T19:06:21.846 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.846 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.847 DEBUG:boto:Params: {} 2023-10-17T19:06:21.847 DEBUG:boto:Token: None 2023-10-17T19:06:21.847 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/ 2023-10-17T19:06:21.848 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o= 2023-10-17T19:06:21.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nPQuRgE/B2qbNu99ghAfa0fTk2o=', 'Content-Length': '0'} 2023-10-17T19:06:21.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029a9fe03081fa3e6-00652edb2d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.853 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:06:21.853 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:06:21.854 DEBUG:boto:Method: PUT 2023-10-17T19:06:21.855 DEBUG:boto:Path: /hbmuog-61/obj185 2023-10-17T19:06:21.856 DEBUG:boto:Data: 2023-10-17T19:06:21.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:06:21.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:06:21.858 DEBUG:boto:Port: 8000 2023-10-17T19:06:21.859 DEBUG:boto:Params: {} 2023-10-17T19:06:21.859 DEBUG:boto:Token: None 2023-10-17T19:06:21.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:06:21 GMT /hbmuog-61/obj185 2023-10-17T19:06:21.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rh2Za537Zo84ygzWMBWFpA6UQpw= 2023-10-17T19:06:21.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:06:21 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rh2Za537Zo84ygzWMBWFpA6UQpw='} 2023-10-17T19:06:21.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc4e1543a377641f-00652edb2d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:06:21.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:06:21.869 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:06:21.920 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:21.921 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:21.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:21.951+0000 7f0608124780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-17T19:06:21.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:21.952+0000 7f0608124780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-17T19:06:21.975 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T19:06:21.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T19:06:21.978 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T19:06:21.981 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T19:06:22.004 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:06:22.005 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:06:22.005 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T19:06:23.007 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:06:23.031 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:06:23.031 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:06:23.032 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T19:06:23.033 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T19:06:23.033 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-61 2023-10-17T19:06:23.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:06:23.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:06:23.035 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:06:23.132 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:06:23.133 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:06:23.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:23.168+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:24.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T19:06:28.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:28.168+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:33.170+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:38.170+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:43.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:43.170+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:48.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:48.170+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:53.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:53.171+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:06:58.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:06:58.172+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:03.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:03.173+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:08.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:08.174+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:13.175 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:13.175+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:18.175 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:18.175+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:23.175+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:28.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:28.175+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:33.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:33.176+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:38.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:38.177+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:43.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:43.179+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:48.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:48.180+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:53.180+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:07:58.181 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:07:58.180+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:03.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:03.180+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:08.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:08.181+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:13.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:13.182+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:18.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:18.182+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:23.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:23.184+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:28.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:28.184+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:33.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:33.184+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:38.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:38.185+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:43.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:43.185+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:48.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:48.186+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:53.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:53.187+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:08:58.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:08:58.189+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:03.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:03.189+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:08.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:08.189+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:13.189+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:18.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:18.190+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:23.191+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:28.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:28.192+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:33.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:33.193+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:38.193+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:43.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:43.193+0000 7fb8a0c4d780 1 waiting to reach incremental sync.. 2023-10-17T19:09:48.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:48.197+0000 7fb8a0c4d780 1 bucket sync caught up with source: 2023-10-17T19:09:48.199 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , 00000000002.92.6, , , , , , 00000000002.61.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.54.6, , , , 00000000001.84.6] 2023-10-17T19:09:48.199 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , 00000000002.92.6, , , , , , 00000000002.61.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.54.6, , , , 00000000001.84.6] 2023-10-17T19:09:48.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:48.197+0000 7fb8a0c4d780 0 bucket checkpoint complete 2023-10-17T19:09:48.217 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-61 2023-10-17T19:09:48.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:09:48.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:09:48.219 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:09:48.283 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:48.284 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:48.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:48.322+0000 7fd8761a9780 1 bucket sync caught up with source: 2023-10-17T19:09:48.324 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, 00000000003.303.6, 00000000002.189.6, 00000000002.362.6, 00000000005.233.6, 00000000004.481.6, 00000000001.280.6, 00000000001.201.6, 00000000004.235.6, 00000000007.294.6, 00000000006.480.6] 2023-10-17T19:09:48.325 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, 00000000003.303.6, 00000000002.189.6, 00000000002.362.6, 00000000005.233.6, 00000000004.481.6, 00000000001.280.6, 00000000001.201.6, 00000000004.235.6, 00000000007.294.6, 00000000006.480.6] 2023-10-17T19:09:48.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:48.322+0000 7fd8761a9780 0 bucket checkpoint complete 2023-10-17T19:09:48.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-61 zones={a2, a1} 2023-10-17T19:09:48.339 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.340 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.340 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.341 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:09:48.341 DEBUG:boto:Data: 2023-10-17T19:09:48.341 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.342 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.342 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.343 DEBUG:boto:Params: {} 2023-10-17T19:09:48.343 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:09:48.344 DEBUG:boto:Token: None 2023-10-17T19:09:48.345 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.345 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3U8dDvuP+wpvDaWiDsVEDyChpYU= 2023-10-17T19:09:48.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3U8dDvuP+wpvDaWiDsVEDyChpYU=', 'Content-Length': '0'} 2023-10-17T19:09:48.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083be65d483277f9a-00652edbfc-1cee-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.351 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.352 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.352 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.353 DEBUG:boto:Path: /hbmuog-61/ 2023-10-17T19:09:48.353 DEBUG:boto:Data: 2023-10-17T19:09:48.353 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.354 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.355 DEBUG:boto:Params: {} 2023-10-17T19:09:48.355 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:09:48.356 DEBUG:boto:Token: None 2023-10-17T19:09:48.356 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3U8dDvuP+wpvDaWiDsVEDyChpYU= 2023-10-17T19:09:48.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3U8dDvuP+wpvDaWiDsVEDyChpYU=', 'Content-Length': '0'} 2023-10-17T19:09:48.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000486340c036d44858-00652edbfc-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T19:09:48.363 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.364 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.364 DEBUG:boto:path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.365 DEBUG:boto:auth_path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.365 DEBUG:boto:Method: GET 2023-10-17T19:09:48.365 DEBUG:boto:Path: /hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.366 DEBUG:boto:Data: 2023-10-17T19:09:48.366 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.367 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.367 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.368 DEBUG:boto:Params: {} 2023-10-17T19:09:48.368 DEBUG:boto:Token: None 2023-10-17T19:09:48.368 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/?versions 2023-10-17T19:09:48.369 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE= 2023-10-17T19:09:48.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE=', 'Content-Length': '0'} 2023-10-17T19:09:48.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e25ebd11ea4ccd2-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.373 DEBUG:boto:b'hbmuog-61999falseobj11nulltrue2023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:09:48.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-17T19:09:48.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-17T19:09:48.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-17T19:09:48.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-17T19:09:48.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-17T19:09:48.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-17T19:09:48.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-17T19:09:48.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-17T19:09:48.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-17T19:09:48.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-17T19:09:48.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:09:48.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-17T19:09:48.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-17T19:09:48.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-17T19:09:48.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:09:48.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-17T19:09:48.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-17T19:09:48.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-17T19:09:48.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:09:48.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-17T19:09:48.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-17T19:09:48.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:09:48.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-17T19:09:48.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-17T19:09:48.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-17T19:09:48.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-17T19:09:48.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-17T19:09:48.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-17T19:09:48.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-17T19:09:48.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-17T19:09:48.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-17T19:09:48.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-17T19:09:48.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-17T19:09:48.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-17T19:09:48.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T19:09:48.388 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.388 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.389 DEBUG:boto:path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.389 DEBUG:boto:auth_path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.390 DEBUG:boto:Method: GET 2023-10-17T19:09:48.390 DEBUG:boto:Path: /hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.391 DEBUG:boto:Data: 2023-10-17T19:09:48.391 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.391 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.392 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.392 DEBUG:boto:Params: {} 2023-10-17T19:09:48.393 DEBUG:boto:Token: None 2023-10-17T19:09:48.393 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/?versions 2023-10-17T19:09:48.394 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE= 2023-10-17T19:09:48.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE=', 'Content-Length': '0'} 2023-10-17T19:09:48.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030655088d580178a-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.398 DEBUG:boto:b'hbmuog-61999falseobj11nulltrue2023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:09:48.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-17T19:09:48.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-17T19:09:48.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-17T19:09:48.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-17T19:09:48.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-17T19:09:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-17T19:09:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-17T19:09:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-17T19:09:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-17T19:09:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-17T19:09:48.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:09:48.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-17T19:09:48.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-17T19:09:48.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-17T19:09:48.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:09:48.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-17T19:09:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-17T19:09:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-17T19:09:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:09:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-17T19:09:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-17T19:09:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:09:48.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-17T19:09:48.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-17T19:09:48.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-17T19:09:48.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-17T19:09:48.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-17T19:09:48.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-17T19:09:48.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-17T19:09:48.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-17T19:09:48.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-17T19:09:48.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-17T19:09:48.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-17T19:09:48.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-17T19:09:48.413 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.414 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.414 DEBUG:boto:path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.415 DEBUG:boto:auth_path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.415 DEBUG:boto:Method: GET 2023-10-17T19:09:48.416 DEBUG:boto:Path: /hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.416 DEBUG:boto:Data: 2023-10-17T19:09:48.416 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.417 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.418 DEBUG:boto:Params: {} 2023-10-17T19:09:48.418 DEBUG:boto:Token: None 2023-10-17T19:09:48.419 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/?versions 2023-10-17T19:09:48.419 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE= 2023-10-17T19:09:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE=', 'Content-Length': '0'} 2023-10-17T19:09:48.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043916ccdf2b82973-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.423 DEBUG:boto:b'hbmuog-61999falseobj11nulltrue2023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.425 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.425 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.426 DEBUG:boto:path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.426 DEBUG:boto:auth_path=/hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.427 DEBUG:boto:Method: GET 2023-10-17T19:09:48.427 DEBUG:boto:Path: /hbmuog-61/?versions&max-keys=999 2023-10-17T19:09:48.427 DEBUG:boto:Data: 2023-10-17T19:09:48.428 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.428 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.429 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.429 DEBUG:boto:Params: {} 2023-10-17T19:09:48.429 DEBUG:boto:Token: None 2023-10-17T19:09:48.430 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/?versions 2023-10-17T19:09:48.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE= 2023-10-17T19:09:48.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vl0l8xgUBUUY+w0gVGV8VjlYwrE=', 'Content-Length': '0'} 2023-10-17T19:09:48.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bab19bb2da0d124-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.435 DEBUG:boto:b'hbmuog-61999falseobj11nulltrue2023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:09:48.438 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.438 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.439 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.439 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.440 DEBUG:boto:Method: GET 2023-10-17T19:09:48.440 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.440 DEBUG:boto:Data: 2023-10-17T19:09:48.441 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.441 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.442 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.442 DEBUG:boto:Params: {} 2023-10-17T19:09:48.443 DEBUG:boto:Token: None 2023-10-17T19:09:48.443 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.444 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI= 2023-10-17T19:09:48.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI=', 'Content-Length': '0'} 2023-10-17T19:09:48.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f0cbf3bbfa8fb3-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.447 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.447 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.448 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.448 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.449 DEBUG:boto:Method: GET 2023-10-17T19:09:48.449 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.450 DEBUG:boto:Data: 2023-10-17T19:09:48.450 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.450 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.451 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.451 DEBUG:boto:Params: {} 2023-10-17T19:09:48.452 DEBUG:boto:Token: None 2023-10-17T19:09:48.452 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI= 2023-10-17T19:09:48.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI=', 'Content-Length': '0'} 2023-10-17T19:09:48.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002becd4f6e370a4c0-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.502 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.505 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.506 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.506 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.506 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.507 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.507 DEBUG:boto:Data: 2023-10-17T19:09:48.508 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.508 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.509 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.509 DEBUG:boto:Params: {} 2023-10-17T19:09:48.509 DEBUG:boto:Token: None 2023-10-17T19:09:48.510 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.510 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rZ33TMth9dQS9oLF1iBsLBJBDxo= 2023-10-17T19:09:48.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rZ33TMth9dQS9oLF1iBsLBJBDxo=', 'Content-Length': '0'} 2023-10-17T19:09:48.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b52f5d139e9b8c-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.513 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.514 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.514 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.515 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.515 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.516 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.516 DEBUG:boto:Data: 2023-10-17T19:09:48.516 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.517 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.517 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.517 DEBUG:boto:Params: {} 2023-10-17T19:09:48.518 DEBUG:boto:Token: None 2023-10-17T19:09:48.518 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.519 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rZ33TMth9dQS9oLF1iBsLBJBDxo= 2023-10-17T19:09:48.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rZ33TMth9dQS9oLF1iBsLBJBDxo=', 'Content-Length': '0'} 2023-10-17T19:09:48.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c2916978a716302-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:09:48.522 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.522 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.523 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.523 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.524 DEBUG:boto:Method: GET 2023-10-17T19:09:48.524 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.525 DEBUG:boto:Data: 2023-10-17T19:09:48.525 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.525 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.526 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.526 DEBUG:boto:Params: {} 2023-10-17T19:09:48.527 DEBUG:boto:Token: None 2023-10-17T19:09:48.527 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.528 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI= 2023-10-17T19:09:48.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI=', 'Content-Length': '0'} 2023-10-17T19:09:48.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005369f6fa5604f011-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.531 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.531 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.531 DEBUG:boto:path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.532 DEBUG:boto:auth_path=/hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.532 DEBUG:boto:Method: GET 2023-10-17T19:09:48.533 DEBUG:boto:Path: /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.533 DEBUG:boto:Data: 2023-10-17T19:09:48.534 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.534 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.535 DEBUG:boto:Params: {} 2023-10-17T19:09:48.535 DEBUG:boto:Token: None 2023-10-17T19:09:48.536 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11?versionId=null 2023-10-17T19:09:48.536 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI= 2023-10-17T19:09:48.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jCxqEBLCgdNSvmQVXe6WiJgNGAI=', 'Content-Length': '0'} 2023-10-17T19:09:48.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065728e2ef1e42c5-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.539 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.539 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.540 DEBUG:boto:path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.540 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.541 DEBUG:boto:Method: GET 2023-10-17T19:09:48.541 DEBUG:boto:Path: /hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.542 DEBUG:boto:Data: 2023-10-17T19:09:48.542 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.543 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.543 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.544 DEBUG:boto:Params: {} 2023-10-17T19:09:48.544 DEBUG:boto:Token: None 2023-10-17T19:09:48.545 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.545 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000435107c1ec658e13-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.549 DEBUG:boto:b'hbmuog-61obj111000falseobj112023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.550 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.550 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.550 DEBUG:boto:path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.551 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.551 DEBUG:boto:Method: GET 2023-10-17T19:09:48.552 DEBUG:boto:Path: /hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.552 DEBUG:boto:Data: 2023-10-17T19:09:48.553 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.553 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.553 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.554 DEBUG:boto:Params: {} 2023-10-17T19:09:48.554 DEBUG:boto:Token: None 2023-10-17T19:09:48.555 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.555 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d5061e85b71310-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.560 DEBUG:boto:b'hbmuog-61obj111000falseobj112023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.561 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.561 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.562 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.562 DEBUG:boto:Path: /hbmuog-61/obj11 2023-10-17T19:09:48.563 DEBUG:boto:Data: 2023-10-17T19:09:48.563 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.563 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.564 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.564 DEBUG:boto:Params: {} 2023-10-17T19:09:48.565 DEBUG:boto:Token: None 2023-10-17T19:09:48.565 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11 2023-10-17T19:09:48.566 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MW3ygdapDzRGhpjKZfPeOA1FUvw= 2023-10-17T19:09:48.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MW3ygdapDzRGhpjKZfPeOA1FUvw=', 'Content-Length': '0'} 2023-10-17T19:09:48.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3fd1acb4be34b42-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.568 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.569 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.569 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.569 DEBUG:boto:Path: /hbmuog-61/obj11 2023-10-17T19:09:48.570 DEBUG:boto:Data: 2023-10-17T19:09:48.570 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.571 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.571 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.571 DEBUG:boto:Params: {} 2023-10-17T19:09:48.572 DEBUG:boto:Token: None 2023-10-17T19:09:48.572 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11 2023-10-17T19:09:48.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MW3ygdapDzRGhpjKZfPeOA1FUvw= 2023-10-17T19:09:48.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MW3ygdapDzRGhpjKZfPeOA1FUvw=', 'Content-Length': '0'} 2023-10-17T19:09:48.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2e082a097e96ea-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:09:48.576 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.576 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.577 DEBUG:boto:Method: GET 2023-10-17T19:09:48.577 DEBUG:boto:Path: /hbmuog-61/obj11 2023-10-17T19:09:48.578 DEBUG:boto:Data: 2023-10-17T19:09:48.578 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.579 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.580 DEBUG:boto:Params: {} 2023-10-17T19:09:48.580 DEBUG:boto:Token: None 2023-10-17T19:09:48.581 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11 2023-10-17T19:09:48.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pto8UMhiwVyE1fRQCIFTha+G7fk= 2023-10-17T19:09:48.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pto8UMhiwVyE1fRQCIFTha+G7fk=', 'Content-Length': '0'} 2023-10-17T19:09:48.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cdc0143cc404409-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.584 DEBUG:boto:path=/hbmuog-61/obj11 2023-10-17T19:09:48.584 DEBUG:boto:auth_path=/hbmuog-61/obj11 2023-10-17T19:09:48.585 DEBUG:boto:Method: GET 2023-10-17T19:09:48.585 DEBUG:boto:Path: /hbmuog-61/obj11 2023-10-17T19:09:48.586 DEBUG:boto:Data: 2023-10-17T19:09:48.586 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.587 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.587 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.587 DEBUG:boto:Params: {} 2023-10-17T19:09:48.588 DEBUG:boto:Token: None 2023-10-17T19:09:48.588 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj11 2023-10-17T19:09:48.589 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pto8UMhiwVyE1fRQCIFTha+G7fk= 2023-10-17T19:09:48.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pto8UMhiwVyE1fRQCIFTha+G7fk=', 'Content-Length': '0'} 2023-10-17T19:09:48.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2bcbf74c08ed41-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.634 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.635 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.635 DEBUG:boto:path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.636 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.636 DEBUG:boto:Method: GET 2023-10-17T19:09:48.637 DEBUG:boto:Path: /hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.637 DEBUG:boto:Data: 2023-10-17T19:09:48.638 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.638 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.639 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.639 DEBUG:boto:Params: {} 2023-10-17T19:09:48.640 DEBUG:boto:Token: None 2023-10-17T19:09:48.640 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.641 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb09e2c89feb3625-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.644 DEBUG:boto:b'hbmuog-61obj111000falseobj112023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.645 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.645 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.646 DEBUG:boto:path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.646 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.647 DEBUG:boto:Method: GET 2023-10-17T19:09:48.647 DEBUG:boto:Path: /hbmuog-61/?prefix=obj11 2023-10-17T19:09:48.647 DEBUG:boto:Data: 2023-10-17T19:09:48.648 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.649 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.649 DEBUG:boto:Params: {} 2023-10-17T19:09:48.650 DEBUG:boto:Token: None 2023-10-17T19:09:48.650 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd854f30802a235-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.656 DEBUG:boto:b'hbmuog-61obj111000falseobj112023-10-17T19:06:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-17T19:09:48.657 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.658 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.658 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.659 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.659 DEBUG:boto:Method: GET 2023-10-17T19:09:48.660 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.660 DEBUG:boto:Data: 2023-10-17T19:09:48.660 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.661 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.662 DEBUG:boto:Params: {} 2023-10-17T19:09:48.662 DEBUG:boto:Token: None 2023-10-17T19:09:48.663 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc= 2023-10-17T19:09:48.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc=', 'Content-Length': '0'} 2023-10-17T19:09:48.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006560ac9d4647ec9c-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.709 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.710 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.710 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.711 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.711 DEBUG:boto:Method: GET 2023-10-17T19:09:48.712 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.712 DEBUG:boto:Data: 2023-10-17T19:09:48.712 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.713 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.713 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.714 DEBUG:boto:Params: {} 2023-10-17T19:09:48.714 DEBUG:boto:Token: None 2023-10-17T19:09:48.715 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc= 2023-10-17T19:09:48.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc=', 'Content-Length': '0'} 2023-10-17T19:09:48.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba7bcc63419449e-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.719 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.720 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.720 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.721 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.721 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.722 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.722 DEBUG:boto:Data: 2023-10-17T19:09:48.723 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.723 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.723 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.724 DEBUG:boto:Params: {} 2023-10-17T19:09:48.724 DEBUG:boto:Token: None 2023-10-17T19:09:48.725 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.725 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y598Z+5A1BVIOSpYI8//D4zsShQ= 2023-10-17T19:09:48.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y598Z+5A1BVIOSpYI8//D4zsShQ=', 'Content-Length': '0'} 2023-10-17T19:09:48.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed926f7b1b63f060-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.729 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.729 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.730 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.730 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.731 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.731 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.731 DEBUG:boto:Data: 2023-10-17T19:09:48.732 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.733 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.733 DEBUG:boto:Params: {} 2023-10-17T19:09:48.733 DEBUG:boto:Token: None 2023-10-17T19:09:48.734 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.734 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y598Z+5A1BVIOSpYI8//D4zsShQ= 2023-10-17T19:09:48.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y598Z+5A1BVIOSpYI8//D4zsShQ=', 'Content-Length': '0'} 2023-10-17T19:09:48.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ae8823221fa310-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-17T19:09:48.737 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.738 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.738 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.739 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.739 DEBUG:boto:Method: GET 2023-10-17T19:09:48.740 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.740 DEBUG:boto:Data: 2023-10-17T19:09:48.740 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.741 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.742 DEBUG:boto:Params: {} 2023-10-17T19:09:48.742 DEBUG:boto:Token: None 2023-10-17T19:09:48.743 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.743 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc= 2023-10-17T19:09:48.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc=', 'Content-Length': '0'} 2023-10-17T19:09:48.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e050d2803320879-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.746 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.746 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.747 DEBUG:boto:path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.747 DEBUG:boto:auth_path=/hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.748 DEBUG:boto:Method: GET 2023-10-17T19:09:48.748 DEBUG:boto:Path: /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.748 DEBUG:boto:Data: 2023-10-17T19:09:48.749 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.750 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.750 DEBUG:boto:Params: {} 2023-10-17T19:09:48.751 DEBUG:boto:Token: None 2023-10-17T19:09:48.751 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111?versionId=null 2023-10-17T19:09:48.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc= 2023-10-17T19:09:48.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:esHbvgE3HDab7ReUwNrXhxhpJhc=', 'Content-Length': '0'} 2023-10-17T19:09:48.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eacb48bf6500c8d4-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.755 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.756 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.756 DEBUG:boto:path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.757 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.757 DEBUG:boto:Method: GET 2023-10-17T19:09:48.757 DEBUG:boto:Path: /hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.758 DEBUG:boto:Data: 2023-10-17T19:09:48.758 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.759 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.760 DEBUG:boto:Params: {} 2023-10-17T19:09:48.760 DEBUG:boto:Token: None 2023-10-17T19:09:48.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072fd089cb88e9293-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.765 DEBUG:boto:b'hbmuog-61obj1111000falseobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.766 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.766 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.767 DEBUG:boto:path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.767 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.768 DEBUG:boto:Method: GET 2023-10-17T19:09:48.768 DEBUG:boto:Path: /hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.769 DEBUG:boto:Data: 2023-10-17T19:09:48.769 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.769 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.770 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.770 DEBUG:boto:Params: {} 2023-10-17T19:09:48.771 DEBUG:boto:Token: None 2023-10-17T19:09:48.772 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ad846ffced9912-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.818 DEBUG:boto:b'hbmuog-61obj1111000falseobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.819 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.820 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.820 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.821 DEBUG:boto:Path: /hbmuog-61/obj111 2023-10-17T19:09:48.821 DEBUG:boto:Data: 2023-10-17T19:09:48.822 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.822 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.823 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.823 DEBUG:boto:Params: {} 2023-10-17T19:09:48.824 DEBUG:boto:Token: None 2023-10-17T19:09:48.824 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111 2023-10-17T19:09:48.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:APmR3PXHNdIoizY791icLSm6jP0= 2023-10-17T19:09:48.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:APmR3PXHNdIoizY791icLSm6jP0=', 'Content-Length': '0'} 2023-10-17T19:09:48.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f997582c97fbb95-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.828 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.829 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.829 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.830 DEBUG:boto:Path: /hbmuog-61/obj111 2023-10-17T19:09:48.830 DEBUG:boto:Data: 2023-10-17T19:09:48.831 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.831 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.831 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.832 DEBUG:boto:Params: {} 2023-10-17T19:09:48.832 DEBUG:boto:Token: None 2023-10-17T19:09:48.833 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111 2023-10-17T19:09:48.833 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:APmR3PXHNdIoizY791icLSm6jP0= 2023-10-17T19:09:48.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:APmR3PXHNdIoizY791icLSm6jP0=', 'Content-Length': '0'} 2023-10-17T19:09:48.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8807833b2c908e-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-17T19:09:48.837 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.838 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.838 DEBUG:boto:Method: GET 2023-10-17T19:09:48.839 DEBUG:boto:Path: /hbmuog-61/obj111 2023-10-17T19:09:48.839 DEBUG:boto:Data: 2023-10-17T19:09:48.840 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.840 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.841 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.841 DEBUG:boto:Params: {} 2023-10-17T19:09:48.842 DEBUG:boto:Token: None 2023-10-17T19:09:48.842 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111 2023-10-17T19:09:48.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kPRyX/+leB507IshCWa6ELItJe0= 2023-10-17T19:09:48.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kPRyX/+leB507IshCWa6ELItJe0=', 'Content-Length': '0'} 2023-10-17T19:09:48.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6587d7b1dcd927e-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.846 DEBUG:boto:path=/hbmuog-61/obj111 2023-10-17T19:09:48.846 DEBUG:boto:auth_path=/hbmuog-61/obj111 2023-10-17T19:09:48.847 DEBUG:boto:Method: GET 2023-10-17T19:09:48.847 DEBUG:boto:Path: /hbmuog-61/obj111 2023-10-17T19:09:48.848 DEBUG:boto:Data: 2023-10-17T19:09:48.848 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.849 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.850 DEBUG:boto:Params: {} 2023-10-17T19:09:48.850 DEBUG:boto:Token: None 2023-10-17T19:09:48.851 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj111 2023-10-17T19:09:48.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kPRyX/+leB507IshCWa6ELItJe0= 2023-10-17T19:09:48.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kPRyX/+leB507IshCWa6ELItJe0=', 'Content-Length': '0'} 2023-10-17T19:09:48.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ce3a249d60376a-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.855 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.855 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.856 DEBUG:boto:path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.856 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.857 DEBUG:boto:Method: GET 2023-10-17T19:09:48.857 DEBUG:boto:Path: /hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.858 DEBUG:boto:Data: 2023-10-17T19:09:48.858 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.859 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.859 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.859 DEBUG:boto:Params: {} 2023-10-17T19:09:48.860 DEBUG:boto:Token: None 2023-10-17T19:09:48.861 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.861 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000349e8394532907f-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.866 DEBUG:boto:b'hbmuog-61obj1111000falseobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.867 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.868 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.868 DEBUG:boto:path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.868 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.869 DEBUG:boto:Method: GET 2023-10-17T19:09:48.869 DEBUG:boto:Path: /hbmuog-61/?prefix=obj111 2023-10-17T19:09:48.870 DEBUG:boto:Data: 2023-10-17T19:09:48.870 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.871 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.872 DEBUG:boto:Params: {} 2023-10-17T19:09:48.872 DEBUG:boto:Token: None 2023-10-17T19:09:48.873 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002961bee27aefc2b7-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.878 DEBUG:boto:b'hbmuog-61obj1111000falseobj1112023-10-17T19:06:21.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-17T19:09:48.879 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.880 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.880 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.881 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.881 DEBUG:boto:Method: GET 2023-10-17T19:09:48.882 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.882 DEBUG:boto:Data: 2023-10-17T19:09:48.882 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.883 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.883 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.884 DEBUG:boto:Params: {} 2023-10-17T19:09:48.884 DEBUG:boto:Token: None 2023-10-17T19:09:48.885 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.886 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk= 2023-10-17T19:09:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk=', 'Content-Length': '0'} 2023-10-17T19:09:48.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdab397af1eb5e4b-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.889 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.890 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.890 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.890 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.891 DEBUG:boto:Method: GET 2023-10-17T19:09:48.891 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.892 DEBUG:boto:Data: 2023-10-17T19:09:48.892 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.893 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.893 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.894 DEBUG:boto:Params: {} 2023-10-17T19:09:48.894 DEBUG:boto:Token: None 2023-10-17T19:09:48.895 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.895 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk= 2023-10-17T19:09:48.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk=', 'Content-Length': '0'} 2023-10-17T19:09:48.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074330748337bf3e4-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.898 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.899 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.899 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.900 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.900 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.901 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.901 DEBUG:boto:Data: 2023-10-17T19:09:48.902 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.902 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.902 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.903 DEBUG:boto:Params: {} 2023-10-17T19:09:48.903 DEBUG:boto:Token: None 2023-10-17T19:09:48.904 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gY8923X72FEmZCXEkhFhcnL79VM= 2023-10-17T19:09:48.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gY8923X72FEmZCXEkhFhcnL79VM=', 'Content-Length': '0'} 2023-10-17T19:09:48.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3259ddea8bf956-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.907 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.908 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.908 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.909 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.909 DEBUG:boto:Method: HEAD 2023-10-17T19:09:48.909 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.910 DEBUG:boto:Data: 2023-10-17T19:09:48.910 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.911 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.911 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.912 DEBUG:boto:Params: {} 2023-10-17T19:09:48.912 DEBUG:boto:Token: None 2023-10-17T19:09:48.913 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.913 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gY8923X72FEmZCXEkhFhcnL79VM= 2023-10-17T19:09:48.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gY8923X72FEmZCXEkhFhcnL79VM=', 'Content-Length': '0'} 2023-10-17T19:09:48.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001672b3b5af1e6d49-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-17T19:09:48.917 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.917 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.918 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.918 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.918 DEBUG:boto:Method: GET 2023-10-17T19:09:48.919 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.919 DEBUG:boto:Data: 2023-10-17T19:09:48.920 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.920 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.921 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.921 DEBUG:boto:Params: {} 2023-10-17T19:09:48.922 DEBUG:boto:Token: None 2023-10-17T19:09:48.923 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk= 2023-10-17T19:09:48.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk=', 'Content-Length': '0'} 2023-10-17T19:09:48.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9baf2e71a5a1d3e-00652edbfc-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.926 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:48.927 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:48.927 DEBUG:boto:path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.928 DEBUG:boto:auth_path=/hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.928 DEBUG:boto:Method: GET 2023-10-17T19:09:48.929 DEBUG:boto:Path: /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.929 DEBUG:boto:Data: 2023-10-17T19:09:48.930 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.930 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.931 DEBUG:boto:Params: {} 2023-10-17T19:09:48.931 DEBUG:boto:Token: None 2023-10-17T19:09:48.932 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/obj115?versionId=null 2023-10-17T19:09:48.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk= 2023-10-17T19:09:48.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Za9xncFl3EWAEkBJSYJOqDa0cXk=', 'Content-Length': '0'} 2023-10-17T19:09:48.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e036d6fadfeb548-00652edbfc-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.935 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.936 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.936 DEBUG:boto:path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.937 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.937 DEBUG:boto:Method: GET 2023-10-17T19:09:48.938 DEBUG:boto:Path: /hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.938 DEBUG:boto:Data: 2023-10-17T19:09:48.939 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.939 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:48.940 DEBUG:boto:Port: 8001 2023-10-17T19:09:48.940 DEBUG:boto:Params: {} 2023-10-17T19:09:48.941 DEBUG:boto:Token: None 2023-10-17T19:09:48.941 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa07c69e40fef62-00652edbfc-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.945 DEBUG:boto:b'hbmuog-61obj1151000falseobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.946 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:48.946 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:48.947 DEBUG:boto:path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.947 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.948 DEBUG:boto:Method: GET 2023-10-17T19:09:48.948 DEBUG:boto:Path: /hbmuog-61/?prefix=obj115 2023-10-17T19:09:48.949 DEBUG:boto:Data: 2023-10-17T19:09:48.949 DEBUG:boto:Headers: {} 2023-10-17T19:09:48.950 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:48.950 DEBUG:boto:Port: 8000 2023-10-17T19:09:48.950 DEBUG:boto:Params: {} 2023-10-17T19:09:48.951 DEBUG:boto:Token: None 2023-10-17T19:09:48.952 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:48 GMT /hbmuog-61/ 2023-10-17T19:09:48.952 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4= 2023-10-17T19:09:48.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:48 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+EtCwXTS3ZR5UG3b59/up4ipch4=', 'Content-Length': '0'} 2023-10-17T19:09:48.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ea95e956868b92-00652edbfc-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:48.998 DEBUG:boto:b'hbmuog-61obj1151000falseobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:48.999 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:49.000 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:49.000 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.001 DEBUG:boto:Path: /hbmuog-61/obj115 2023-10-17T19:09:49.001 DEBUG:boto:Data: 2023-10-17T19:09:49.001 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.002 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.002 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.003 DEBUG:boto:Params: {} 2023-10-17T19:09:49.003 DEBUG:boto:Token: None 2023-10-17T19:09:49.004 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj115 2023-10-17T19:09:49.004 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aTAR5Ny5EQZu6l+Oe8VX1/3KCKg= 2023-10-17T19:09:49.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aTAR5Ny5EQZu6l+Oe8VX1/3KCKg=', 'Content-Length': '0'} 2023-10-17T19:09:49.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed5715402b215bd-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.008 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:49.009 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:49.010 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.011 DEBUG:boto:Path: /hbmuog-61/obj115 2023-10-17T19:09:49.012 DEBUG:boto:Data: 2023-10-17T19:09:49.013 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.014 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.014 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.015 DEBUG:boto:Params: {} 2023-10-17T19:09:49.015 DEBUG:boto:Token: None 2023-10-17T19:09:49.016 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj115 2023-10-17T19:09:49.016 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aTAR5Ny5EQZu6l+Oe8VX1/3KCKg= 2023-10-17T19:09:49.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aTAR5Ny5EQZu6l+Oe8VX1/3KCKg=', 'Content-Length': '0'} 2023-10-17T19:09:49.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec97cdf41253ace-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-17T19:09:49.020 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:49.020 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:49.021 DEBUG:boto:Method: GET 2023-10-17T19:09:49.021 DEBUG:boto:Path: /hbmuog-61/obj115 2023-10-17T19:09:49.022 DEBUG:boto:Data: 2023-10-17T19:09:49.022 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.023 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.023 DEBUG:boto:Params: {} 2023-10-17T19:09:49.024 DEBUG:boto:Token: None 2023-10-17T19:09:49.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj115 2023-10-17T19:09:49.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dpcf6/cVL300YYebiwyJ/3+18fE= 2023-10-17T19:09:49.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dpcf6/cVL300YYebiwyJ/3+18fE=', 'Content-Length': '0'} 2023-10-17T19:09:49.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f6080aeb3fc08e-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.074 DEBUG:boto:path=/hbmuog-61/obj115 2023-10-17T19:09:49.075 DEBUG:boto:auth_path=/hbmuog-61/obj115 2023-10-17T19:09:49.075 DEBUG:boto:Method: GET 2023-10-17T19:09:49.076 DEBUG:boto:Path: /hbmuog-61/obj115 2023-10-17T19:09:49.076 DEBUG:boto:Data: 2023-10-17T19:09:49.077 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.077 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.077 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.078 DEBUG:boto:Params: {} 2023-10-17T19:09:49.078 DEBUG:boto:Token: None 2023-10-17T19:09:49.079 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj115 2023-10-17T19:09:49.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dpcf6/cVL300YYebiwyJ/3+18fE= 2023-10-17T19:09:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dpcf6/cVL300YYebiwyJ/3+18fE=', 'Content-Length': '0'} 2023-10-17T19:09:49.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea47ef40887450a-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.083 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.083 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.083 DEBUG:boto:path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.084 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.084 DEBUG:boto:Method: GET 2023-10-17T19:09:49.085 DEBUG:boto:Path: /hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.085 DEBUG:boto:Data: 2023-10-17T19:09:49.085 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.086 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.086 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.087 DEBUG:boto:Params: {} 2023-10-17T19:09:49.088 DEBUG:boto:Token: None 2023-10-17T19:09:49.089 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a65d45ebd1d63c8-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.093 DEBUG:boto:b'hbmuog-61obj1151000falseobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.094 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.094 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.095 DEBUG:boto:path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.095 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.096 DEBUG:boto:Method: GET 2023-10-17T19:09:49.096 DEBUG:boto:Path: /hbmuog-61/?prefix=obj115 2023-10-17T19:09:49.097 DEBUG:boto:Data: 2023-10-17T19:09:49.097 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.098 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.098 DEBUG:boto:Params: {} 2023-10-17T19:09:49.099 DEBUG:boto:Token: None 2023-10-17T19:09:49.099 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.100 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003569d9a51af912c7-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.104 DEBUG:boto:b'hbmuog-61obj1151000falseobj1152023-10-17T19:06:20.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-17T19:09:49.105 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.106 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.106 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.107 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.107 DEBUG:boto:Method: GET 2023-10-17T19:09:49.107 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.108 DEBUG:boto:Data: 2023-10-17T19:09:49.108 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.109 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.110 DEBUG:boto:Params: {} 2023-10-17T19:09:49.110 DEBUG:boto:Token: None 2023-10-17T19:09:49.111 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo= 2023-10-17T19:09:49.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo=', 'Content-Length': '0'} 2023-10-17T19:09:49.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a4a7010f0e7125-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.115 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.115 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.116 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.116 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.117 DEBUG:boto:Method: GET 2023-10-17T19:09:49.118 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.118 DEBUG:boto:Data: 2023-10-17T19:09:49.119 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.119 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.120 DEBUG:boto:Params: {} 2023-10-17T19:09:49.120 DEBUG:boto:Token: None 2023-10-17T19:09:49.121 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.121 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo= 2023-10-17T19:09:49.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo=', 'Content-Length': '0'} 2023-10-17T19:09:49.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef1e0ac97e38857-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.127 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.128 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.128 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.129 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.130 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.130 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.131 DEBUG:boto:Data: 2023-10-17T19:09:49.132 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.132 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.133 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.134 DEBUG:boto:Params: {} 2023-10-17T19:09:49.135 DEBUG:boto:Token: None 2023-10-17T19:09:49.135 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.136 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K73r//zcrUL5yVa3hZ93Op2yIlk= 2023-10-17T19:09:49.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K73r//zcrUL5yVa3hZ93Op2yIlk=', 'Content-Length': '0'} 2023-10-17T19:09:49.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044437d3644962039-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.139 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.140 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.140 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.140 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.141 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.141 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.142 DEBUG:boto:Data: 2023-10-17T19:09:49.142 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.142 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.143 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.143 DEBUG:boto:Params: {} 2023-10-17T19:09:49.144 DEBUG:boto:Token: None 2023-10-17T19:09:49.145 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.145 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:K73r//zcrUL5yVa3hZ93Op2yIlk= 2023-10-17T19:09:49.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:K73r//zcrUL5yVa3hZ93Op2yIlk=', 'Content-Length': '0'} 2023-10-17T19:09:49.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000810888482a368cb5-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-17T19:09:49.149 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.149 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.150 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.150 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.151 DEBUG:boto:Method: GET 2023-10-17T19:09:49.151 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.152 DEBUG:boto:Data: 2023-10-17T19:09:49.152 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.153 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.153 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.154 DEBUG:boto:Params: {} 2023-10-17T19:09:49.154 DEBUG:boto:Token: None 2023-10-17T19:09:49.155 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.156 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo= 2023-10-17T19:09:49.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo=', 'Content-Length': '0'} 2023-10-17T19:09:49.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd68420299ee73ef-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.159 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.160 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.161 DEBUG:boto:path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.162 DEBUG:boto:auth_path=/hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.163 DEBUG:boto:Method: GET 2023-10-17T19:09:49.163 DEBUG:boto:Path: /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.164 DEBUG:boto:Data: 2023-10-17T19:09:49.165 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.166 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.166 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.167 DEBUG:boto:Params: {} 2023-10-17T19:09:49.168 DEBUG:boto:Token: None 2023-10-17T19:09:49.168 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116?versionId=null 2023-10-17T19:09:49.169 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo= 2023-10-17T19:09:49.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cZxpaW3svbdN/HRLNiKfdtjntPo=', 'Content-Length': '0'} 2023-10-17T19:09:49.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d335fed0af03c2e7-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.214 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.215 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.216 DEBUG:boto:path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.216 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.217 DEBUG:boto:Method: GET 2023-10-17T19:09:49.217 DEBUG:boto:Path: /hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.218 DEBUG:boto:Data: 2023-10-17T19:09:49.218 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.219 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.219 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.220 DEBUG:boto:Params: {} 2023-10-17T19:09:49.220 DEBUG:boto:Token: None 2023-10-17T19:09:49.221 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.221 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef33b12e989fca6f-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.225 DEBUG:boto:b'hbmuog-61obj1161000falseobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.226 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.226 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.227 DEBUG:boto:path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.227 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.228 DEBUG:boto:Method: GET 2023-10-17T19:09:49.228 DEBUG:boto:Path: /hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.229 DEBUG:boto:Data: 2023-10-17T19:09:49.229 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.230 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.230 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.231 DEBUG:boto:Params: {} 2023-10-17T19:09:49.231 DEBUG:boto:Token: None 2023-10-17T19:09:49.232 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.232 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b15ab878b0ba1c0-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.237 DEBUG:boto:b'hbmuog-61obj1161000falseobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.238 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.238 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.239 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.239 DEBUG:boto:Path: /hbmuog-61/obj116 2023-10-17T19:09:49.240 DEBUG:boto:Data: 2023-10-17T19:09:49.240 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.241 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.242 DEBUG:boto:Params: {} 2023-10-17T19:09:49.242 DEBUG:boto:Token: None 2023-10-17T19:09:49.243 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116 2023-10-17T19:09:49.243 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mv7sTHtvs0ab9ZrZXA0KZAuaM0Q= 2023-10-17T19:09:49.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mv7sTHtvs0ab9ZrZXA0KZAuaM0Q=', 'Content-Length': '0'} 2023-10-17T19:09:49.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7dff7f562a3aaf9-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.246 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.247 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.247 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.248 DEBUG:boto:Path: /hbmuog-61/obj116 2023-10-17T19:09:49.248 DEBUG:boto:Data: 2023-10-17T19:09:49.249 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.250 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.250 DEBUG:boto:Params: {} 2023-10-17T19:09:49.251 DEBUG:boto:Token: None 2023-10-17T19:09:49.251 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116 2023-10-17T19:09:49.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mv7sTHtvs0ab9ZrZXA0KZAuaM0Q= 2023-10-17T19:09:49.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mv7sTHtvs0ab9ZrZXA0KZAuaM0Q=', 'Content-Length': '0'} 2023-10-17T19:09:49.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106826dc89fe63b7-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-17T19:09:49.255 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.256 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.256 DEBUG:boto:Method: GET 2023-10-17T19:09:49.257 DEBUG:boto:Path: /hbmuog-61/obj116 2023-10-17T19:09:49.257 DEBUG:boto:Data: 2023-10-17T19:09:49.258 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.259 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.259 DEBUG:boto:Params: {} 2023-10-17T19:09:49.260 DEBUG:boto:Token: None 2023-10-17T19:09:49.261 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116 2023-10-17T19:09:49.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2ZlRDNIE5mDe5Brakx/0i3J0AWg= 2023-10-17T19:09:49.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2ZlRDNIE5mDe5Brakx/0i3J0AWg=', 'Content-Length': '0'} 2023-10-17T19:09:49.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6a36def0ca5508-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.264 DEBUG:boto:path=/hbmuog-61/obj116 2023-10-17T19:09:49.264 DEBUG:boto:auth_path=/hbmuog-61/obj116 2023-10-17T19:09:49.265 DEBUG:boto:Method: GET 2023-10-17T19:09:49.265 DEBUG:boto:Path: /hbmuog-61/obj116 2023-10-17T19:09:49.266 DEBUG:boto:Data: 2023-10-17T19:09:49.266 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.267 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.268 DEBUG:boto:Params: {} 2023-10-17T19:09:49.268 DEBUG:boto:Token: None 2023-10-17T19:09:49.269 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj116 2023-10-17T19:09:49.269 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2ZlRDNIE5mDe5Brakx/0i3J0AWg= 2023-10-17T19:09:49.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2ZlRDNIE5mDe5Brakx/0i3J0AWg=', 'Content-Length': '0'} 2023-10-17T19:09:49.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9dd45045449ba94-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.273 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.273 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.274 DEBUG:boto:path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.274 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.275 DEBUG:boto:Method: GET 2023-10-17T19:09:49.275 DEBUG:boto:Path: /hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.275 DEBUG:boto:Data: 2023-10-17T19:09:49.276 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.276 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.277 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.277 DEBUG:boto:Params: {} 2023-10-17T19:09:49.278 DEBUG:boto:Token: None 2023-10-17T19:09:49.278 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.279 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e27a5b281409dad-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.282 DEBUG:boto:b'hbmuog-61obj1161000falseobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.283 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.284 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.284 DEBUG:boto:path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.284 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.285 DEBUG:boto:Method: GET 2023-10-17T19:09:49.285 DEBUG:boto:Path: /hbmuog-61/?prefix=obj116 2023-10-17T19:09:49.286 DEBUG:boto:Data: 2023-10-17T19:09:49.286 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.287 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.288 DEBUG:boto:Params: {} 2023-10-17T19:09:49.288 DEBUG:boto:Token: None 2023-10-17T19:09:49.289 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed739bb8cc452d1-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.294 DEBUG:boto:b'hbmuog-61obj1161000falseobj1162023-10-17T19:06:21.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-17T19:09:49.296 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.296 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.297 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.297 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.298 DEBUG:boto:Method: GET 2023-10-17T19:09:49.298 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.299 DEBUG:boto:Data: 2023-10-17T19:09:49.300 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.301 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.301 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.302 DEBUG:boto:Params: {} 2023-10-17T19:09:49.303 DEBUG:boto:Token: None 2023-10-17T19:09:49.304 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8= 2023-10-17T19:09:49.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8=', 'Content-Length': '0'} 2023-10-17T19:09:49.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542e3a0afd11c51f-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.309 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.309 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.310 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.310 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.311 DEBUG:boto:Method: GET 2023-10-17T19:09:49.311 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.311 DEBUG:boto:Data: 2023-10-17T19:09:49.312 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.312 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.313 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.313 DEBUG:boto:Params: {} 2023-10-17T19:09:49.314 DEBUG:boto:Token: None 2023-10-17T19:09:49.315 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8= 2023-10-17T19:09:49.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8=', 'Content-Length': '0'} 2023-10-17T19:09:49.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b975c77ce122e94-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.318 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.319 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.319 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.320 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.320 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.320 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.321 DEBUG:boto:Data: 2023-10-17T19:09:49.321 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.322 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.322 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.323 DEBUG:boto:Params: {} 2023-10-17T19:09:49.323 DEBUG:boto:Token: None 2023-10-17T19:09:49.323 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.324 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wXRISfLik7B+Uye0JolgnpWIyW4= 2023-10-17T19:09:49.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wXRISfLik7B+Uye0JolgnpWIyW4=', 'Content-Length': '0'} 2023-10-17T19:09:49.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c323b2962cb7ba02-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.328 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.329 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.330 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.330 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.331 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.333 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.334 DEBUG:boto:Data: 2023-10-17T19:09:49.335 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.335 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.335 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.336 DEBUG:boto:Params: {} 2023-10-17T19:09:49.336 DEBUG:boto:Token: None 2023-10-17T19:09:49.337 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wXRISfLik7B+Uye0JolgnpWIyW4= 2023-10-17T19:09:49.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wXRISfLik7B+Uye0JolgnpWIyW4=', 'Content-Length': '0'} 2023-10-17T19:09:49.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f81bef4e03af090-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-17T19:09:49.341 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.341 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.342 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.342 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.343 DEBUG:boto:Method: GET 2023-10-17T19:09:49.343 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.344 DEBUG:boto:Data: 2023-10-17T19:09:49.344 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.345 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.346 DEBUG:boto:Params: {} 2023-10-17T19:09:49.346 DEBUG:boto:Token: None 2023-10-17T19:09:49.347 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8= 2023-10-17T19:09:49.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8=', 'Content-Length': '0'} 2023-10-17T19:09:49.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf7f5e6f884b54c-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.394 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.395 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.395 DEBUG:boto:path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.396 DEBUG:boto:auth_path=/hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.396 DEBUG:boto:Method: GET 2023-10-17T19:09:49.397 DEBUG:boto:Path: /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.397 DEBUG:boto:Data: 2023-10-17T19:09:49.398 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.398 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.399 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.399 DEBUG:boto:Params: {} 2023-10-17T19:09:49.400 DEBUG:boto:Token: None 2023-10-17T19:09:49.400 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124?versionId=null 2023-10-17T19:09:49.401 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8= 2023-10-17T19:09:49.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ntd8eA/+JahdEfC/ijC7VmGUyd8=', 'Content-Length': '0'} 2023-10-17T19:09:49.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8263a78ab02782-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.404 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.404 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.405 DEBUG:boto:path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.405 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.406 DEBUG:boto:Method: GET 2023-10-17T19:09:49.406 DEBUG:boto:Path: /hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.407 DEBUG:boto:Data: 2023-10-17T19:09:49.407 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.408 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.408 DEBUG:boto:Params: {} 2023-10-17T19:09:49.409 DEBUG:boto:Token: None 2023-10-17T19:09:49.409 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.410 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a9041639f2a86cf-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.413 DEBUG:boto:b'hbmuog-61obj1241000falseobj1242023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.414 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.414 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.415 DEBUG:boto:path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.415 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.416 DEBUG:boto:Method: GET 2023-10-17T19:09:49.416 DEBUG:boto:Path: /hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.417 DEBUG:boto:Data: 2023-10-17T19:09:49.417 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.418 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.418 DEBUG:boto:Params: {} 2023-10-17T19:09:49.419 DEBUG:boto:Token: None 2023-10-17T19:09:49.420 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001268771cc72699ea-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.466 DEBUG:boto:b'hbmuog-61obj1241000falseobj1242023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.467 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.468 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.468 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.469 DEBUG:boto:Path: /hbmuog-61/obj124 2023-10-17T19:09:49.469 DEBUG:boto:Data: 2023-10-17T19:09:49.469 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.470 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.470 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.471 DEBUG:boto:Params: {} 2023-10-17T19:09:49.471 DEBUG:boto:Token: None 2023-10-17T19:09:49.472 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124 2023-10-17T19:09:49.472 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZKFa6SQmTyiBsxbOuKbqS/0ehxE= 2023-10-17T19:09:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZKFa6SQmTyiBsxbOuKbqS/0ehxE=', 'Content-Length': '0'} 2023-10-17T19:09:49.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a9670f12b72e73-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.475 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.476 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.476 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.476 DEBUG:boto:Path: /hbmuog-61/obj124 2023-10-17T19:09:49.477 DEBUG:boto:Data: 2023-10-17T19:09:49.477 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.478 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.478 DEBUG:boto:Params: {} 2023-10-17T19:09:49.479 DEBUG:boto:Token: None 2023-10-17T19:09:49.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124 2023-10-17T19:09:49.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZKFa6SQmTyiBsxbOuKbqS/0ehxE= 2023-10-17T19:09:49.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZKFa6SQmTyiBsxbOuKbqS/0ehxE=', 'Content-Length': '0'} 2023-10-17T19:09:49.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233b82a6230f5d6b-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-17T19:09:49.483 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.484 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.484 DEBUG:boto:Method: GET 2023-10-17T19:09:49.484 DEBUG:boto:Path: /hbmuog-61/obj124 2023-10-17T19:09:49.485 DEBUG:boto:Data: 2023-10-17T19:09:49.485 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.486 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.486 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.487 DEBUG:boto:Params: {} 2023-10-17T19:09:49.487 DEBUG:boto:Token: None 2023-10-17T19:09:49.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124 2023-10-17T19:09:49.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jIKC9vLlU/5RVqzhIrPlo44db8= 2023-10-17T19:09:49.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jIKC9vLlU/5RVqzhIrPlo44db8=', 'Content-Length': '0'} 2023-10-17T19:09:49.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b39f10b0518abc1-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.491 DEBUG:boto:path=/hbmuog-61/obj124 2023-10-17T19:09:49.491 DEBUG:boto:auth_path=/hbmuog-61/obj124 2023-10-17T19:09:49.492 DEBUG:boto:Method: GET 2023-10-17T19:09:49.492 DEBUG:boto:Path: /hbmuog-61/obj124 2023-10-17T19:09:49.493 DEBUG:boto:Data: 2023-10-17T19:09:49.493 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.494 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.494 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.494 DEBUG:boto:Params: {} 2023-10-17T19:09:49.495 DEBUG:boto:Token: None 2023-10-17T19:09:49.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj124 2023-10-17T19:09:49.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5jIKC9vLlU/5RVqzhIrPlo44db8= 2023-10-17T19:09:49.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5jIKC9vLlU/5RVqzhIrPlo44db8=', 'Content-Length': '0'} 2023-10-17T19:09:49.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fbb92058829c28-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.499 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.499 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.500 DEBUG:boto:path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.500 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.501 DEBUG:boto:Method: GET 2023-10-17T19:09:49.501 DEBUG:boto:Path: /hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.502 DEBUG:boto:Data: 2023-10-17T19:09:49.502 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.503 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.503 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.503 DEBUG:boto:Params: {} 2023-10-17T19:09:49.504 DEBUG:boto:Token: None 2023-10-17T19:09:49.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001705d975a5f89760-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.554 DEBUG:boto:b'hbmuog-61obj1241000falseobj1242023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.555 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.556 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.556 DEBUG:boto:path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.557 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.557 DEBUG:boto:Method: GET 2023-10-17T19:09:49.558 DEBUG:boto:Path: /hbmuog-61/?prefix=obj124 2023-10-17T19:09:49.558 DEBUG:boto:Data: 2023-10-17T19:09:49.559 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.559 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.560 DEBUG:boto:Params: {} 2023-10-17T19:09:49.560 DEBUG:boto:Token: None 2023-10-17T19:09:49.561 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e4a287014235dde-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.566 DEBUG:boto:b'hbmuog-61obj1241000falseobj1242023-10-17T19:06:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-17T19:09:49.567 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.568 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.568 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.569 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.569 DEBUG:boto:Method: GET 2023-10-17T19:09:49.569 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.570 DEBUG:boto:Data: 2023-10-17T19:09:49.570 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.571 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.571 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.572 DEBUG:boto:Params: {} 2023-10-17T19:09:49.572 DEBUG:boto:Token: None 2023-10-17T19:09:49.573 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0= 2023-10-17T19:09:49.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0=', 'Content-Length': '0'} 2023-10-17T19:09:49.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e363a4b7bb5a9f-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.576 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.577 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.577 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.578 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.578 DEBUG:boto:Method: GET 2023-10-17T19:09:49.579 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.579 DEBUG:boto:Data: 2023-10-17T19:09:49.580 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.580 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.580 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.581 DEBUG:boto:Params: {} 2023-10-17T19:09:49.581 DEBUG:boto:Token: None 2023-10-17T19:09:49.582 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.582 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0= 2023-10-17T19:09:49.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0=', 'Content-Length': '0'} 2023-10-17T19:09:49.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf96d38b8fdc9c2e-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.586 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.586 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.587 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.587 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.587 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.588 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.588 DEBUG:boto:Data: 2023-10-17T19:09:49.589 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.589 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.589 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.590 DEBUG:boto:Params: {} 2023-10-17T19:09:49.590 DEBUG:boto:Token: None 2023-10-17T19:09:49.591 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DvNSbS4aBoifzEQ/6l0lAEFHexM= 2023-10-17T19:09:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DvNSbS4aBoifzEQ/6l0lAEFHexM=', 'Content-Length': '0'} 2023-10-17T19:09:49.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e7ffaa62e594e2-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.594 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.595 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.595 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.596 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.596 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.597 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.597 DEBUG:boto:Data: 2023-10-17T19:09:49.598 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.599 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.599 DEBUG:boto:Params: {} 2023-10-17T19:09:49.599 DEBUG:boto:Token: None 2023-10-17T19:09:49.600 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DvNSbS4aBoifzEQ/6l0lAEFHexM= 2023-10-17T19:09:49.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DvNSbS4aBoifzEQ/6l0lAEFHexM=', 'Content-Length': '0'} 2023-10-17T19:09:49.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea6949652301a8c6-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-17T19:09:49.604 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.605 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.605 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.605 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.606 DEBUG:boto:Method: GET 2023-10-17T19:09:49.606 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.607 DEBUG:boto:Data: 2023-10-17T19:09:49.607 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.608 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.608 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.609 DEBUG:boto:Params: {} 2023-10-17T19:09:49.609 DEBUG:boto:Token: None 2023-10-17T19:09:49.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.610 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0= 2023-10-17T19:09:49.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0=', 'Content-Length': '0'} 2023-10-17T19:09:49.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb39411ceae1769d-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.613 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.613 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.614 DEBUG:boto:path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.614 DEBUG:boto:auth_path=/hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.615 DEBUG:boto:Method: GET 2023-10-17T19:09:49.615 DEBUG:boto:Path: /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.616 DEBUG:boto:Data: 2023-10-17T19:09:49.616 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.617 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.617 DEBUG:boto:Params: {} 2023-10-17T19:09:49.618 DEBUG:boto:Token: None 2023-10-17T19:09:49.618 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145?versionId=null 2023-10-17T19:09:49.619 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0= 2023-10-17T19:09:49.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/QqVRtsDGLOkbah9RsJVicwEDr0=', 'Content-Length': '0'} 2023-10-17T19:09:49.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba2b2dcfe8f8b59-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.622 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.623 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.623 DEBUG:boto:path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.623 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.624 DEBUG:boto:Method: GET 2023-10-17T19:09:49.624 DEBUG:boto:Path: /hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.625 DEBUG:boto:Data: 2023-10-17T19:09:49.625 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.626 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.626 DEBUG:boto:Params: {} 2023-10-17T19:09:49.627 DEBUG:boto:Token: None 2023-10-17T19:09:49.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac1b8551d55d5f8-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.631 DEBUG:boto:b'hbmuog-61obj1451000falseobj1452023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.632 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.632 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.633 DEBUG:boto:path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.633 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.634 DEBUG:boto:Method: GET 2023-10-17T19:09:49.634 DEBUG:boto:Path: /hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.635 DEBUG:boto:Data: 2023-10-17T19:09:49.635 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.636 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.636 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.636 DEBUG:boto:Params: {} 2023-10-17T19:09:49.637 DEBUG:boto:Token: None 2023-10-17T19:09:49.638 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.638 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fadb02ad15b1621a-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.643 DEBUG:boto:b'hbmuog-61obj1451000falseobj1452023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.643 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.644 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.644 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.645 DEBUG:boto:Path: /hbmuog-61/obj145 2023-10-17T19:09:49.645 DEBUG:boto:Data: 2023-10-17T19:09:49.646 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.646 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.646 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.647 DEBUG:boto:Params: {} 2023-10-17T19:09:49.647 DEBUG:boto:Token: None 2023-10-17T19:09:49.648 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145 2023-10-17T19:09:49.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GCAHFgXjpvyxEjvPVFvzRMclELc= 2023-10-17T19:09:49.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GCAHFgXjpvyxEjvPVFvzRMclELc=', 'Content-Length': '0'} 2023-10-17T19:09:49.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a14eb0e3e55c0b0-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.651 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.651 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.652 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.652 DEBUG:boto:Path: /hbmuog-61/obj145 2023-10-17T19:09:49.653 DEBUG:boto:Data: 2023-10-17T19:09:49.653 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.654 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.654 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.654 DEBUG:boto:Params: {} 2023-10-17T19:09:49.655 DEBUG:boto:Token: None 2023-10-17T19:09:49.655 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145 2023-10-17T19:09:49.656 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GCAHFgXjpvyxEjvPVFvzRMclELc= 2023-10-17T19:09:49.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GCAHFgXjpvyxEjvPVFvzRMclELc=', 'Content-Length': '0'} 2023-10-17T19:09:49.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcff78f35cecbd1-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-17T19:09:49.659 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.660 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.660 DEBUG:boto:Method: GET 2023-10-17T19:09:49.661 DEBUG:boto:Path: /hbmuog-61/obj145 2023-10-17T19:09:49.661 DEBUG:boto:Data: 2023-10-17T19:09:49.662 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.662 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.663 DEBUG:boto:Params: {} 2023-10-17T19:09:49.663 DEBUG:boto:Token: None 2023-10-17T19:09:49.664 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145 2023-10-17T19:09:49.664 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iag17PzeXjKyma5jttwWx+U1MKQ= 2023-10-17T19:09:49.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iag17PzeXjKyma5jttwWx+U1MKQ=', 'Content-Length': '0'} 2023-10-17T19:09:49.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a96d1e6cd01f0bd6-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.714 DEBUG:boto:path=/hbmuog-61/obj145 2023-10-17T19:09:49.715 DEBUG:boto:auth_path=/hbmuog-61/obj145 2023-10-17T19:09:49.715 DEBUG:boto:Method: GET 2023-10-17T19:09:49.716 DEBUG:boto:Path: /hbmuog-61/obj145 2023-10-17T19:09:49.716 DEBUG:boto:Data: 2023-10-17T19:09:49.717 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.718 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.718 DEBUG:boto:Params: {} 2023-10-17T19:09:49.719 DEBUG:boto:Token: None 2023-10-17T19:09:49.719 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj145 2023-10-17T19:09:49.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iag17PzeXjKyma5jttwWx+U1MKQ= 2023-10-17T19:09:49.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iag17PzeXjKyma5jttwWx+U1MKQ=', 'Content-Length': '0'} 2023-10-17T19:09:49.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f0ee00af6f4c276-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.723 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.723 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.724 DEBUG:boto:path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.724 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.725 DEBUG:boto:Method: GET 2023-10-17T19:09:49.725 DEBUG:boto:Path: /hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.725 DEBUG:boto:Data: 2023-10-17T19:09:49.726 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.727 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.727 DEBUG:boto:Params: {} 2023-10-17T19:09:49.728 DEBUG:boto:Token: None 2023-10-17T19:09:49.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.729 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116169b7df420aaa-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.732 DEBUG:boto:b'hbmuog-61obj1451000falseobj1452023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.733 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.733 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.734 DEBUG:boto:path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.734 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.735 DEBUG:boto:Method: GET 2023-10-17T19:09:49.735 DEBUG:boto:Path: /hbmuog-61/?prefix=obj145 2023-10-17T19:09:49.735 DEBUG:boto:Data: 2023-10-17T19:09:49.736 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.736 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.737 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.737 DEBUG:boto:Params: {} 2023-10-17T19:09:49.737 DEBUG:boto:Token: None 2023-10-17T19:09:49.738 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.738 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f441ea5e565925-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.743 DEBUG:boto:b'hbmuog-61obj1451000falseobj1452023-10-17T19:06:21.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-17T19:09:49.746 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.746 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.747 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.747 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.748 DEBUG:boto:Method: GET 2023-10-17T19:09:49.748 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.749 DEBUG:boto:Data: 2023-10-17T19:09:49.749 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.749 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.750 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.750 DEBUG:boto:Params: {} 2023-10-17T19:09:49.751 DEBUG:boto:Token: None 2023-10-17T19:09:49.751 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y= 2023-10-17T19:09:49.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y=', 'Content-Length': '0'} 2023-10-17T19:09:49.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000249424fa61c1d79-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.755 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.755 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.756 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.756 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.757 DEBUG:boto:Method: GET 2023-10-17T19:09:49.757 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.758 DEBUG:boto:Data: 2023-10-17T19:09:49.758 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.759 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.759 DEBUG:boto:Params: {} 2023-10-17T19:09:49.760 DEBUG:boto:Token: None 2023-10-17T19:09:49.760 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.761 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y= 2023-10-17T19:09:49.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y=', 'Content-Length': '0'} 2023-10-17T19:09:49.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b564abbcf2fe22e2-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.764 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.765 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.765 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.765 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.766 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.766 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.767 DEBUG:boto:Data: 2023-10-17T19:09:49.767 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.768 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.768 DEBUG:boto:Params: {} 2023-10-17T19:09:49.769 DEBUG:boto:Token: None 2023-10-17T19:09:49.769 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.770 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XgdrPqXk9pTOBcoGkzW5CGeR1eA= 2023-10-17T19:09:49.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XgdrPqXk9pTOBcoGkzW5CGeR1eA=', 'Content-Length': '0'} 2023-10-17T19:09:49.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c54c934939b459-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.773 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.773 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.774 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.774 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.775 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.775 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.775 DEBUG:boto:Data: 2023-10-17T19:09:49.776 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.776 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.777 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.777 DEBUG:boto:Params: {} 2023-10-17T19:09:49.777 DEBUG:boto:Token: None 2023-10-17T19:09:49.778 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.778 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XgdrPqXk9pTOBcoGkzW5CGeR1eA= 2023-10-17T19:09:49.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XgdrPqXk9pTOBcoGkzW5CGeR1eA=', 'Content-Length': '0'} 2023-10-17T19:09:49.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002469070744f545aa-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-17T19:09:49.782 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.782 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.783 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.783 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.784 DEBUG:boto:Method: GET 2023-10-17T19:09:49.784 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.784 DEBUG:boto:Data: 2023-10-17T19:09:49.785 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.785 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.786 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.786 DEBUG:boto:Params: {} 2023-10-17T19:09:49.787 DEBUG:boto:Token: None 2023-10-17T19:09:49.787 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.788 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y= 2023-10-17T19:09:49.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y=', 'Content-Length': '0'} 2023-10-17T19:09:49.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ddd9b8068cc962-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.834 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.835 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.836 DEBUG:boto:path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.836 DEBUG:boto:auth_path=/hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.837 DEBUG:boto:Method: GET 2023-10-17T19:09:49.837 DEBUG:boto:Path: /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.838 DEBUG:boto:Data: 2023-10-17T19:09:49.838 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.839 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.839 DEBUG:boto:Params: {} 2023-10-17T19:09:49.840 DEBUG:boto:Token: None 2023-10-17T19:09:49.841 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148?versionId=null 2023-10-17T19:09:49.842 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y= 2023-10-17T19:09:49.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JCKgBqR3erwBUEa94zR1f6F03+Y=', 'Content-Length': '0'} 2023-10-17T19:09:49.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a92b9bcc3ae70d-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.845 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.846 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.846 DEBUG:boto:path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.847 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.847 DEBUG:boto:Method: GET 2023-10-17T19:09:49.848 DEBUG:boto:Path: /hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.848 DEBUG:boto:Data: 2023-10-17T19:09:49.849 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.849 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.850 DEBUG:boto:Params: {} 2023-10-17T19:09:49.851 DEBUG:boto:Token: None 2023-10-17T19:09:49.851 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.852 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e0ad8bbfc0d68d-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.855 DEBUG:boto:b'hbmuog-61obj1481000falseobj1482023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.856 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.856 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.857 DEBUG:boto:path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.857 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.858 DEBUG:boto:Method: GET 2023-10-17T19:09:49.858 DEBUG:boto:Path: /hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.859 DEBUG:boto:Data: 2023-10-17T19:09:49.859 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.860 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.860 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.861 DEBUG:boto:Params: {} 2023-10-17T19:09:49.861 DEBUG:boto:Token: None 2023-10-17T19:09:49.862 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.863 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099215a28d4c97377-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.868 DEBUG:boto:b'hbmuog-61obj1481000falseobj1482023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.868 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.869 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.869 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.870 DEBUG:boto:Path: /hbmuog-61/obj148 2023-10-17T19:09:49.870 DEBUG:boto:Data: 2023-10-17T19:09:49.871 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.871 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.872 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.872 DEBUG:boto:Params: {} 2023-10-17T19:09:49.873 DEBUG:boto:Token: None 2023-10-17T19:09:49.873 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148 2023-10-17T19:09:49.874 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gsqh/dLQB9R4Hww0w6j4mtp+kfk= 2023-10-17T19:09:49.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gsqh/dLQB9R4Hww0w6j4mtp+kfk=', 'Content-Length': '0'} 2023-10-17T19:09:49.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5948c41a0b96a96-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.877 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.877 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.878 DEBUG:boto:Method: HEAD 2023-10-17T19:09:49.878 DEBUG:boto:Path: /hbmuog-61/obj148 2023-10-17T19:09:49.879 DEBUG:boto:Data: 2023-10-17T19:09:49.879 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.880 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.880 DEBUG:boto:Params: {} 2023-10-17T19:09:49.881 DEBUG:boto:Token: None 2023-10-17T19:09:49.881 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148 2023-10-17T19:09:49.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gsqh/dLQB9R4Hww0w6j4mtp+kfk= 2023-10-17T19:09:49.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gsqh/dLQB9R4Hww0w6j4mtp+kfk=', 'Content-Length': '0'} 2023-10-17T19:09:49.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0696083aadc96c6-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-17T19:09:49.885 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.886 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.886 DEBUG:boto:Method: GET 2023-10-17T19:09:49.887 DEBUG:boto:Path: /hbmuog-61/obj148 2023-10-17T19:09:49.887 DEBUG:boto:Data: 2023-10-17T19:09:49.887 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.888 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.888 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.889 DEBUG:boto:Params: {} 2023-10-17T19:09:49.889 DEBUG:boto:Token: None 2023-10-17T19:09:49.890 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148 2023-10-17T19:09:49.890 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kAV+g+o111qjRWYqYeFDXe8nNzA= 2023-10-17T19:09:49.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kAV+g+o111qjRWYqYeFDXe8nNzA=', 'Content-Length': '0'} 2023-10-17T19:09:49.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e86241cb759a27-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.893 DEBUG:boto:path=/hbmuog-61/obj148 2023-10-17T19:09:49.893 DEBUG:boto:auth_path=/hbmuog-61/obj148 2023-10-17T19:09:49.894 DEBUG:boto:Method: GET 2023-10-17T19:09:49.894 DEBUG:boto:Path: /hbmuog-61/obj148 2023-10-17T19:09:49.895 DEBUG:boto:Data: 2023-10-17T19:09:49.895 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.896 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.896 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.897 DEBUG:boto:Params: {} 2023-10-17T19:09:49.897 DEBUG:boto:Token: None 2023-10-17T19:09:49.898 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj148 2023-10-17T19:09:49.898 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kAV+g+o111qjRWYqYeFDXe8nNzA= 2023-10-17T19:09:49.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kAV+g+o111qjRWYqYeFDXe8nNzA=', 'Content-Length': '0'} 2023-10-17T19:09:49.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008265b83c3eb046a3-00652edbfd-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.945 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.946 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.946 DEBUG:boto:path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.947 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.947 DEBUG:boto:Method: GET 2023-10-17T19:09:49.948 DEBUG:boto:Path: /hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.948 DEBUG:boto:Data: 2023-10-17T19:09:49.948 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.949 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.949 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.950 DEBUG:boto:Params: {} 2023-10-17T19:09:49.950 DEBUG:boto:Token: None 2023-10-17T19:09:49.951 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.951 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e5f039ca672ff4-00652edbfd-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.955 DEBUG:boto:b'hbmuog-61obj1481000falseobj1482023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.956 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:49.956 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:49.956 DEBUG:boto:path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.957 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.957 DEBUG:boto:Method: GET 2023-10-17T19:09:49.958 DEBUG:boto:Path: /hbmuog-61/?prefix=obj148 2023-10-17T19:09:49.958 DEBUG:boto:Data: 2023-10-17T19:09:49.959 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.959 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:49.960 DEBUG:boto:Port: 8000 2023-10-17T19:09:49.960 DEBUG:boto:Params: {} 2023-10-17T19:09:49.960 DEBUG:boto:Token: None 2023-10-17T19:09:49.961 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/ 2023-10-17T19:09:49.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw= 2023-10-17T19:09:49.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9WKGyXdo5e05mO4VmORLMu1Cymw=', 'Content-Length': '0'} 2023-10-17T19:09:49.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc9c8beea0f7b04-00652edbfd-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:49.967 DEBUG:boto:b'hbmuog-61obj1481000falseobj1482023-10-17T19:06:21.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:49.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-17T19:09:49.968 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:49.968 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:49.969 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:49.969 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:49.970 DEBUG:boto:Method: GET 2023-10-17T19:09:49.970 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:49.971 DEBUG:boto:Data: 2023-10-17T19:09:49.971 DEBUG:boto:Headers: {} 2023-10-17T19:09:49.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:49.972 DEBUG:boto:Port: 8001 2023-10-17T19:09:49.973 DEBUG:boto:Params: {} 2023-10-17T19:09:49.973 DEBUG:boto:Token: None 2023-10-17T19:09:49.974 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:49 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:49.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zEEr8KJoU47tpUj/r/fU8KrUkLU= 2023-10-17T19:09:49.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:49 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zEEr8KJoU47tpUj/r/fU8KrUkLU=', 'Content-Length': '0'} 2023-10-17T19:09:49.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de05fc8e3dfb60eb-00652edbfd-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.018 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.019 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.019 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.020 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.020 DEBUG:boto:Method: GET 2023-10-17T19:09:50.021 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.021 DEBUG:boto:Data: 2023-10-17T19:09:50.021 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.022 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.023 DEBUG:boto:Params: {} 2023-10-17T19:09:50.023 DEBUG:boto:Token: None 2023-10-17T19:09:50.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY= 2023-10-17T19:09:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY=', 'Content-Length': '0'} 2023-10-17T19:09:50.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000612342bd1f59ad83-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.027 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.028 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.028 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.029 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.029 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.029 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.030 DEBUG:boto:Data: 2023-10-17T19:09:50.030 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.031 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.031 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.032 DEBUG:boto:Params: {} 2023-10-17T19:09:50.032 DEBUG:boto:Token: None 2023-10-17T19:09:50.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Uby4bcfYNxcy3U/uJgerBG1Wems= 2023-10-17T19:09:50.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uby4bcfYNxcy3U/uJgerBG1Wems=', 'Content-Length': '0'} 2023-10-17T19:09:50.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786191aab9c06027-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.036 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.036 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.037 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.037 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.038 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.038 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.039 DEBUG:boto:Data: 2023-10-17T19:09:50.039 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.040 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.040 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.040 DEBUG:boto:Params: {} 2023-10-17T19:09:50.041 DEBUG:boto:Token: None 2023-10-17T19:09:50.041 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.042 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Uby4bcfYNxcy3U/uJgerBG1Wems= 2023-10-17T19:09:50.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uby4bcfYNxcy3U/uJgerBG1Wems=', 'Content-Length': '0'} 2023-10-17T19:09:50.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f42b1f22a57efe8-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-17T19:09:50.045 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.046 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.046 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.047 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.047 DEBUG:boto:Method: GET 2023-10-17T19:09:50.047 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.048 DEBUG:boto:Data: 2023-10-17T19:09:50.048 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.049 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.049 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.050 DEBUG:boto:Params: {} 2023-10-17T19:09:50.050 DEBUG:boto:Token: None 2023-10-17T19:09:50.051 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.051 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY= 2023-10-17T19:09:50.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY=', 'Content-Length': '0'} 2023-10-17T19:09:50.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a95d90a4d612b7f6-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.054 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.054 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.055 DEBUG:boto:path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.055 DEBUG:boto:auth_path=/hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.056 DEBUG:boto:Method: GET 2023-10-17T19:09:50.056 DEBUG:boto:Path: /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.057 DEBUG:boto:Data: 2023-10-17T19:09:50.057 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.058 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.058 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.059 DEBUG:boto:Params: {} 2023-10-17T19:09:50.059 DEBUG:boto:Token: None 2023-10-17T19:09:50.060 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155?versionId=null 2023-10-17T19:09:50.060 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY= 2023-10-17T19:09:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XypxKCiizzw3torYsAlLHGlaYLY=', 'Content-Length': '0'} 2023-10-17T19:09:50.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4033d82916c44a5-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.063 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.064 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.064 DEBUG:boto:path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.064 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.065 DEBUG:boto:Method: GET 2023-10-17T19:09:50.065 DEBUG:boto:Path: /hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.066 DEBUG:boto:Data: 2023-10-17T19:09:50.066 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.067 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.067 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.067 DEBUG:boto:Params: {} 2023-10-17T19:09:50.068 DEBUG:boto:Token: None 2023-10-17T19:09:50.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.069 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004921535ab14ab944-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.072 DEBUG:boto:b'hbmuog-61obj1551000falseobj1552023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.073 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.074 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.074 DEBUG:boto:path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.075 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.075 DEBUG:boto:Method: GET 2023-10-17T19:09:50.075 DEBUG:boto:Path: /hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.076 DEBUG:boto:Data: 2023-10-17T19:09:50.076 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.077 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.077 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.078 DEBUG:boto:Params: {} 2023-10-17T19:09:50.078 DEBUG:boto:Token: None 2023-10-17T19:09:50.079 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053dd0903d98303c6-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.126 DEBUG:boto:b'hbmuog-61obj1551000falseobj1552023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.127 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.128 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.128 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.129 DEBUG:boto:Path: /hbmuog-61/obj155 2023-10-17T19:09:50.129 DEBUG:boto:Data: 2023-10-17T19:09:50.129 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.130 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.131 DEBUG:boto:Params: {} 2023-10-17T19:09:50.131 DEBUG:boto:Token: None 2023-10-17T19:09:50.132 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155 2023-10-17T19:09:50.132 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IemY/w10R/oNaMiY7UN5j1znuR4= 2023-10-17T19:09:50.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IemY/w10R/oNaMiY7UN5j1znuR4=', 'Content-Length': '0'} 2023-10-17T19:09:50.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e624831d35c3e55-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.135 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.136 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.136 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.136 DEBUG:boto:Path: /hbmuog-61/obj155 2023-10-17T19:09:50.137 DEBUG:boto:Data: 2023-10-17T19:09:50.137 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.138 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.139 DEBUG:boto:Params: {} 2023-10-17T19:09:50.139 DEBUG:boto:Token: None 2023-10-17T19:09:50.140 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155 2023-10-17T19:09:50.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IemY/w10R/oNaMiY7UN5j1znuR4= 2023-10-17T19:09:50.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IemY/w10R/oNaMiY7UN5j1znuR4=', 'Content-Length': '0'} 2023-10-17T19:09:50.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd104a60fcc8f8c0-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-17T19:09:50.143 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.144 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.144 DEBUG:boto:Method: GET 2023-10-17T19:09:50.145 DEBUG:boto:Path: /hbmuog-61/obj155 2023-10-17T19:09:50.145 DEBUG:boto:Data: 2023-10-17T19:09:50.146 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.146 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.149 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.149 DEBUG:boto:Params: {} 2023-10-17T19:09:50.150 DEBUG:boto:Token: None 2023-10-17T19:09:50.150 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155 2023-10-17T19:09:50.151 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:snG9ShyBRmkQ0Ahz+UiT3sLZ6IM= 2023-10-17T19:09:50.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:snG9ShyBRmkQ0Ahz+UiT3sLZ6IM=', 'Content-Length': '0'} 2023-10-17T19:09:50.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29186fd1627407c-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.194 DEBUG:boto:path=/hbmuog-61/obj155 2023-10-17T19:09:50.195 DEBUG:boto:auth_path=/hbmuog-61/obj155 2023-10-17T19:09:50.195 DEBUG:boto:Method: GET 2023-10-17T19:09:50.196 DEBUG:boto:Path: /hbmuog-61/obj155 2023-10-17T19:09:50.196 DEBUG:boto:Data: 2023-10-17T19:09:50.197 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.198 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.198 DEBUG:boto:Params: {} 2023-10-17T19:09:50.199 DEBUG:boto:Token: None 2023-10-17T19:09:50.199 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj155 2023-10-17T19:09:50.200 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:snG9ShyBRmkQ0Ahz+UiT3sLZ6IM= 2023-10-17T19:09:50.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:snG9ShyBRmkQ0Ahz+UiT3sLZ6IM=', 'Content-Length': '0'} 2023-10-17T19:09:50.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe1f1cdd60ebf16-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.203 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.204 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.204 DEBUG:boto:path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.204 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.205 DEBUG:boto:Method: GET 2023-10-17T19:09:50.205 DEBUG:boto:Path: /hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.206 DEBUG:boto:Data: 2023-10-17T19:09:50.206 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.207 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.207 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.208 DEBUG:boto:Params: {} 2023-10-17T19:09:50.208 DEBUG:boto:Token: None 2023-10-17T19:09:50.209 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000248aaea38774ce8a-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.213 DEBUG:boto:b'hbmuog-61obj1551000falseobj1552023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.213 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.214 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.214 DEBUG:boto:path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.215 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.215 DEBUG:boto:Method: GET 2023-10-17T19:09:50.215 DEBUG:boto:Path: /hbmuog-61/?prefix=obj155 2023-10-17T19:09:50.216 DEBUG:boto:Data: 2023-10-17T19:09:50.216 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.217 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.218 DEBUG:boto:Params: {} 2023-10-17T19:09:50.218 DEBUG:boto:Token: None 2023-10-17T19:09:50.219 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a4da117b3e587c-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.224 DEBUG:boto:b'hbmuog-61obj1551000falseobj1552023-10-17T19:06:21.496Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T19:09:50.225 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.226 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.226 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.227 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.227 DEBUG:boto:Method: GET 2023-10-17T19:09:50.228 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.228 DEBUG:boto:Data: 2023-10-17T19:09:50.229 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.229 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.229 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.230 DEBUG:boto:Params: {} 2023-10-17T19:09:50.230 DEBUG:boto:Token: None 2023-10-17T19:09:50.231 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.231 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs= 2023-10-17T19:09:50.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs=', 'Content-Length': '0'} 2023-10-17T19:09:50.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081494ac660753f63-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.234 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.235 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.235 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.235 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.236 DEBUG:boto:Method: GET 2023-10-17T19:09:50.236 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.237 DEBUG:boto:Data: 2023-10-17T19:09:50.237 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.237 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.238 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.238 DEBUG:boto:Params: {} 2023-10-17T19:09:50.239 DEBUG:boto:Token: None 2023-10-17T19:09:50.239 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs= 2023-10-17T19:09:50.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs=', 'Content-Length': '0'} 2023-10-17T19:09:50.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bb2c8f77eb1bae-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.282 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.283 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.283 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.283 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.284 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.284 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.284 DEBUG:boto:Data: 2023-10-17T19:09:50.285 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.285 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.286 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.286 DEBUG:boto:Params: {} 2023-10-17T19:09:50.286 DEBUG:boto:Token: None 2023-10-17T19:09:50.287 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.287 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AZLki1T6yO3IrTgp5GUbN7DTDeU= 2023-10-17T19:09:50.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AZLki1T6yO3IrTgp5GUbN7DTDeU=', 'Content-Length': '0'} 2023-10-17T19:09:50.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e473df7bd4de88aa-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.290 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.291 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.291 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.291 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.292 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.292 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.293 DEBUG:boto:Data: 2023-10-17T19:09:50.293 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.293 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.294 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.294 DEBUG:boto:Params: {} 2023-10-17T19:09:50.295 DEBUG:boto:Token: None 2023-10-17T19:09:50.295 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AZLki1T6yO3IrTgp5GUbN7DTDeU= 2023-10-17T19:09:50.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AZLki1T6yO3IrTgp5GUbN7DTDeU=', 'Content-Length': '0'} 2023-10-17T19:09:50.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002698ccb958ee835c-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T19:09:50.299 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.299 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.299 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.300 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.300 DEBUG:boto:Method: GET 2023-10-17T19:09:50.301 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.301 DEBUG:boto:Data: 2023-10-17T19:09:50.301 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.302 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.303 DEBUG:boto:Params: {} 2023-10-17T19:09:50.303 DEBUG:boto:Token: None 2023-10-17T19:09:50.304 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs= 2023-10-17T19:09:50.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs=', 'Content-Length': '0'} 2023-10-17T19:09:50.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b5ed2326f02332-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.307 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.308 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.308 DEBUG:boto:path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.308 DEBUG:boto:auth_path=/hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.309 DEBUG:boto:Method: GET 2023-10-17T19:09:50.309 DEBUG:boto:Path: /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.310 DEBUG:boto:Data: 2023-10-17T19:09:50.310 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.311 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.311 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.311 DEBUG:boto:Params: {} 2023-10-17T19:09:50.312 DEBUG:boto:Token: None 2023-10-17T19:09:50.312 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17?versionId=null 2023-10-17T19:09:50.313 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs= 2023-10-17T19:09:50.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EiEtfxy/wWLWFojvlH5OnarvtIs=', 'Content-Length': '0'} 2023-10-17T19:09:50.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e887349a69ae25-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.316 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.316 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.317 DEBUG:boto:path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.317 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.317 DEBUG:boto:Method: GET 2023-10-17T19:09:50.318 DEBUG:boto:Path: /hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.318 DEBUG:boto:Data: 2023-10-17T19:09:50.319 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.319 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.320 DEBUG:boto:Params: {} 2023-10-17T19:09:50.320 DEBUG:boto:Token: None 2023-10-17T19:09:50.321 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73fed578ef22121-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.324 DEBUG:boto:b'hbmuog-61obj171000falseobj172023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.325 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.326 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.326 DEBUG:boto:path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.326 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.327 DEBUG:boto:Method: GET 2023-10-17T19:09:50.327 DEBUG:boto:Path: /hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.328 DEBUG:boto:Data: 2023-10-17T19:09:50.328 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.328 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.329 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.329 DEBUG:boto:Params: {} 2023-10-17T19:09:50.330 DEBUG:boto:Token: None 2023-10-17T19:09:50.330 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.331 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483a36ecaaf0368b-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.335 DEBUG:boto:b'hbmuog-61obj171000falseobj172023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.336 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.336 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.337 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.337 DEBUG:boto:Path: /hbmuog-61/obj17 2023-10-17T19:09:50.338 DEBUG:boto:Data: 2023-10-17T19:09:50.338 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.338 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.339 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.339 DEBUG:boto:Params: {} 2023-10-17T19:09:50.340 DEBUG:boto:Token: None 2023-10-17T19:09:50.340 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17 2023-10-17T19:09:50.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:06rdxt3fQiMZPNoQsYxGHs9lYuw= 2023-10-17T19:09:50.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:06rdxt3fQiMZPNoQsYxGHs9lYuw=', 'Content-Length': '0'} 2023-10-17T19:09:50.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b07d43291f208e-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.343 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.343 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.344 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.344 DEBUG:boto:Path: /hbmuog-61/obj17 2023-10-17T19:09:50.345 DEBUG:boto:Data: 2023-10-17T19:09:50.345 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.346 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.346 DEBUG:boto:Params: {} 2023-10-17T19:09:50.347 DEBUG:boto:Token: None 2023-10-17T19:09:50.347 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17 2023-10-17T19:09:50.348 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:06rdxt3fQiMZPNoQsYxGHs9lYuw= 2023-10-17T19:09:50.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:06rdxt3fQiMZPNoQsYxGHs9lYuw=', 'Content-Length': '0'} 2023-10-17T19:09:50.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea9ecfdf168425d-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-17T19:09:50.351 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.351 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.352 DEBUG:boto:Method: GET 2023-10-17T19:09:50.353 DEBUG:boto:Path: /hbmuog-61/obj17 2023-10-17T19:09:50.354 DEBUG:boto:Data: 2023-10-17T19:09:50.354 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.354 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.355 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.355 DEBUG:boto:Params: {} 2023-10-17T19:09:50.356 DEBUG:boto:Token: None 2023-10-17T19:09:50.356 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17 2023-10-17T19:09:50.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YXzTGLbgnvEPc+xtiXG2FNU453c= 2023-10-17T19:09:50.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YXzTGLbgnvEPc+xtiXG2FNU453c=', 'Content-Length': '0'} 2023-10-17T19:09:50.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda2c3b31b343c18-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.402 DEBUG:boto:path=/hbmuog-61/obj17 2023-10-17T19:09:50.403 DEBUG:boto:auth_path=/hbmuog-61/obj17 2023-10-17T19:09:50.403 DEBUG:boto:Method: GET 2023-10-17T19:09:50.404 DEBUG:boto:Path: /hbmuog-61/obj17 2023-10-17T19:09:50.404 DEBUG:boto:Data: 2023-10-17T19:09:50.405 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.405 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.405 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.406 DEBUG:boto:Params: {} 2023-10-17T19:09:50.407 DEBUG:boto:Token: None 2023-10-17T19:09:50.407 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj17 2023-10-17T19:09:50.407 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YXzTGLbgnvEPc+xtiXG2FNU453c= 2023-10-17T19:09:50.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YXzTGLbgnvEPc+xtiXG2FNU453c=', 'Content-Length': '0'} 2023-10-17T19:09:50.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e104e0e58a822bc9-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.410 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.411 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.411 DEBUG:boto:path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.411 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.412 DEBUG:boto:Method: GET 2023-10-17T19:09:50.412 DEBUG:boto:Path: /hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.413 DEBUG:boto:Data: 2023-10-17T19:09:50.413 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.414 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.414 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.414 DEBUG:boto:Params: {} 2023-10-17T19:09:50.415 DEBUG:boto:Token: None 2023-10-17T19:09:50.415 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001abef67f7356dc7-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.419 DEBUG:boto:b'hbmuog-61obj171000falseobj172023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.420 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.420 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.421 DEBUG:boto:path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.422 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.422 DEBUG:boto:Method: GET 2023-10-17T19:09:50.422 DEBUG:boto:Path: /hbmuog-61/?prefix=obj17 2023-10-17T19:09:50.423 DEBUG:boto:Data: 2023-10-17T19:09:50.423 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.424 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.424 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.424 DEBUG:boto:Params: {} 2023-10-17T19:09:50.425 DEBUG:boto:Token: None 2023-10-17T19:09:50.425 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.426 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff77141247b51d06-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.430 DEBUG:boto:b'hbmuog-61obj171000falseobj172023-10-17T19:06:19.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-17T19:09:50.431 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.432 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.432 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.432 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.433 DEBUG:boto:Method: GET 2023-10-17T19:09:50.433 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.434 DEBUG:boto:Data: 2023-10-17T19:09:50.434 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.434 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.435 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.435 DEBUG:boto:Params: {} 2023-10-17T19:09:50.436 DEBUG:boto:Token: None 2023-10-17T19:09:50.436 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk= 2023-10-17T19:09:50.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk=', 'Content-Length': '0'} 2023-10-17T19:09:50.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd77a0d17c1b6e8-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.440 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.440 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.441 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.441 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.441 DEBUG:boto:Method: GET 2023-10-17T19:09:50.442 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.442 DEBUG:boto:Data: 2023-10-17T19:09:50.443 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.443 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.444 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.444 DEBUG:boto:Params: {} 2023-10-17T19:09:50.444 DEBUG:boto:Token: None 2023-10-17T19:09:50.445 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk= 2023-10-17T19:09:50.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk=', 'Content-Length': '0'} 2023-10-17T19:09:50.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bae8e8d939f805-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.448 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.448 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.449 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.449 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.450 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.450 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.450 DEBUG:boto:Data: 2023-10-17T19:09:50.451 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.452 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.452 DEBUG:boto:Params: {} 2023-10-17T19:09:50.452 DEBUG:boto:Token: None 2023-10-17T19:09:50.453 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SFEbIFHr3tHt0DcPQwCxTFC9vR0= 2023-10-17T19:09:50.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SFEbIFHr3tHt0DcPQwCxTFC9vR0=', 'Content-Length': '0'} 2023-10-17T19:09:50.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094238b7782a6da89-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.456 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.457 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.457 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.457 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.458 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.458 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.458 DEBUG:boto:Data: 2023-10-17T19:09:50.459 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.459 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.460 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.460 DEBUG:boto:Params: {} 2023-10-17T19:09:50.460 DEBUG:boto:Token: None 2023-10-17T19:09:50.461 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.461 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SFEbIFHr3tHt0DcPQwCxTFC9vR0= 2023-10-17T19:09:50.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SFEbIFHr3tHt0DcPQwCxTFC9vR0=', 'Content-Length': '0'} 2023-10-17T19:09:50.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3dbfeead10d5abb-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-17T19:09:50.464 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.464 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.465 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.465 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.466 DEBUG:boto:Method: GET 2023-10-17T19:09:50.466 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.466 DEBUG:boto:Data: 2023-10-17T19:09:50.467 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.468 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.468 DEBUG:boto:Params: {} 2023-10-17T19:09:50.468 DEBUG:boto:Token: None 2023-10-17T19:09:50.469 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk= 2023-10-17T19:09:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk=', 'Content-Length': '0'} 2023-10-17T19:09:50.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001582b9739cb00d99-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.514 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.515 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.515 DEBUG:boto:path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.516 DEBUG:boto:auth_path=/hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.516 DEBUG:boto:Method: GET 2023-10-17T19:09:50.517 DEBUG:boto:Path: /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.517 DEBUG:boto:Data: 2023-10-17T19:09:50.518 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.519 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.519 DEBUG:boto:Params: {} 2023-10-17T19:09:50.520 DEBUG:boto:Token: None 2023-10-17T19:09:50.521 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185?versionId=null 2023-10-17T19:09:50.521 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk= 2023-10-17T19:09:50.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ps+KciH0TwvsZWxTjizUhr8eUgk=', 'Content-Length': '0'} 2023-10-17T19:09:50.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4578fed7024818-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.525 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.525 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.525 DEBUG:boto:path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.526 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.526 DEBUG:boto:Method: GET 2023-10-17T19:09:50.527 DEBUG:boto:Path: /hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.527 DEBUG:boto:Data: 2023-10-17T19:09:50.528 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.528 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.529 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.529 DEBUG:boto:Params: {} 2023-10-17T19:09:50.530 DEBUG:boto:Token: None 2023-10-17T19:09:50.530 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c403e9f1a9db8ddb-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.534 DEBUG:boto:b'hbmuog-61obj1851000falseobj1852023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.535 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.535 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.536 DEBUG:boto:path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.536 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.537 DEBUG:boto:Method: GET 2023-10-17T19:09:50.537 DEBUG:boto:Path: /hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.538 DEBUG:boto:Data: 2023-10-17T19:09:50.538 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.539 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.540 DEBUG:boto:Params: {} 2023-10-17T19:09:50.540 DEBUG:boto:Token: None 2023-10-17T19:09:50.541 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5adea86a34bf9a2-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.546 DEBUG:boto:b'hbmuog-61obj1851000falseobj1852023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.547 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.547 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.548 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.548 DEBUG:boto:Path: /hbmuog-61/obj185 2023-10-17T19:09:50.549 DEBUG:boto:Data: 2023-10-17T19:09:50.549 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.550 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.550 DEBUG:boto:Params: {} 2023-10-17T19:09:50.551 DEBUG:boto:Token: None 2023-10-17T19:09:50.551 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185 2023-10-17T19:09:50.552 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NnV41nKeroh8NcyD0GaWvwcb12w= 2023-10-17T19:09:50.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NnV41nKeroh8NcyD0GaWvwcb12w=', 'Content-Length': '0'} 2023-10-17T19:09:50.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e954e1cfe59d49-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.555 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.555 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.556 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.556 DEBUG:boto:Path: /hbmuog-61/obj185 2023-10-17T19:09:50.557 DEBUG:boto:Data: 2023-10-17T19:09:50.557 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.558 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.558 DEBUG:boto:Params: {} 2023-10-17T19:09:50.561 DEBUG:boto:Token: None 2023-10-17T19:09:50.561 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185 2023-10-17T19:09:50.562 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NnV41nKeroh8NcyD0GaWvwcb12w= 2023-10-17T19:09:50.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NnV41nKeroh8NcyD0GaWvwcb12w=', 'Content-Length': '0'} 2023-10-17T19:09:50.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047807b836a5a96b-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-17T19:09:50.565 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.566 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.566 DEBUG:boto:Method: GET 2023-10-17T19:09:50.567 DEBUG:boto:Path: /hbmuog-61/obj185 2023-10-17T19:09:50.567 DEBUG:boto:Data: 2023-10-17T19:09:50.567 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.568 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.569 DEBUG:boto:Params: {} 2023-10-17T19:09:50.569 DEBUG:boto:Token: None 2023-10-17T19:09:50.570 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185 2023-10-17T19:09:50.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZyFfeidAU9lBFEC3y4xRqpiqQFI= 2023-10-17T19:09:50.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZyFfeidAU9lBFEC3y4xRqpiqQFI=', 'Content-Length': '0'} 2023-10-17T19:09:50.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fead597c55ab1c-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.573 DEBUG:boto:path=/hbmuog-61/obj185 2023-10-17T19:09:50.574 DEBUG:boto:auth_path=/hbmuog-61/obj185 2023-10-17T19:09:50.574 DEBUG:boto:Method: GET 2023-10-17T19:09:50.575 DEBUG:boto:Path: /hbmuog-61/obj185 2023-10-17T19:09:50.575 DEBUG:boto:Data: 2023-10-17T19:09:50.576 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.577 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.577 DEBUG:boto:Params: {} 2023-10-17T19:09:50.578 DEBUG:boto:Token: None 2023-10-17T19:09:50.578 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj185 2023-10-17T19:09:50.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZyFfeidAU9lBFEC3y4xRqpiqQFI= 2023-10-17T19:09:50.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZyFfeidAU9lBFEC3y4xRqpiqQFI=', 'Content-Length': '0'} 2023-10-17T19:09:50.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab9297399bcc501-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.582 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.583 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.583 DEBUG:boto:path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.584 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.584 DEBUG:boto:Method: GET 2023-10-17T19:09:50.585 DEBUG:boto:Path: /hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.585 DEBUG:boto:Data: 2023-10-17T19:09:50.585 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.586 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.586 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.587 DEBUG:boto:Params: {} 2023-10-17T19:09:50.587 DEBUG:boto:Token: None 2023-10-17T19:09:50.588 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.588 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1af809b8bbef0a-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.592 DEBUG:boto:b'hbmuog-61obj1851000falseobj1852023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.593 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.593 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.594 DEBUG:boto:path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.594 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.595 DEBUG:boto:Method: GET 2023-10-17T19:09:50.595 DEBUG:boto:Path: /hbmuog-61/?prefix=obj185 2023-10-17T19:09:50.596 DEBUG:boto:Data: 2023-10-17T19:09:50.596 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.597 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.597 DEBUG:boto:Params: {} 2023-10-17T19:09:50.598 DEBUG:boto:Token: None 2023-10-17T19:09:50.598 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.599 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4597acacbb6da65-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.604 DEBUG:boto:b'hbmuog-61obj1851000falseobj1852023-10-17T19:06:21.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-17T19:09:50.605 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.606 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.606 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.607 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.607 DEBUG:boto:Method: GET 2023-10-17T19:09:50.608 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.608 DEBUG:boto:Data: 2023-10-17T19:09:50.609 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.609 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.610 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.610 DEBUG:boto:Params: {} 2023-10-17T19:09:50.611 DEBUG:boto:Token: None 2023-10-17T19:09:50.611 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ= 2023-10-17T19:09:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ=', 'Content-Length': '0'} 2023-10-17T19:09:50.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b092d1ee2bf3b86-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.615 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.616 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.616 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.617 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.617 DEBUG:boto:Method: GET 2023-10-17T19:09:50.618 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.618 DEBUG:boto:Data: 2023-10-17T19:09:50.619 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.620 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.620 DEBUG:boto:Params: {} 2023-10-17T19:09:50.620 DEBUG:boto:Token: None 2023-10-17T19:09:50.621 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ= 2023-10-17T19:09:50.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ=', 'Content-Length': '0'} 2023-10-17T19:09:50.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a567a3afda40e1-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.625 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.625 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.626 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.626 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.627 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.627 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.628 DEBUG:boto:Data: 2023-10-17T19:09:50.628 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.628 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.629 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.629 DEBUG:boto:Params: {} 2023-10-17T19:09:50.630 DEBUG:boto:Token: None 2023-10-17T19:09:50.630 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.631 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XIXPIC8WjO4ghLxeiGEna7wIYSk= 2023-10-17T19:09:50.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XIXPIC8WjO4ghLxeiGEna7wIYSk=', 'Content-Length': '0'} 2023-10-17T19:09:50.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c9e6c28975a573-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.634 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.634 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.635 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.635 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.635 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.636 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.636 DEBUG:boto:Data: 2023-10-17T19:09:50.637 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.637 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.638 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.638 DEBUG:boto:Params: {} 2023-10-17T19:09:50.639 DEBUG:boto:Token: None 2023-10-17T19:09:50.639 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.640 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XIXPIC8WjO4ghLxeiGEna7wIYSk= 2023-10-17T19:09:50.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XIXPIC8WjO4ghLxeiGEna7wIYSk=', 'Content-Length': '0'} 2023-10-17T19:09:50.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e35d6b42870897-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-17T19:09:50.643 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.644 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.644 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.645 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.645 DEBUG:boto:Method: GET 2023-10-17T19:09:50.645 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.646 DEBUG:boto:Data: 2023-10-17T19:09:50.646 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.647 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.647 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.648 DEBUG:boto:Params: {} 2023-10-17T19:09:50.648 DEBUG:boto:Token: None 2023-10-17T19:09:50.649 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.649 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ= 2023-10-17T19:09:50.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ=', 'Content-Length': '0'} 2023-10-17T19:09:50.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002928dbaece3a7974-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.694 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.695 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.695 DEBUG:boto:path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.696 DEBUG:boto:auth_path=/hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.696 DEBUG:boto:Method: GET 2023-10-17T19:09:50.697 DEBUG:boto:Path: /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.697 DEBUG:boto:Data: 2023-10-17T19:09:50.698 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.698 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.699 DEBUG:boto:Params: {} 2023-10-17T19:09:50.699 DEBUG:boto:Token: None 2023-10-17T19:09:50.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19?versionId=null 2023-10-17T19:09:50.700 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ= 2023-10-17T19:09:50.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fMTZOoLQZxeSWcEIMhLlpup39PQ=', 'Content-Length': '0'} 2023-10-17T19:09:50.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d348c9af52428471-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.704 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.704 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.705 DEBUG:boto:path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.705 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.706 DEBUG:boto:Method: GET 2023-10-17T19:09:50.706 DEBUG:boto:Path: /hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.707 DEBUG:boto:Data: 2023-10-17T19:09:50.707 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.708 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.708 DEBUG:boto:Params: {} 2023-10-17T19:09:50.709 DEBUG:boto:Token: None 2023-10-17T19:09:50.709 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.710 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f21f85fd9eb13e44-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.713 DEBUG:boto:b'hbmuog-61obj191000falseobj192023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.714 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.715 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.715 DEBUG:boto:path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.716 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.716 DEBUG:boto:Method: GET 2023-10-17T19:09:50.717 DEBUG:boto:Path: /hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.718 DEBUG:boto:Data: 2023-10-17T19:09:50.718 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.720 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.720 DEBUG:boto:Params: {} 2023-10-17T19:09:50.721 DEBUG:boto:Token: None 2023-10-17T19:09:50.722 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.723 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4fee4080079883-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.728 DEBUG:boto:b'hbmuog-61obj191000falseobj192023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.728 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.729 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.729 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.730 DEBUG:boto:Path: /hbmuog-61/obj19 2023-10-17T19:09:50.731 DEBUG:boto:Data: 2023-10-17T19:09:50.731 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.733 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.734 DEBUG:boto:Params: {} 2023-10-17T19:09:50.734 DEBUG:boto:Token: None 2023-10-17T19:09:50.735 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19 2023-10-17T19:09:50.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LhQWOIZ+rK4iT3QpJgzZL4SO/3U= 2023-10-17T19:09:50.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LhQWOIZ+rK4iT3QpJgzZL4SO/3U=', 'Content-Length': '0'} 2023-10-17T19:09:50.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a67dcc107fe5896-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.739 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.739 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.740 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.740 DEBUG:boto:Path: /hbmuog-61/obj19 2023-10-17T19:09:50.741 DEBUG:boto:Data: 2023-10-17T19:09:50.741 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.741 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.742 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.742 DEBUG:boto:Params: {} 2023-10-17T19:09:50.743 DEBUG:boto:Token: None 2023-10-17T19:09:50.743 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19 2023-10-17T19:09:50.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LhQWOIZ+rK4iT3QpJgzZL4SO/3U= 2023-10-17T19:09:50.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LhQWOIZ+rK4iT3QpJgzZL4SO/3U=', 'Content-Length': '0'} 2023-10-17T19:09:50.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc5f574c7d6f4bb-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-17T19:09:50.747 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.747 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.748 DEBUG:boto:Method: GET 2023-10-17T19:09:50.748 DEBUG:boto:Path: /hbmuog-61/obj19 2023-10-17T19:09:50.749 DEBUG:boto:Data: 2023-10-17T19:09:50.749 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.750 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.751 DEBUG:boto:Params: {} 2023-10-17T19:09:50.751 DEBUG:boto:Token: None 2023-10-17T19:09:50.752 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19 2023-10-17T19:09:50.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n6/kyITUZFu7Sr0iD6R3731FgPo= 2023-10-17T19:09:50.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n6/kyITUZFu7Sr0iD6R3731FgPo=', 'Content-Length': '0'} 2023-10-17T19:09:50.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008000fbb581186e8-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.755 DEBUG:boto:path=/hbmuog-61/obj19 2023-10-17T19:09:50.755 DEBUG:boto:auth_path=/hbmuog-61/obj19 2023-10-17T19:09:50.756 DEBUG:boto:Method: GET 2023-10-17T19:09:50.756 DEBUG:boto:Path: /hbmuog-61/obj19 2023-10-17T19:09:50.757 DEBUG:boto:Data: 2023-10-17T19:09:50.757 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.758 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.758 DEBUG:boto:Params: {} 2023-10-17T19:09:50.759 DEBUG:boto:Token: None 2023-10-17T19:09:50.759 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj19 2023-10-17T19:09:50.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:n6/kyITUZFu7Sr0iD6R3731FgPo= 2023-10-17T19:09:50.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:n6/kyITUZFu7Sr0iD6R3731FgPo=', 'Content-Length': '0'} 2023-10-17T19:09:50.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4277042c1e26ed-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.806 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.807 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.808 DEBUG:boto:path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.808 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.808 DEBUG:boto:Method: GET 2023-10-17T19:09:50.809 DEBUG:boto:Path: /hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.809 DEBUG:boto:Data: 2023-10-17T19:09:50.810 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.810 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.811 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.811 DEBUG:boto:Params: {} 2023-10-17T19:09:50.812 DEBUG:boto:Token: None 2023-10-17T19:09:50.812 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.813 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd892741f8bff546-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.816 DEBUG:boto:b'hbmuog-61obj191000falseobj192023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.817 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.818 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.818 DEBUG:boto:path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.819 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.819 DEBUG:boto:Method: GET 2023-10-17T19:09:50.820 DEBUG:boto:Path: /hbmuog-61/?prefix=obj19 2023-10-17T19:09:50.820 DEBUG:boto:Data: 2023-10-17T19:09:50.821 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.821 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.822 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.822 DEBUG:boto:Params: {} 2023-10-17T19:09:50.823 DEBUG:boto:Token: None 2023-10-17T19:09:50.823 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.824 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001311c10818e93985-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.829 DEBUG:boto:b'hbmuog-61obj191000falseobj192023-10-17T19:06:20.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:09:50.831 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.831 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.832 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.832 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.833 DEBUG:boto:Method: GET 2023-10-17T19:09:50.833 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.833 DEBUG:boto:Data: 2023-10-17T19:09:50.834 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.834 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.835 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.835 DEBUG:boto:Params: {} 2023-10-17T19:09:50.836 DEBUG:boto:Token: None 2023-10-17T19:09:50.836 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw= 2023-10-17T19:09:50.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw=', 'Content-Length': '0'} 2023-10-17T19:09:50.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099292e13587db4c8-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.840 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.840 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.841 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.841 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.842 DEBUG:boto:Method: GET 2023-10-17T19:09:50.842 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.843 DEBUG:boto:Data: 2023-10-17T19:09:50.843 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.844 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.844 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.845 DEBUG:boto:Params: {} 2023-10-17T19:09:50.845 DEBUG:boto:Token: None 2023-10-17T19:09:50.846 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.846 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw= 2023-10-17T19:09:50.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw=', 'Content-Length': '0'} 2023-10-17T19:09:50.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eefee0c182c4b08-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.850 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.850 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.850 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.851 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.851 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.852 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.852 DEBUG:boto:Data: 2023-10-17T19:09:50.853 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.853 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.854 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.854 DEBUG:boto:Params: {} 2023-10-17T19:09:50.855 DEBUG:boto:Token: None 2023-10-17T19:09:50.855 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:POqZRRRGYGcLHTEtLVb+nRmsmew= 2023-10-17T19:09:50.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:POqZRRRGYGcLHTEtLVb+nRmsmew=', 'Content-Length': '0'} 2023-10-17T19:09:50.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f79829ca72971f-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.859 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.859 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.860 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.860 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.861 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.861 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.861 DEBUG:boto:Data: 2023-10-17T19:09:50.862 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.862 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.863 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.863 DEBUG:boto:Params: {} 2023-10-17T19:09:50.864 DEBUG:boto:Token: None 2023-10-17T19:09:50.864 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:POqZRRRGYGcLHTEtLVb+nRmsmew= 2023-10-17T19:09:50.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:POqZRRRGYGcLHTEtLVb+nRmsmew=', 'Content-Length': '0'} 2023-10-17T19:09:50.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000807661a4363174b7-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:09:50.869 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.869 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.869 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.870 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.870 DEBUG:boto:Method: GET 2023-10-17T19:09:50.871 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.871 DEBUG:boto:Data: 2023-10-17T19:09:50.872 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.873 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.873 DEBUG:boto:Params: {} 2023-10-17T19:09:50.874 DEBUG:boto:Token: None 2023-10-17T19:09:50.874 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw= 2023-10-17T19:09:50.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw=', 'Content-Length': '0'} 2023-10-17T19:09:50.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f3e0d70e8fbb96-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.878 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.878 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.881 DEBUG:boto:path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.882 DEBUG:boto:auth_path=/hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.882 DEBUG:boto:Method: GET 2023-10-17T19:09:50.883 DEBUG:boto:Path: /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.883 DEBUG:boto:Data: 2023-10-17T19:09:50.884 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.884 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.885 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.885 DEBUG:boto:Params: {} 2023-10-17T19:09:50.886 DEBUG:boto:Token: None 2023-10-17T19:09:50.886 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22?versionId=null 2023-10-17T19:09:50.887 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw= 2023-10-17T19:09:50.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVoBuF2WF9FwzL5/owkc94auwTw=', 'Content-Length': '0'} 2023-10-17T19:09:50.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d391d3b4c7f83e-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.890 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.891 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.891 DEBUG:boto:path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.892 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.892 DEBUG:boto:Method: GET 2023-10-17T19:09:50.893 DEBUG:boto:Path: /hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.893 DEBUG:boto:Data: 2023-10-17T19:09:50.893 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.894 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.895 DEBUG:boto:Params: {} 2023-10-17T19:09:50.895 DEBUG:boto:Token: None 2023-10-17T19:09:50.896 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47639dca9cb5588-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.900 DEBUG:boto:b'hbmuog-61obj221000falseobj222023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.901 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.901 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.902 DEBUG:boto:path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.902 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.903 DEBUG:boto:Method: GET 2023-10-17T19:09:50.903 DEBUG:boto:Path: /hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.903 DEBUG:boto:Data: 2023-10-17T19:09:50.904 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.904 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.905 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.905 DEBUG:boto:Params: {} 2023-10-17T19:09:50.906 DEBUG:boto:Token: None 2023-10-17T19:09:50.906 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.907 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebebd97fe28fdb7-00652edbfe-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.912 DEBUG:boto:b'hbmuog-61obj221000falseobj222023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.913 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.913 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.914 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.914 DEBUG:boto:Path: /hbmuog-61/obj22 2023-10-17T19:09:50.915 DEBUG:boto:Data: 2023-10-17T19:09:50.915 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.916 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.916 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.917 DEBUG:boto:Params: {} 2023-10-17T19:09:50.917 DEBUG:boto:Token: None 2023-10-17T19:09:50.918 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22 2023-10-17T19:09:50.918 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xcqgVZRhkkdaww6krL9npwgJzE8= 2023-10-17T19:09:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xcqgVZRhkkdaww6krL9npwgJzE8=', 'Content-Length': '0'} 2023-10-17T19:09:50.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006feefb4125a89ed8-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.921 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.922 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.922 DEBUG:boto:Method: HEAD 2023-10-17T19:09:50.923 DEBUG:boto:Path: /hbmuog-61/obj22 2023-10-17T19:09:50.923 DEBUG:boto:Data: 2023-10-17T19:09:50.924 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.924 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.925 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.925 DEBUG:boto:Params: {} 2023-10-17T19:09:50.926 DEBUG:boto:Token: None 2023-10-17T19:09:50.926 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22 2023-10-17T19:09:50.927 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xcqgVZRhkkdaww6krL9npwgJzE8= 2023-10-17T19:09:50.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xcqgVZRhkkdaww6krL9npwgJzE8=', 'Content-Length': '0'} 2023-10-17T19:09:50.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62281eace789d89-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:09:50.930 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.931 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.931 DEBUG:boto:Method: GET 2023-10-17T19:09:50.932 DEBUG:boto:Path: /hbmuog-61/obj22 2023-10-17T19:09:50.932 DEBUG:boto:Data: 2023-10-17T19:09:50.933 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.933 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.934 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.934 DEBUG:boto:Params: {} 2023-10-17T19:09:50.935 DEBUG:boto:Token: None 2023-10-17T19:09:50.935 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22 2023-10-17T19:09:50.936 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rJxJKzl3gF+cCnDuUH8WSx/nqUE= 2023-10-17T19:09:50.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rJxJKzl3gF+cCnDuUH8WSx/nqUE=', 'Content-Length': '0'} 2023-10-17T19:09:50.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb3587f66229d8f-00652edbfe-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.978 DEBUG:boto:path=/hbmuog-61/obj22 2023-10-17T19:09:50.979 DEBUG:boto:auth_path=/hbmuog-61/obj22 2023-10-17T19:09:50.980 DEBUG:boto:Method: GET 2023-10-17T19:09:50.980 DEBUG:boto:Path: /hbmuog-61/obj22 2023-10-17T19:09:50.981 DEBUG:boto:Data: 2023-10-17T19:09:50.981 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:50.982 DEBUG:boto:Port: 8000 2023-10-17T19:09:50.983 DEBUG:boto:Params: {} 2023-10-17T19:09:50.983 DEBUG:boto:Token: None 2023-10-17T19:09:50.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/obj22 2023-10-17T19:09:50.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rJxJKzl3gF+cCnDuUH8WSx/nqUE= 2023-10-17T19:09:50.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rJxJKzl3gF+cCnDuUH8WSx/nqUE=', 'Content-Length': '0'} 2023-10-17T19:09:50.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778eda3b28f94a94-00652edbfe-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.987 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.988 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.988 DEBUG:boto:path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.989 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.989 DEBUG:boto:Method: GET 2023-10-17T19:09:50.989 DEBUG:boto:Path: /hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.990 DEBUG:boto:Data: 2023-10-17T19:09:50.990 DEBUG:boto:Headers: {} 2023-10-17T19:09:50.991 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:50.991 DEBUG:boto:Port: 8001 2023-10-17T19:09:50.992 DEBUG:boto:Params: {} 2023-10-17T19:09:50.992 DEBUG:boto:Token: None 2023-10-17T19:09:50.993 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:50 GMT /hbmuog-61/ 2023-10-17T19:09:50.993 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI= 2023-10-17T19:09:50.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Df2S1TgUpi9mSRrwid07OYGoOtI=', 'Content-Length': '0'} 2023-10-17T19:09:50.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe7d7573c371cb4-00652edbfe-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:50.996 DEBUG:boto:b'hbmuog-61obj221000falseobj222023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:50.997 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:50.998 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:50.998 DEBUG:boto:path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.999 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj22 2023-10-17T19:09:50.999 DEBUG:boto:Method: GET 2023-10-17T19:09:50.999 DEBUG:boto:Path: /hbmuog-61/?prefix=obj22 2023-10-17T19:09:51.000 DEBUG:boto:Data: 2023-10-17T19:09:51.000 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.001 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.001 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.001 DEBUG:boto:Params: {} 2023-10-17T19:09:51.002 DEBUG:boto:Token: None 2023-10-17T19:09:51.002 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.003 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000961e3064d78011b8-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.008 DEBUG:boto:b'hbmuog-61obj221000falseobj222023-10-17T19:06:19.679Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-17T19:09:51.009 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.009 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.010 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.010 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.011 DEBUG:boto:Method: GET 2023-10-17T19:09:51.011 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.012 DEBUG:boto:Data: 2023-10-17T19:09:51.012 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.012 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.013 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.013 DEBUG:boto:Params: {} 2023-10-17T19:09:51.014 DEBUG:boto:Token: None 2023-10-17T19:09:51.014 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.015 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds= 2023-10-17T19:09:51.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds=', 'Content-Length': '0'} 2023-10-17T19:09:51.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a74d4b88ea42cc-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.018 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.018 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.019 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.019 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.020 DEBUG:boto:Method: GET 2023-10-17T19:09:51.020 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.021 DEBUG:boto:Data: 2023-10-17T19:09:51.021 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.022 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.022 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.023 DEBUG:boto:Params: {} 2023-10-17T19:09:51.023 DEBUG:boto:Token: None 2023-10-17T19:09:51.024 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.024 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds= 2023-10-17T19:09:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds=', 'Content-Length': '0'} 2023-10-17T19:09:51.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004075e9ec78caaaee-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.027 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.028 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.028 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.028 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.029 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.029 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.030 DEBUG:boto:Data: 2023-10-17T19:09:51.030 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.030 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.031 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.031 DEBUG:boto:Params: {} 2023-10-17T19:09:51.032 DEBUG:boto:Token: None 2023-10-17T19:09:51.032 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.033 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CxgaJE9I7qTdOpCz5BIEdPoxJFQ= 2023-10-17T19:09:51.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxgaJE9I7qTdOpCz5BIEdPoxJFQ=', 'Content-Length': '0'} 2023-10-17T19:09:51.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea013690436217a-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.035 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.036 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.036 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.037 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.037 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.037 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.038 DEBUG:boto:Data: 2023-10-17T19:09:51.038 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.039 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.039 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.039 DEBUG:boto:Params: {} 2023-10-17T19:09:51.040 DEBUG:boto:Token: None 2023-10-17T19:09:51.040 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.041 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CxgaJE9I7qTdOpCz5BIEdPoxJFQ= 2023-10-17T19:09:51.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxgaJE9I7qTdOpCz5BIEdPoxJFQ=', 'Content-Length': '0'} 2023-10-17T19:09:51.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f12ed9deb114daa-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-17T19:09:51.044 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.045 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.045 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.045 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.046 DEBUG:boto:Method: GET 2023-10-17T19:09:51.046 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.047 DEBUG:boto:Data: 2023-10-17T19:09:51.047 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.048 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.048 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.048 DEBUG:boto:Params: {} 2023-10-17T19:09:51.049 DEBUG:boto:Token: None 2023-10-17T19:09:51.049 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.050 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds= 2023-10-17T19:09:51.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds=', 'Content-Length': '0'} 2023-10-17T19:09:51.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdc517973642fbb-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.053 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.054 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.054 DEBUG:boto:path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.054 DEBUG:boto:auth_path=/hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.055 DEBUG:boto:Method: GET 2023-10-17T19:09:51.055 DEBUG:boto:Path: /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.056 DEBUG:boto:Data: 2023-10-17T19:09:51.056 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.057 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.057 DEBUG:boto:Params: {} 2023-10-17T19:09:51.058 DEBUG:boto:Token: None 2023-10-17T19:09:51.058 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23?versionId=null 2023-10-17T19:09:51.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds= 2023-10-17T19:09:51.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lKdiezjXmiWIY9V5Fd1PVE5XNds=', 'Content-Length': '0'} 2023-10-17T19:09:51.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000718f025f818f4ad4-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.102 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.103 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.104 DEBUG:boto:path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.104 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.105 DEBUG:boto:Method: GET 2023-10-17T19:09:51.105 DEBUG:boto:Path: /hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.105 DEBUG:boto:Data: 2023-10-17T19:09:51.106 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.107 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.107 DEBUG:boto:Params: {} 2023-10-17T19:09:51.108 DEBUG:boto:Token: None 2023-10-17T19:09:51.108 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004965ba3345b13d9f-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.112 DEBUG:boto:b'hbmuog-61obj231000falseobj232023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.113 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.113 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.114 DEBUG:boto:path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.114 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.115 DEBUG:boto:Method: GET 2023-10-17T19:09:51.115 DEBUG:boto:Path: /hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.115 DEBUG:boto:Data: 2023-10-17T19:09:51.116 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.116 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.117 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.117 DEBUG:boto:Params: {} 2023-10-17T19:09:51.118 DEBUG:boto:Token: None 2023-10-17T19:09:51.118 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.119 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097c3822cea22cd9-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.124 DEBUG:boto:b'hbmuog-61obj231000falseobj232023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.124 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.125 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.125 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.126 DEBUG:boto:Path: /hbmuog-61/obj23 2023-10-17T19:09:51.126 DEBUG:boto:Data: 2023-10-17T19:09:51.127 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.127 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.128 DEBUG:boto:Params: {} 2023-10-17T19:09:51.128 DEBUG:boto:Token: None 2023-10-17T19:09:51.129 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23 2023-10-17T19:09:51.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:By2yL8JPIpz7hZD3Y2en3jiySrM= 2023-10-17T19:09:51.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:By2yL8JPIpz7hZD3Y2en3jiySrM=', 'Content-Length': '0'} 2023-10-17T19:09:51.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d70326b4c3be502-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.132 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.133 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.133 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.134 DEBUG:boto:Path: /hbmuog-61/obj23 2023-10-17T19:09:51.134 DEBUG:boto:Data: 2023-10-17T19:09:51.135 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.135 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.136 DEBUG:boto:Params: {} 2023-10-17T19:09:51.136 DEBUG:boto:Token: None 2023-10-17T19:09:51.137 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23 2023-10-17T19:09:51.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:By2yL8JPIpz7hZD3Y2en3jiySrM= 2023-10-17T19:09:51.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:By2yL8JPIpz7hZD3Y2en3jiySrM=', 'Content-Length': '0'} 2023-10-17T19:09:51.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ae90c3988c4816-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-17T19:09:51.141 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.141 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.142 DEBUG:boto:Method: GET 2023-10-17T19:09:51.142 DEBUG:boto:Path: /hbmuog-61/obj23 2023-10-17T19:09:51.143 DEBUG:boto:Data: 2023-10-17T19:09:51.143 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.144 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.145 DEBUG:boto:Params: {} 2023-10-17T19:09:51.145 DEBUG:boto:Token: None 2023-10-17T19:09:51.146 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23 2023-10-17T19:09:51.146 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2aAEjNK20Kl7U4vigsYX3VhlK0= 2023-10-17T19:09:51.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2aAEjNK20Kl7U4vigsYX3VhlK0=', 'Content-Length': '0'} 2023-10-17T19:09:51.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000088b61c24b50d27d-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.190 DEBUG:boto:path=/hbmuog-61/obj23 2023-10-17T19:09:51.191 DEBUG:boto:auth_path=/hbmuog-61/obj23 2023-10-17T19:09:51.191 DEBUG:boto:Method: GET 2023-10-17T19:09:51.192 DEBUG:boto:Path: /hbmuog-61/obj23 2023-10-17T19:09:51.192 DEBUG:boto:Data: 2023-10-17T19:09:51.193 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.193 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.194 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.194 DEBUG:boto:Params: {} 2023-10-17T19:09:51.195 DEBUG:boto:Token: None 2023-10-17T19:09:51.195 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj23 2023-10-17T19:09:51.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N2aAEjNK20Kl7U4vigsYX3VhlK0= 2023-10-17T19:09:51.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N2aAEjNK20Kl7U4vigsYX3VhlK0=', 'Content-Length': '0'} 2023-10-17T19:09:51.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f503aedfb590fa35-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.199 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.199 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.200 DEBUG:boto:path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.200 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.201 DEBUG:boto:Method: GET 2023-10-17T19:09:51.201 DEBUG:boto:Path: /hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.202 DEBUG:boto:Data: 2023-10-17T19:09:51.202 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.202 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.203 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.203 DEBUG:boto:Params: {} 2023-10-17T19:09:51.204 DEBUG:boto:Token: None 2023-10-17T19:09:51.204 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.205 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d3efc758f66b1f-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.208 DEBUG:boto:b'hbmuog-61obj231000falseobj232023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.209 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.209 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.210 DEBUG:boto:path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.210 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.211 DEBUG:boto:Method: GET 2023-10-17T19:09:51.211 DEBUG:boto:Path: /hbmuog-61/?prefix=obj23 2023-10-17T19:09:51.211 DEBUG:boto:Data: 2023-10-17T19:09:51.212 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.212 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.213 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.213 DEBUG:boto:Params: {} 2023-10-17T19:09:51.214 DEBUG:boto:Token: None 2023-10-17T19:09:51.214 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4197b75f98dc79f-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.262 DEBUG:boto:b'hbmuog-61obj231000falseobj232023-10-17T19:06:20.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-17T19:09:51.264 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.264 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.265 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.265 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.265 DEBUG:boto:Method: GET 2023-10-17T19:09:51.266 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.266 DEBUG:boto:Data: 2023-10-17T19:09:51.267 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.268 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.268 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.269 DEBUG:boto:Params: {} 2023-10-17T19:09:51.269 DEBUG:boto:Token: None 2023-10-17T19:09:51.270 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.270 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA= 2023-10-17T19:09:51.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA=', 'Content-Length': '0'} 2023-10-17T19:09:51.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddd900c42c4d90e-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.274 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.274 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.275 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.275 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.276 DEBUG:boto:Method: GET 2023-10-17T19:09:51.276 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.277 DEBUG:boto:Data: 2023-10-17T19:09:51.278 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.279 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.279 DEBUG:boto:Params: {} 2023-10-17T19:09:51.280 DEBUG:boto:Token: None 2023-10-17T19:09:51.280 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.281 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA= 2023-10-17T19:09:51.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA=', 'Content-Length': '0'} 2023-10-17T19:09:51.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bece61a9c3a50a0c-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.285 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.286 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.286 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.287 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.288 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.289 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.289 DEBUG:boto:Data: 2023-10-17T19:09:51.290 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.291 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.292 DEBUG:boto:Params: {} 2023-10-17T19:09:51.293 DEBUG:boto:Token: None 2023-10-17T19:09:51.294 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fZF5xKuMdLv4xO9j8mJQPbFbsuI= 2023-10-17T19:09:51.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fZF5xKuMdLv4xO9j8mJQPbFbsuI=', 'Content-Length': '0'} 2023-10-17T19:09:51.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfc823a15a18c317-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.298 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.299 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.299 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.300 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.300 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.300 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.301 DEBUG:boto:Data: 2023-10-17T19:09:51.301 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.302 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.303 DEBUG:boto:Params: {} 2023-10-17T19:09:51.304 DEBUG:boto:Token: None 2023-10-17T19:09:51.305 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.306 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fZF5xKuMdLv4xO9j8mJQPbFbsuI= 2023-10-17T19:09:51.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fZF5xKuMdLv4xO9j8mJQPbFbsuI=', 'Content-Length': '0'} 2023-10-17T19:09:51.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121af0b604f70ad4-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-17T19:09:51.311 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.311 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.312 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.313 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.313 DEBUG:boto:Method: GET 2023-10-17T19:09:51.314 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.314 DEBUG:boto:Data: 2023-10-17T19:09:51.315 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.316 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.317 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.317 DEBUG:boto:Params: {} 2023-10-17T19:09:51.318 DEBUG:boto:Token: None 2023-10-17T19:09:51.319 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.320 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA= 2023-10-17T19:09:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA=', 'Content-Length': '0'} 2023-10-17T19:09:51.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1b76f6a1d1dd77-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.324 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.324 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.325 DEBUG:boto:path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.325 DEBUG:boto:auth_path=/hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.325 DEBUG:boto:Method: GET 2023-10-17T19:09:51.326 DEBUG:boto:Path: /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.326 DEBUG:boto:Data: 2023-10-17T19:09:51.327 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.328 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.328 DEBUG:boto:Params: {} 2023-10-17T19:09:51.329 DEBUG:boto:Token: None 2023-10-17T19:09:51.329 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29?versionId=null 2023-10-17T19:09:51.330 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA= 2023-10-17T19:09:51.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TLBCAkYMC23bQs7aUEuCyeiSZfA=', 'Content-Length': '0'} 2023-10-17T19:09:51.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d41404818c27096-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.333 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.334 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.335 DEBUG:boto:path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.336 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.336 DEBUG:boto:Method: GET 2023-10-17T19:09:51.337 DEBUG:boto:Path: /hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.338 DEBUG:boto:Data: 2023-10-17T19:09:51.338 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.339 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.340 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.340 DEBUG:boto:Params: {} 2023-10-17T19:09:51.341 DEBUG:boto:Token: None 2023-10-17T19:09:51.341 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.342 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8b3c7e0c34262a0-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.347 DEBUG:boto:b'hbmuog-61obj291000falseobj292023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.348 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.349 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.349 DEBUG:boto:path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.350 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.350 DEBUG:boto:Method: GET 2023-10-17T19:09:51.351 DEBUG:boto:Path: /hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.351 DEBUG:boto:Data: 2023-10-17T19:09:51.352 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.353 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.353 DEBUG:boto:Params: {} 2023-10-17T19:09:51.354 DEBUG:boto:Token: None 2023-10-17T19:09:51.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb61d381b9c3443-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.360 DEBUG:boto:b'hbmuog-61obj291000falseobj292023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.361 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.361 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.362 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.362 DEBUG:boto:Path: /hbmuog-61/obj29 2023-10-17T19:09:51.362 DEBUG:boto:Data: 2023-10-17T19:09:51.363 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.364 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.364 DEBUG:boto:Params: {} 2023-10-17T19:09:51.365 DEBUG:boto:Token: None 2023-10-17T19:09:51.365 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29 2023-10-17T19:09:51.366 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q992U7YdxSxWAw+wQlisXT2ByFg= 2023-10-17T19:09:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q992U7YdxSxWAw+wQlisXT2ByFg=', 'Content-Length': '0'} 2023-10-17T19:09:51.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3349969b7df563-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.369 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.370 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.370 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.371 DEBUG:boto:Path: /hbmuog-61/obj29 2023-10-17T19:09:51.371 DEBUG:boto:Data: 2023-10-17T19:09:51.372 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.372 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.373 DEBUG:boto:Params: {} 2023-10-17T19:09:51.373 DEBUG:boto:Token: None 2023-10-17T19:09:51.374 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29 2023-10-17T19:09:51.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q992U7YdxSxWAw+wQlisXT2ByFg= 2023-10-17T19:09:51.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q992U7YdxSxWAw+wQlisXT2ByFg=', 'Content-Length': '0'} 2023-10-17T19:09:51.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1849129869ce55c-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-17T19:09:51.378 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.378 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.379 DEBUG:boto:Method: GET 2023-10-17T19:09:51.381 DEBUG:boto:Path: /hbmuog-61/obj29 2023-10-17T19:09:51.382 DEBUG:boto:Data: 2023-10-17T19:09:51.382 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.383 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.383 DEBUG:boto:Params: {} 2023-10-17T19:09:51.384 DEBUG:boto:Token: None 2023-10-17T19:09:51.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29 2023-10-17T19:09:51.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QACp0t+16zW+TAafKqPAHIQHIKs= 2023-10-17T19:09:51.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QACp0t+16zW+TAafKqPAHIQHIKs=', 'Content-Length': '0'} 2023-10-17T19:09:51.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff0c638d108a0727-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.388 DEBUG:boto:path=/hbmuog-61/obj29 2023-10-17T19:09:51.388 DEBUG:boto:auth_path=/hbmuog-61/obj29 2023-10-17T19:09:51.389 DEBUG:boto:Method: GET 2023-10-17T19:09:51.389 DEBUG:boto:Path: /hbmuog-61/obj29 2023-10-17T19:09:51.389 DEBUG:boto:Data: 2023-10-17T19:09:51.390 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.391 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.391 DEBUG:boto:Params: {} 2023-10-17T19:09:51.392 DEBUG:boto:Token: None 2023-10-17T19:09:51.392 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj29 2023-10-17T19:09:51.393 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QACp0t+16zW+TAafKqPAHIQHIKs= 2023-10-17T19:09:51.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QACp0t+16zW+TAafKqPAHIQHIKs=', 'Content-Length': '0'} 2023-10-17T19:09:51.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008577a58ece501b22-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.396 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.396 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.397 DEBUG:boto:path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.397 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.398 DEBUG:boto:Method: GET 2023-10-17T19:09:51.398 DEBUG:boto:Path: /hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.399 DEBUG:boto:Data: 2023-10-17T19:09:51.399 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.400 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.400 DEBUG:boto:Params: {} 2023-10-17T19:09:51.401 DEBUG:boto:Token: None 2023-10-17T19:09:51.401 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae4b7c9382943d1b-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.450 DEBUG:boto:b'hbmuog-61obj291000falseobj292023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.451 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.451 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.452 DEBUG:boto:path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.452 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.453 DEBUG:boto:Method: GET 2023-10-17T19:09:51.453 DEBUG:boto:Path: /hbmuog-61/?prefix=obj29 2023-10-17T19:09:51.454 DEBUG:boto:Data: 2023-10-17T19:09:51.454 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.455 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.455 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.455 DEBUG:boto:Params: {} 2023-10-17T19:09:51.456 DEBUG:boto:Token: None 2023-10-17T19:09:51.457 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.457 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096c615857247ef49-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.462 DEBUG:boto:b'hbmuog-61obj291000falseobj292023-10-17T19:06:20.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-17T19:09:51.463 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.464 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.464 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.465 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.465 DEBUG:boto:Method: GET 2023-10-17T19:09:51.466 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.466 DEBUG:boto:Data: 2023-10-17T19:09:51.466 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.467 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.468 DEBUG:boto:Params: {} 2023-10-17T19:09:51.468 DEBUG:boto:Token: None 2023-10-17T19:09:51.469 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.469 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q= 2023-10-17T19:09:51.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q=', 'Content-Length': '0'} 2023-10-17T19:09:51.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4611d7c362cb63c-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.472 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.473 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.473 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.474 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.474 DEBUG:boto:Method: GET 2023-10-17T19:09:51.475 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.475 DEBUG:boto:Data: 2023-10-17T19:09:51.476 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.477 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.477 DEBUG:boto:Params: {} 2023-10-17T19:09:51.478 DEBUG:boto:Token: None 2023-10-17T19:09:51.478 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q= 2023-10-17T19:09:51.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q=', 'Content-Length': '0'} 2023-10-17T19:09:51.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003317daa1f9c6d088-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.482 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.482 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.483 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.483 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.483 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.484 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.484 DEBUG:boto:Data: 2023-10-17T19:09:51.485 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.485 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.486 DEBUG:boto:Params: {} 2023-10-17T19:09:51.486 DEBUG:boto:Token: None 2023-10-17T19:09:51.487 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PYoD/08kaifnAzzEwbzpJoUyT6g= 2023-10-17T19:09:51.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PYoD/08kaifnAzzEwbzpJoUyT6g=', 'Content-Length': '0'} 2023-10-17T19:09:51.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002484152ca5048d1d-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.490 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.491 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.491 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.491 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.492 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.492 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.493 DEBUG:boto:Data: 2023-10-17T19:09:51.493 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.494 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.494 DEBUG:boto:Params: {} 2023-10-17T19:09:51.495 DEBUG:boto:Token: None 2023-10-17T19:09:51.495 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PYoD/08kaifnAzzEwbzpJoUyT6g= 2023-10-17T19:09:51.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PYoD/08kaifnAzzEwbzpJoUyT6g=', 'Content-Length': '0'} 2023-10-17T19:09:51.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2cc16abfeb2189-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-17T19:09:51.499 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.499 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.500 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.500 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.501 DEBUG:boto:Method: GET 2023-10-17T19:09:51.501 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.502 DEBUG:boto:Data: 2023-10-17T19:09:51.502 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.503 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.503 DEBUG:boto:Params: {} 2023-10-17T19:09:51.504 DEBUG:boto:Token: None 2023-10-17T19:09:51.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.505 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q= 2023-10-17T19:09:51.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q=', 'Content-Length': '0'} 2023-10-17T19:09:51.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef27b7767462806-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.507 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.508 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.508 DEBUG:boto:path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.509 DEBUG:boto:auth_path=/hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.509 DEBUG:boto:Method: GET 2023-10-17T19:09:51.510 DEBUG:boto:Path: /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.510 DEBUG:boto:Data: 2023-10-17T19:09:51.510 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.511 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.511 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.512 DEBUG:boto:Params: {} 2023-10-17T19:09:51.512 DEBUG:boto:Token: None 2023-10-17T19:09:51.513 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31?versionId=null 2023-10-17T19:09:51.513 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q= 2023-10-17T19:09:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vb4I55AfWzXZ8vfizkml7j8E85Q=', 'Content-Length': '0'} 2023-10-17T19:09:51.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b52783b595efa70-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.516 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.516 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.517 DEBUG:boto:path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.517 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.518 DEBUG:boto:Method: GET 2023-10-17T19:09:51.518 DEBUG:boto:Path: /hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.519 DEBUG:boto:Data: 2023-10-17T19:09:51.519 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.519 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.520 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.520 DEBUG:boto:Params: {} 2023-10-17T19:09:51.521 DEBUG:boto:Token: None 2023-10-17T19:09:51.521 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.522 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d224b2bf8a0721-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.525 DEBUG:boto:b'hbmuog-61obj311000falseobj312023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.526 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.526 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.526 DEBUG:boto:path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.527 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.527 DEBUG:boto:Method: GET 2023-10-17T19:09:51.528 DEBUG:boto:Path: /hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.528 DEBUG:boto:Data: 2023-10-17T19:09:51.528 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.529 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.529 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.530 DEBUG:boto:Params: {} 2023-10-17T19:09:51.530 DEBUG:boto:Token: None 2023-10-17T19:09:51.531 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b45832114358e0-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.536 DEBUG:boto:b'hbmuog-61obj311000falseobj312023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.537 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.537 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.537 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.538 DEBUG:boto:Path: /hbmuog-61/obj31 2023-10-17T19:09:51.538 DEBUG:boto:Data: 2023-10-17T19:09:51.539 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.539 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.539 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.540 DEBUG:boto:Params: {} 2023-10-17T19:09:51.540 DEBUG:boto:Token: None 2023-10-17T19:09:51.541 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31 2023-10-17T19:09:51.541 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PVKL41w/yEO7dZm/pbGRnhUwPow= 2023-10-17T19:09:51.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PVKL41w/yEO7dZm/pbGRnhUwPow=', 'Content-Length': '0'} 2023-10-17T19:09:51.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9f60758070cf14-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.544 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.544 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.545 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.545 DEBUG:boto:Path: /hbmuog-61/obj31 2023-10-17T19:09:51.546 DEBUG:boto:Data: 2023-10-17T19:09:51.546 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.547 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.547 DEBUG:boto:Params: {} 2023-10-17T19:09:51.548 DEBUG:boto:Token: None 2023-10-17T19:09:51.548 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31 2023-10-17T19:09:51.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PVKL41w/yEO7dZm/pbGRnhUwPow= 2023-10-17T19:09:51.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PVKL41w/yEO7dZm/pbGRnhUwPow=', 'Content-Length': '0'} 2023-10-17T19:09:51.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c7b5413b7aea3c6-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-17T19:09:51.551 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.552 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.552 DEBUG:boto:Method: GET 2023-10-17T19:09:51.553 DEBUG:boto:Path: /hbmuog-61/obj31 2023-10-17T19:09:51.553 DEBUG:boto:Data: 2023-10-17T19:09:51.553 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.554 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.554 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.555 DEBUG:boto:Params: {} 2023-10-17T19:09:51.555 DEBUG:boto:Token: None 2023-10-17T19:09:51.556 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31 2023-10-17T19:09:51.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4u9dCvTr7q1ypT44zMo4c/gV/0A= 2023-10-17T19:09:51.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4u9dCvTr7q1ypT44zMo4c/gV/0A=', 'Content-Length': '0'} 2023-10-17T19:09:51.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021156f38081c736b-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.559 DEBUG:boto:path=/hbmuog-61/obj31 2023-10-17T19:09:51.559 DEBUG:boto:auth_path=/hbmuog-61/obj31 2023-10-17T19:09:51.560 DEBUG:boto:Method: GET 2023-10-17T19:09:51.560 DEBUG:boto:Path: /hbmuog-61/obj31 2023-10-17T19:09:51.560 DEBUG:boto:Data: 2023-10-17T19:09:51.561 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.561 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.562 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.562 DEBUG:boto:Params: {} 2023-10-17T19:09:51.562 DEBUG:boto:Token: None 2023-10-17T19:09:51.563 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj31 2023-10-17T19:09:51.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4u9dCvTr7q1ypT44zMo4c/gV/0A= 2023-10-17T19:09:51.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4u9dCvTr7q1ypT44zMo4c/gV/0A=', 'Content-Length': '0'} 2023-10-17T19:09:51.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3027d2a04c8778a-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.606 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.607 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.607 DEBUG:boto:path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.608 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.608 DEBUG:boto:Method: GET 2023-10-17T19:09:51.609 DEBUG:boto:Path: /hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.609 DEBUG:boto:Data: 2023-10-17T19:09:51.609 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.610 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.610 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.611 DEBUG:boto:Params: {} 2023-10-17T19:09:51.611 DEBUG:boto:Token: None 2023-10-17T19:09:51.612 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.612 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c10aeafe63bed7-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.616 DEBUG:boto:b'hbmuog-61obj311000falseobj312023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.616 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.617 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.617 DEBUG:boto:path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.618 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.618 DEBUG:boto:Method: GET 2023-10-17T19:09:51.619 DEBUG:boto:Path: /hbmuog-61/?prefix=obj31 2023-10-17T19:09:51.619 DEBUG:boto:Data: 2023-10-17T19:09:51.619 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.620 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.621 DEBUG:boto:Params: {} 2023-10-17T19:09:51.621 DEBUG:boto:Token: None 2023-10-17T19:09:51.622 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.622 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000030bbd34e336d372-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.627 DEBUG:boto:b'hbmuog-61obj311000falseobj312023-10-17T19:06:21.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:09:51.628 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.629 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.629 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.629 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.630 DEBUG:boto:Method: GET 2023-10-17T19:09:51.630 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.631 DEBUG:boto:Data: 2023-10-17T19:09:51.631 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.632 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.632 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.632 DEBUG:boto:Params: {} 2023-10-17T19:09:51.633 DEBUG:boto:Token: None 2023-10-17T19:09:51.633 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI= 2023-10-17T19:09:51.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI=', 'Content-Length': '0'} 2023-10-17T19:09:51.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ede21b0978d97f-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.678 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.679 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.679 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.680 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.680 DEBUG:boto:Method: GET 2023-10-17T19:09:51.681 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.681 DEBUG:boto:Data: 2023-10-17T19:09:51.682 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.682 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.683 DEBUG:boto:Params: {} 2023-10-17T19:09:51.683 DEBUG:boto:Token: None 2023-10-17T19:09:51.684 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI= 2023-10-17T19:09:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI=', 'Content-Length': '0'} 2023-10-17T19:09:51.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e72d5af508c5d2d-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.688 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.689 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.690 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.690 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.690 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.691 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.691 DEBUG:boto:Data: 2023-10-17T19:09:51.692 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.692 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.693 DEBUG:boto:Params: {} 2023-10-17T19:09:51.693 DEBUG:boto:Token: None 2023-10-17T19:09:51.694 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eXMDrBaL9Ps90isF/KpmyxssSmQ= 2023-10-17T19:09:51.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eXMDrBaL9Ps90isF/KpmyxssSmQ=', 'Content-Length': '0'} 2023-10-17T19:09:51.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dca93164dd326906-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.698 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.698 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.699 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.699 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.699 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.700 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.700 DEBUG:boto:Data: 2023-10-17T19:09:51.701 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.702 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.702 DEBUG:boto:Params: {} 2023-10-17T19:09:51.703 DEBUG:boto:Token: None 2023-10-17T19:09:51.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.703 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eXMDrBaL9Ps90isF/KpmyxssSmQ= 2023-10-17T19:09:51.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eXMDrBaL9Ps90isF/KpmyxssSmQ=', 'Content-Length': '0'} 2023-10-17T19:09:51.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae1cda0f1819002-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:09:51.707 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.707 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.708 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.708 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.709 DEBUG:boto:Method: GET 2023-10-17T19:09:51.709 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.709 DEBUG:boto:Data: 2023-10-17T19:09:51.710 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.711 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.711 DEBUG:boto:Params: {} 2023-10-17T19:09:51.712 DEBUG:boto:Token: None 2023-10-17T19:09:51.712 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI= 2023-10-17T19:09:51.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI=', 'Content-Length': '0'} 2023-10-17T19:09:51.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80a351aac2398e5-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.716 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.716 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.717 DEBUG:boto:path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.717 DEBUG:boto:auth_path=/hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.718 DEBUG:boto:Method: GET 2023-10-17T19:09:51.718 DEBUG:boto:Path: /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.718 DEBUG:boto:Data: 2023-10-17T19:09:51.719 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.720 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.720 DEBUG:boto:Params: {} 2023-10-17T19:09:51.721 DEBUG:boto:Token: None 2023-10-17T19:09:51.721 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33?versionId=null 2023-10-17T19:09:51.722 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI= 2023-10-17T19:09:51.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BVYCBQxLZnXFu4MUadwa+aYvzSI=', 'Content-Length': '0'} 2023-10-17T19:09:51.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020fef45500bac4b6-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.725 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.725 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.726 DEBUG:boto:path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.726 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.727 DEBUG:boto:Method: GET 2023-10-17T19:09:51.727 DEBUG:boto:Path: /hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.728 DEBUG:boto:Data: 2023-10-17T19:09:51.728 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.728 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.729 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.729 DEBUG:boto:Params: {} 2023-10-17T19:09:51.730 DEBUG:boto:Token: None 2023-10-17T19:09:51.730 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.731 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f0f0b718d70747-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.735 DEBUG:boto:b'hbmuog-61obj331000falseobj332023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.735 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.736 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.736 DEBUG:boto:path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.737 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.737 DEBUG:boto:Method: GET 2023-10-17T19:09:51.737 DEBUG:boto:Path: /hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.738 DEBUG:boto:Data: 2023-10-17T19:09:51.738 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.739 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.739 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.740 DEBUG:boto:Params: {} 2023-10-17T19:09:51.740 DEBUG:boto:Token: None 2023-10-17T19:09:51.741 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001519c20a47c05e5-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.746 DEBUG:boto:b'hbmuog-61obj331000falseobj332023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.747 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.747 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.748 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.748 DEBUG:boto:Path: /hbmuog-61/obj33 2023-10-17T19:09:51.749 DEBUG:boto:Data: 2023-10-17T19:09:51.749 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.750 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.750 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.750 DEBUG:boto:Params: {} 2023-10-17T19:09:51.751 DEBUG:boto:Token: None 2023-10-17T19:09:51.752 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33 2023-10-17T19:09:51.752 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EXi3YzF5c89IF63sCXl05OvfeiM= 2023-10-17T19:09:51.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EXi3YzF5c89IF63sCXl05OvfeiM=', 'Content-Length': '0'} 2023-10-17T19:09:51.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4b01fa99b2c5b9-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.755 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.756 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.756 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.756 DEBUG:boto:Path: /hbmuog-61/obj33 2023-10-17T19:09:51.757 DEBUG:boto:Data: 2023-10-17T19:09:51.757 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.758 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.759 DEBUG:boto:Params: {} 2023-10-17T19:09:51.759 DEBUG:boto:Token: None 2023-10-17T19:09:51.760 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33 2023-10-17T19:09:51.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EXi3YzF5c89IF63sCXl05OvfeiM= 2023-10-17T19:09:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EXi3YzF5c89IF63sCXl05OvfeiM=', 'Content-Length': '0'} 2023-10-17T19:09:51.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028cbcc0714513c2a-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:09:51.763 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.764 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.764 DEBUG:boto:Method: GET 2023-10-17T19:09:51.765 DEBUG:boto:Path: /hbmuog-61/obj33 2023-10-17T19:09:51.765 DEBUG:boto:Data: 2023-10-17T19:09:51.765 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.766 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.767 DEBUG:boto:Params: {} 2023-10-17T19:09:51.767 DEBUG:boto:Token: None 2023-10-17T19:09:51.768 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33 2023-10-17T19:09:51.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:woWy0NvnmbiT6rDIhDTmmGgAaco= 2023-10-17T19:09:51.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:woWy0NvnmbiT6rDIhDTmmGgAaco=', 'Content-Length': '0'} 2023-10-17T19:09:51.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91273a670866ca7-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.814 DEBUG:boto:path=/hbmuog-61/obj33 2023-10-17T19:09:51.815 DEBUG:boto:auth_path=/hbmuog-61/obj33 2023-10-17T19:09:51.815 DEBUG:boto:Method: GET 2023-10-17T19:09:51.816 DEBUG:boto:Path: /hbmuog-61/obj33 2023-10-17T19:09:51.816 DEBUG:boto:Data: 2023-10-17T19:09:51.817 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.818 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.818 DEBUG:boto:Params: {} 2023-10-17T19:09:51.818 DEBUG:boto:Token: None 2023-10-17T19:09:51.819 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj33 2023-10-17T19:09:51.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:woWy0NvnmbiT6rDIhDTmmGgAaco= 2023-10-17T19:09:51.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:woWy0NvnmbiT6rDIhDTmmGgAaco=', 'Content-Length': '0'} 2023-10-17T19:09:51.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088705188ea330418-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.823 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.823 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.824 DEBUG:boto:path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.824 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.824 DEBUG:boto:Method: GET 2023-10-17T19:09:51.825 DEBUG:boto:Path: /hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.825 DEBUG:boto:Data: 2023-10-17T19:09:51.826 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.827 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.827 DEBUG:boto:Params: {} 2023-10-17T19:09:51.827 DEBUG:boto:Token: None 2023-10-17T19:09:51.828 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003568b9bc0753285-00652edbff-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.832 DEBUG:boto:b'hbmuog-61obj331000falseobj332023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.832 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.833 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.833 DEBUG:boto:path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.834 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.834 DEBUG:boto:Method: GET 2023-10-17T19:09:51.835 DEBUG:boto:Path: /hbmuog-61/?prefix=obj33 2023-10-17T19:09:51.835 DEBUG:boto:Data: 2023-10-17T19:09:51.835 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.836 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.837 DEBUG:boto:Params: {} 2023-10-17T19:09:51.837 DEBUG:boto:Token: None 2023-10-17T19:09:51.838 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:51.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:51.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026ed5ec7b81e2764-00652edbff-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.843 DEBUG:boto:b'hbmuog-61obj331000falseobj332023-10-17T19:06:19.741Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:51.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T19:09:51.845 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.845 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.845 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.846 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.846 DEBUG:boto:Method: GET 2023-10-17T19:09:51.847 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.847 DEBUG:boto:Data: 2023-10-17T19:09:51.848 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.848 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.848 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.849 DEBUG:boto:Params: {} 2023-10-17T19:09:51.849 DEBUG:boto:Token: None 2023-10-17T19:09:51.850 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY= 2023-10-17T19:09:51.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY=', 'Content-Length': '0'} 2023-10-17T19:09:51.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d1dbc8602184c5-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.854 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.854 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.855 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.855 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.856 DEBUG:boto:Method: GET 2023-10-17T19:09:51.856 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.856 DEBUG:boto:Data: 2023-10-17T19:09:51.857 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.858 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.858 DEBUG:boto:Params: {} 2023-10-17T19:09:51.859 DEBUG:boto:Token: None 2023-10-17T19:09:51.859 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY= 2023-10-17T19:09:51.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY=', 'Content-Length': '0'} 2023-10-17T19:09:51.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef943d681c49bff1-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.902 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.903 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.904 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.904 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.904 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.905 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.905 DEBUG:boto:Data: 2023-10-17T19:09:51.906 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.906 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.907 DEBUG:boto:Params: {} 2023-10-17T19:09:51.907 DEBUG:boto:Token: None 2023-10-17T19:09:51.908 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.909 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A/W0blLL8qi5VncBrlWaWNS6jS8= 2023-10-17T19:09:51.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A/W0blLL8qi5VncBrlWaWNS6jS8=', 'Content-Length': '0'} 2023-10-17T19:09:51.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f00f8def5a26ee9-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.912 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.912 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.913 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.913 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.914 DEBUG:boto:Method: HEAD 2023-10-17T19:09:51.915 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.916 DEBUG:boto:Data: 2023-10-17T19:09:51.917 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.917 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.918 DEBUG:boto:Params: {} 2023-10-17T19:09:51.918 DEBUG:boto:Token: None 2023-10-17T19:09:51.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A/W0blLL8qi5VncBrlWaWNS6jS8= 2023-10-17T19:09:51.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A/W0blLL8qi5VncBrlWaWNS6jS8=', 'Content-Length': '0'} 2023-10-17T19:09:51.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b301b1511c3c7937-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T19:09:51.923 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.923 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.923 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.924 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.924 DEBUG:boto:Method: GET 2023-10-17T19:09:51.925 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.925 DEBUG:boto:Data: 2023-10-17T19:09:51.926 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.926 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.927 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.927 DEBUG:boto:Params: {} 2023-10-17T19:09:51.928 DEBUG:boto:Token: None 2023-10-17T19:09:51.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY= 2023-10-17T19:09:51.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY=', 'Content-Length': '0'} 2023-10-17T19:09:51.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5e52be239e32ec-00652edbff-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.978 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:51.980 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:51.980 DEBUG:boto:path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.981 DEBUG:boto:auth_path=/hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.982 DEBUG:boto:Method: GET 2023-10-17T19:09:51.982 DEBUG:boto:Path: /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.983 DEBUG:boto:Data: 2023-10-17T19:09:51.983 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.984 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:51.984 DEBUG:boto:Port: 8000 2023-10-17T19:09:51.985 DEBUG:boto:Params: {} 2023-10-17T19:09:51.985 DEBUG:boto:Token: None 2023-10-17T19:09:51.986 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/obj34?versionId=null 2023-10-17T19:09:51.987 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY= 2023-10-17T19:09:51.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3+Px277ubHPQkpIdgKeNrdrMPoY=', 'Content-Length': '0'} 2023-10-17T19:09:51.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09a378276ff9c27-00652edbff-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:51.990 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:51.990 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:51.991 DEBUG:boto:path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:51.991 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:51.992 DEBUG:boto:Method: GET 2023-10-17T19:09:51.992 DEBUG:boto:Path: /hbmuog-61/?prefix=obj34 2023-10-17T19:09:51.992 DEBUG:boto:Data: 2023-10-17T19:09:51.993 DEBUG:boto:Headers: {} 2023-10-17T19:09:51.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:51.994 DEBUG:boto:Port: 8001 2023-10-17T19:09:51.997 DEBUG:boto:Params: {} 2023-10-17T19:09:51.998 DEBUG:boto:Token: None 2023-10-17T19:09:51.999 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:51 GMT /hbmuog-61/ 2023-10-17T19:09:51.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks= 2023-10-17T19:09:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x7T6EMnvA1lq5103YzU/b6faeks=', 'Content-Length': '0'} 2023-10-17T19:09:52.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee44952ff847564-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.003 DEBUG:boto:b'hbmuog-61obj341000falseobj342023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.003 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.004 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.004 DEBUG:boto:path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.005 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.005 DEBUG:boto:Method: GET 2023-10-17T19:09:52.006 DEBUG:boto:Path: /hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.006 DEBUG:boto:Data: 2023-10-17T19:09:52.007 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.007 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.008 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.008 DEBUG:boto:Params: {} 2023-10-17T19:09:52.008 DEBUG:boto:Token: None 2023-10-17T19:09:52.009 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.009 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231a7d40b3c0dd72-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.015 DEBUG:boto:b'hbmuog-61obj341000falseobj342023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.016 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:52.016 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:52.016 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.017 DEBUG:boto:Path: /hbmuog-61/obj34 2023-10-17T19:09:52.017 DEBUG:boto:Data: 2023-10-17T19:09:52.018 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.018 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.018 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.019 DEBUG:boto:Params: {} 2023-10-17T19:09:52.019 DEBUG:boto:Token: None 2023-10-17T19:09:52.020 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj34 2023-10-17T19:09:52.020 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bBupSuUSW4B/bZ3fDGvtfVkq2q8= 2023-10-17T19:09:52.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bBupSuUSW4B/bZ3fDGvtfVkq2q8=', 'Content-Length': '0'} 2023-10-17T19:09:52.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536c26daf526a878-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.023 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:52.024 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:52.024 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.025 DEBUG:boto:Path: /hbmuog-61/obj34 2023-10-17T19:09:52.025 DEBUG:boto:Data: 2023-10-17T19:09:52.025 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.026 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.027 DEBUG:boto:Params: {} 2023-10-17T19:09:52.027 DEBUG:boto:Token: None 2023-10-17T19:09:52.027 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj34 2023-10-17T19:09:52.028 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bBupSuUSW4B/bZ3fDGvtfVkq2q8= 2023-10-17T19:09:52.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bBupSuUSW4B/bZ3fDGvtfVkq2q8=', 'Content-Length': '0'} 2023-10-17T19:09:52.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bd63c0d2cb0126-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-17T19:09:52.031 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:52.032 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:52.032 DEBUG:boto:Method: GET 2023-10-17T19:09:52.032 DEBUG:boto:Path: /hbmuog-61/obj34 2023-10-17T19:09:52.033 DEBUG:boto:Data: 2023-10-17T19:09:52.033 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.034 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.034 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.035 DEBUG:boto:Params: {} 2023-10-17T19:09:52.035 DEBUG:boto:Token: None 2023-10-17T19:09:52.036 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj34 2023-10-17T19:09:52.036 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZ0B1fg0uq5uJSMzwLCkCHDkhho= 2023-10-17T19:09:52.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZ0B1fg0uq5uJSMzwLCkCHDkhho=', 'Content-Length': '0'} 2023-10-17T19:09:52.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b01b5d38e928cf-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.082 DEBUG:boto:path=/hbmuog-61/obj34 2023-10-17T19:09:52.083 DEBUG:boto:auth_path=/hbmuog-61/obj34 2023-10-17T19:09:52.083 DEBUG:boto:Method: GET 2023-10-17T19:09:52.084 DEBUG:boto:Path: /hbmuog-61/obj34 2023-10-17T19:09:52.084 DEBUG:boto:Data: 2023-10-17T19:09:52.085 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.085 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.085 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.086 DEBUG:boto:Params: {} 2023-10-17T19:09:52.086 DEBUG:boto:Token: None 2023-10-17T19:09:52.087 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj34 2023-10-17T19:09:52.087 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZZ0B1fg0uq5uJSMzwLCkCHDkhho= 2023-10-17T19:09:52.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZZ0B1fg0uq5uJSMzwLCkCHDkhho=', 'Content-Length': '0'} 2023-10-17T19:09:52.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eabdeca70b0b934-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.091 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.091 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.092 DEBUG:boto:path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.092 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.094 DEBUG:boto:Method: GET 2023-10-17T19:09:52.094 DEBUG:boto:Path: /hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.095 DEBUG:boto:Data: 2023-10-17T19:09:52.095 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.096 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.096 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.096 DEBUG:boto:Params: {} 2023-10-17T19:09:52.097 DEBUG:boto:Token: None 2023-10-17T19:09:52.098 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.098 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651b27a11f22877d-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.102 DEBUG:boto:b'hbmuog-61obj341000falseobj342023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.103 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.103 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.104 DEBUG:boto:path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.104 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.104 DEBUG:boto:Method: GET 2023-10-17T19:09:52.105 DEBUG:boto:Path: /hbmuog-61/?prefix=obj34 2023-10-17T19:09:52.105 DEBUG:boto:Data: 2023-10-17T19:09:52.106 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.106 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.107 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.107 DEBUG:boto:Params: {} 2023-10-17T19:09:52.108 DEBUG:boto:Token: None 2023-10-17T19:09:52.108 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e05ed07f6025f3a-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.115 DEBUG:boto:b'hbmuog-61obj341000falseobj342023-10-17T19:06:20.009Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-17T19:09:52.116 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.116 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.117 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.117 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.118 DEBUG:boto:Method: GET 2023-10-17T19:09:52.118 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.119 DEBUG:boto:Data: 2023-10-17T19:09:52.119 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.119 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.120 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.120 DEBUG:boto:Params: {} 2023-10-17T19:09:52.121 DEBUG:boto:Token: None 2023-10-17T19:09:52.121 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.122 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q= 2023-10-17T19:09:52.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q=', 'Content-Length': '0'} 2023-10-17T19:09:52.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6b0d837edfa517-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.125 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.126 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.126 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.127 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.127 DEBUG:boto:Method: GET 2023-10-17T19:09:52.128 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.128 DEBUG:boto:Data: 2023-10-17T19:09:52.129 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.129 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.130 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.130 DEBUG:boto:Params: {} 2023-10-17T19:09:52.131 DEBUG:boto:Token: None 2023-10-17T19:09:52.131 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.132 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q= 2023-10-17T19:09:52.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q=', 'Content-Length': '0'} 2023-10-17T19:09:52.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6f473b1da12c4a-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.179 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.180 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.180 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.180 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.181 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.181 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.182 DEBUG:boto:Data: 2023-10-17T19:09:52.182 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.183 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.183 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.184 DEBUG:boto:Params: {} 2023-10-17T19:09:52.184 DEBUG:boto:Token: None 2023-10-17T19:09:52.185 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.185 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RjLeVX8YW5K/HSxjxXz6ReySxpk= 2023-10-17T19:09:52.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RjLeVX8YW5K/HSxjxXz6ReySxpk=', 'Content-Length': '0'} 2023-10-17T19:09:52.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000536d0eb0ec0ebed-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.189 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.189 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.189 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.190 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.190 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.191 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.191 DEBUG:boto:Data: 2023-10-17T19:09:52.191 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.192 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.192 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.193 DEBUG:boto:Params: {} 2023-10-17T19:09:52.193 DEBUG:boto:Token: None 2023-10-17T19:09:52.194 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.194 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RjLeVX8YW5K/HSxjxXz6ReySxpk= 2023-10-17T19:09:52.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RjLeVX8YW5K/HSxjxXz6ReySxpk=', 'Content-Length': '0'} 2023-10-17T19:09:52.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000856df860d7e581cc-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-17T19:09:52.200 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.201 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.201 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.201 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.202 DEBUG:boto:Method: GET 2023-10-17T19:09:52.202 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.203 DEBUG:boto:Data: 2023-10-17T19:09:52.203 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.204 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.204 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.204 DEBUG:boto:Params: {} 2023-10-17T19:09:52.205 DEBUG:boto:Token: None 2023-10-17T19:09:52.205 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.206 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q= 2023-10-17T19:09:52.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q=', 'Content-Length': '0'} 2023-10-17T19:09:52.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256b0a7c174f2585-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.209 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.210 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.210 DEBUG:boto:path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.210 DEBUG:boto:auth_path=/hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.211 DEBUG:boto:Method: GET 2023-10-17T19:09:52.211 DEBUG:boto:Path: /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.212 DEBUG:boto:Data: 2023-10-17T19:09:52.212 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.213 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.213 DEBUG:boto:Params: {} 2023-10-17T19:09:52.214 DEBUG:boto:Token: None 2023-10-17T19:09:52.214 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37?versionId=null 2023-10-17T19:09:52.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q= 2023-10-17T19:09:52.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P+DKGANc/Pm4wSWPfvebr2hZf9Q=', 'Content-Length': '0'} 2023-10-17T19:09:52.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e153c95a0244fb47-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.218 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.219 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.219 DEBUG:boto:path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.220 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.220 DEBUG:boto:Method: GET 2023-10-17T19:09:52.221 DEBUG:boto:Path: /hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.221 DEBUG:boto:Data: 2023-10-17T19:09:52.222 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.223 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.223 DEBUG:boto:Params: {} 2023-10-17T19:09:52.224 DEBUG:boto:Token: None 2023-10-17T19:09:52.224 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.225 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a01408a08659efd-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.228 DEBUG:boto:b'hbmuog-61obj371000falseobj372023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.229 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.229 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.230 DEBUG:boto:path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.230 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.231 DEBUG:boto:Method: GET 2023-10-17T19:09:52.231 DEBUG:boto:Path: /hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.231 DEBUG:boto:Data: 2023-10-17T19:09:52.232 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.232 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.233 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.233 DEBUG:boto:Params: {} 2023-10-17T19:09:52.234 DEBUG:boto:Token: None 2023-10-17T19:09:52.234 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.235 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b5f359b41f5862e-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.239 DEBUG:boto:b'hbmuog-61obj371000falseobj372023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.240 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.241 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.241 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.241 DEBUG:boto:Path: /hbmuog-61/obj37 2023-10-17T19:09:52.242 DEBUG:boto:Data: 2023-10-17T19:09:52.242 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.243 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.243 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.243 DEBUG:boto:Params: {} 2023-10-17T19:09:52.244 DEBUG:boto:Token: None 2023-10-17T19:09:52.244 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37 2023-10-17T19:09:52.245 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VQsGapzchKJqnUScLoHVdbWHW98= 2023-10-17T19:09:52.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VQsGapzchKJqnUScLoHVdbWHW98=', 'Content-Length': '0'} 2023-10-17T19:09:52.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066834142cc57961d-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.248 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.248 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.249 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.249 DEBUG:boto:Path: /hbmuog-61/obj37 2023-10-17T19:09:52.250 DEBUG:boto:Data: 2023-10-17T19:09:52.250 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.251 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.252 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.252 DEBUG:boto:Params: {} 2023-10-17T19:09:52.253 DEBUG:boto:Token: None 2023-10-17T19:09:52.254 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37 2023-10-17T19:09:52.255 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VQsGapzchKJqnUScLoHVdbWHW98= 2023-10-17T19:09:52.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VQsGapzchKJqnUScLoHVdbWHW98=', 'Content-Length': '0'} 2023-10-17T19:09:52.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe721fe5f353c19d-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-17T19:09:52.259 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.260 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.261 DEBUG:boto:Method: GET 2023-10-17T19:09:52.262 DEBUG:boto:Path: /hbmuog-61/obj37 2023-10-17T19:09:52.262 DEBUG:boto:Data: 2023-10-17T19:09:52.263 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.264 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.264 DEBUG:boto:Params: {} 2023-10-17T19:09:52.265 DEBUG:boto:Token: None 2023-10-17T19:09:52.266 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37 2023-10-17T19:09:52.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:psUdWHVvnjBlpddIe2FpvHBpVDY= 2023-10-17T19:09:52.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:psUdWHVvnjBlpddIe2FpvHBpVDY=', 'Content-Length': '0'} 2023-10-17T19:09:52.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce1661f2069a89d5-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.310 DEBUG:boto:path=/hbmuog-61/obj37 2023-10-17T19:09:52.311 DEBUG:boto:auth_path=/hbmuog-61/obj37 2023-10-17T19:09:52.312 DEBUG:boto:Method: GET 2023-10-17T19:09:52.312 DEBUG:boto:Path: /hbmuog-61/obj37 2023-10-17T19:09:52.312 DEBUG:boto:Data: 2023-10-17T19:09:52.313 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.314 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.314 DEBUG:boto:Params: {} 2023-10-17T19:09:52.315 DEBUG:boto:Token: None 2023-10-17T19:09:52.315 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj37 2023-10-17T19:09:52.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:psUdWHVvnjBlpddIe2FpvHBpVDY= 2023-10-17T19:09:52.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:psUdWHVvnjBlpddIe2FpvHBpVDY=', 'Content-Length': '0'} 2023-10-17T19:09:52.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2676d28fee6eb9-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.319 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.320 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.320 DEBUG:boto:path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.320 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.321 DEBUG:boto:Method: GET 2023-10-17T19:09:52.321 DEBUG:boto:Path: /hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.322 DEBUG:boto:Data: 2023-10-17T19:09:52.322 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.323 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.323 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.323 DEBUG:boto:Params: {} 2023-10-17T19:09:52.324 DEBUG:boto:Token: None 2023-10-17T19:09:52.324 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.325 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f91b50a0fdd2e4-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.329 DEBUG:boto:b'hbmuog-61obj371000falseobj372023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.329 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.330 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.330 DEBUG:boto:path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.331 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.331 DEBUG:boto:Method: GET 2023-10-17T19:09:52.332 DEBUG:boto:Path: /hbmuog-61/?prefix=obj37 2023-10-17T19:09:52.332 DEBUG:boto:Data: 2023-10-17T19:09:52.333 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.333 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.334 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.334 DEBUG:boto:Params: {} 2023-10-17T19:09:52.334 DEBUG:boto:Token: None 2023-10-17T19:09:52.335 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.335 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0477af188cd860-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.343 DEBUG:boto:b'hbmuog-61obj371000falseobj372023-10-17T19:06:21.737Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-17T19:09:52.344 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.345 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.345 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.346 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.346 DEBUG:boto:Method: GET 2023-10-17T19:09:52.347 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.347 DEBUG:boto:Data: 2023-10-17T19:09:52.348 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.348 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.349 DEBUG:boto:Params: {} 2023-10-17T19:09:52.349 DEBUG:boto:Token: None 2023-10-17T19:09:52.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4= 2023-10-17T19:09:52.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4=', 'Content-Length': '0'} 2023-10-17T19:09:52.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004191a0bdf95ab920-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.354 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.355 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.355 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.356 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.356 DEBUG:boto:Method: GET 2023-10-17T19:09:52.357 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.357 DEBUG:boto:Data: 2023-10-17T19:09:52.358 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.358 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.358 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.359 DEBUG:boto:Params: {} 2023-10-17T19:09:52.359 DEBUG:boto:Token: None 2023-10-17T19:09:52.360 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.361 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4= 2023-10-17T19:09:52.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4=', 'Content-Length': '0'} 2023-10-17T19:09:52.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653d0c4b80f1e069-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.364 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.365 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.365 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.365 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.366 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.366 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.367 DEBUG:boto:Data: 2023-10-17T19:09:52.367 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.368 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.368 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.368 DEBUG:boto:Params: {} 2023-10-17T19:09:52.369 DEBUG:boto:Token: None 2023-10-17T19:09:52.369 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.370 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3qUCcNvon7Y43Hz6cUJzriodUKY= 2023-10-17T19:09:52.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3qUCcNvon7Y43Hz6cUJzriodUKY=', 'Content-Length': '0'} 2023-10-17T19:09:52.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb1cddf85bf12073-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.373 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.374 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.374 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.374 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.375 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.375 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.376 DEBUG:boto:Data: 2023-10-17T19:09:52.376 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.376 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.377 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.377 DEBUG:boto:Params: {} 2023-10-17T19:09:52.378 DEBUG:boto:Token: None 2023-10-17T19:09:52.378 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.379 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3qUCcNvon7Y43Hz6cUJzriodUKY= 2023-10-17T19:09:52.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3qUCcNvon7Y43Hz6cUJzriodUKY=', 'Content-Length': '0'} 2023-10-17T19:09:52.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e10d3694d0d8014-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-17T19:09:52.383 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.383 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.384 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.384 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.384 DEBUG:boto:Method: GET 2023-10-17T19:09:52.385 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.385 DEBUG:boto:Data: 2023-10-17T19:09:52.386 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.387 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.387 DEBUG:boto:Params: {} 2023-10-17T19:09:52.388 DEBUG:boto:Token: None 2023-10-17T19:09:52.388 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.389 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4= 2023-10-17T19:09:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4=', 'Content-Length': '0'} 2023-10-17T19:09:52.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771f03104d8a16b3-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.392 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.392 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.393 DEBUG:boto:path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.393 DEBUG:boto:auth_path=/hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.394 DEBUG:boto:Method: GET 2023-10-17T19:09:52.394 DEBUG:boto:Path: /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.395 DEBUG:boto:Data: 2023-10-17T19:09:52.395 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.396 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.396 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.397 DEBUG:boto:Params: {} 2023-10-17T19:09:52.397 DEBUG:boto:Token: None 2023-10-17T19:09:52.398 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38?versionId=null 2023-10-17T19:09:52.398 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4= 2023-10-17T19:09:52.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/lGslgK7gCqyT9gPww5PTFDtRh4=', 'Content-Length': '0'} 2023-10-17T19:09:52.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a9892ba6354b0d-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.442 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.443 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.444 DEBUG:boto:path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.444 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.445 DEBUG:boto:Method: GET 2023-10-17T19:09:52.445 DEBUG:boto:Path: /hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.446 DEBUG:boto:Data: 2023-10-17T19:09:52.446 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.447 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.448 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.448 DEBUG:boto:Params: {} 2023-10-17T19:09:52.449 DEBUG:boto:Token: None 2023-10-17T19:09:52.449 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.450 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b36b02599f31d05-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.453 DEBUG:boto:b'hbmuog-61obj381000falseobj382023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.454 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.455 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.455 DEBUG:boto:path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.456 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.456 DEBUG:boto:Method: GET 2023-10-17T19:09:52.456 DEBUG:boto:Path: /hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.457 DEBUG:boto:Data: 2023-10-17T19:09:52.457 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.458 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.458 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.459 DEBUG:boto:Params: {} 2023-10-17T19:09:52.459 DEBUG:boto:Token: None 2023-10-17T19:09:52.460 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000991f2df6378ad55c-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.466 DEBUG:boto:b'hbmuog-61obj381000falseobj382023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.466 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.467 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.467 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.468 DEBUG:boto:Path: /hbmuog-61/obj38 2023-10-17T19:09:52.468 DEBUG:boto:Data: 2023-10-17T19:09:52.468 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.469 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.470 DEBUG:boto:Params: {} 2023-10-17T19:09:52.470 DEBUG:boto:Token: None 2023-10-17T19:09:52.471 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38 2023-10-17T19:09:52.471 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UW5lju0OZ32hZy96tghcKF7TRyc= 2023-10-17T19:09:52.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UW5lju0OZ32hZy96tghcKF7TRyc=', 'Content-Length': '0'} 2023-10-17T19:09:52.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b20c7d83dbdfbf8-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.474 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.475 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.475 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.476 DEBUG:boto:Path: /hbmuog-61/obj38 2023-10-17T19:09:52.476 DEBUG:boto:Data: 2023-10-17T19:09:52.476 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.477 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.477 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.478 DEBUG:boto:Params: {} 2023-10-17T19:09:52.478 DEBUG:boto:Token: None 2023-10-17T19:09:52.479 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38 2023-10-17T19:09:52.479 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UW5lju0OZ32hZy96tghcKF7TRyc= 2023-10-17T19:09:52.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UW5lju0OZ32hZy96tghcKF7TRyc=', 'Content-Length': '0'} 2023-10-17T19:09:52.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fe14149fbb134a-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-17T19:09:52.483 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.483 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.483 DEBUG:boto:Method: GET 2023-10-17T19:09:52.484 DEBUG:boto:Path: /hbmuog-61/obj38 2023-10-17T19:09:52.484 DEBUG:boto:Data: 2023-10-17T19:09:52.485 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.485 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.486 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.486 DEBUG:boto:Params: {} 2023-10-17T19:09:52.487 DEBUG:boto:Token: None 2023-10-17T19:09:52.487 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38 2023-10-17T19:09:52.488 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y9O+AJKQIq3zmW7LIPDQfQdypDQ= 2023-10-17T19:09:52.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y9O+AJKQIq3zmW7LIPDQfQdypDQ=', 'Content-Length': '0'} 2023-10-17T19:09:52.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e319dc6f614a402a-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.491 DEBUG:boto:path=/hbmuog-61/obj38 2023-10-17T19:09:52.491 DEBUG:boto:auth_path=/hbmuog-61/obj38 2023-10-17T19:09:52.492 DEBUG:boto:Method: GET 2023-10-17T19:09:52.492 DEBUG:boto:Path: /hbmuog-61/obj38 2023-10-17T19:09:52.492 DEBUG:boto:Data: 2023-10-17T19:09:52.493 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.493 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.494 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.494 DEBUG:boto:Params: {} 2023-10-17T19:09:52.495 DEBUG:boto:Token: None 2023-10-17T19:09:52.495 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj38 2023-10-17T19:09:52.496 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y9O+AJKQIq3zmW7LIPDQfQdypDQ= 2023-10-17T19:09:52.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y9O+AJKQIq3zmW7LIPDQfQdypDQ=', 'Content-Length': '0'} 2023-10-17T19:09:52.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6159ea8327b12e-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.498 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.499 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.499 DEBUG:boto:path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.500 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.500 DEBUG:boto:Method: GET 2023-10-17T19:09:52.501 DEBUG:boto:Path: /hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.501 DEBUG:boto:Data: 2023-10-17T19:09:52.501 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.502 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.502 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.503 DEBUG:boto:Params: {} 2023-10-17T19:09:52.503 DEBUG:boto:Token: None 2023-10-17T19:09:52.504 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ef1a4679905a30-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.508 DEBUG:boto:b'hbmuog-61obj381000falseobj382023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.509 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.509 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.510 DEBUG:boto:path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.510 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.510 DEBUG:boto:Method: GET 2023-10-17T19:09:52.511 DEBUG:boto:Path: /hbmuog-61/?prefix=obj38 2023-10-17T19:09:52.511 DEBUG:boto:Data: 2023-10-17T19:09:52.512 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.512 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.513 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.513 DEBUG:boto:Params: {} 2023-10-17T19:09:52.514 DEBUG:boto:Token: None 2023-10-17T19:09:52.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b48d49d67da2e1e4-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.520 DEBUG:boto:b'hbmuog-61obj381000falseobj382023-10-17T19:06:20.336Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:09:52.521 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.521 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.522 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.522 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.523 DEBUG:boto:Method: GET 2023-10-17T19:09:52.523 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.524 DEBUG:boto:Data: 2023-10-17T19:09:52.524 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.525 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.525 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.525 DEBUG:boto:Params: {} 2023-10-17T19:09:52.526 DEBUG:boto:Token: None 2023-10-17T19:09:52.526 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.527 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14= 2023-10-17T19:09:52.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14=', 'Content-Length': '0'} 2023-10-17T19:09:52.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf7130b8e2e7b15-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.531 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.531 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.532 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.532 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.532 DEBUG:boto:Method: GET 2023-10-17T19:09:52.533 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.533 DEBUG:boto:Data: 2023-10-17T19:09:52.534 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.534 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.535 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.535 DEBUG:boto:Params: {} 2023-10-17T19:09:52.535 DEBUG:boto:Token: None 2023-10-17T19:09:52.536 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.536 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14= 2023-10-17T19:09:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14=', 'Content-Length': '0'} 2023-10-17T19:09:52.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bc2bce1cd2e2c4-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.586 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.587 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.588 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.588 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.588 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.589 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.589 DEBUG:boto:Data: 2023-10-17T19:09:52.590 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.590 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.591 DEBUG:boto:Params: {} 2023-10-17T19:09:52.591 DEBUG:boto:Token: None 2023-10-17T19:09:52.592 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JXyDcOGqNA1Hs6TzOy8okMeHjnI= 2023-10-17T19:09:52.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JXyDcOGqNA1Hs6TzOy8okMeHjnI=', 'Content-Length': '0'} 2023-10-17T19:09:52.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d046906c5050aafd-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.595 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.596 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.596 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.597 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.597 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.598 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.598 DEBUG:boto:Data: 2023-10-17T19:09:52.598 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.599 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.600 DEBUG:boto:Params: {} 2023-10-17T19:09:52.600 DEBUG:boto:Token: None 2023-10-17T19:09:52.601 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.601 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JXyDcOGqNA1Hs6TzOy8okMeHjnI= 2023-10-17T19:09:52.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JXyDcOGqNA1Hs6TzOy8okMeHjnI=', 'Content-Length': '0'} 2023-10-17T19:09:52.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d704acca0bc8bd6e-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:09:52.605 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.605 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.606 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.606 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.606 DEBUG:boto:Method: GET 2023-10-17T19:09:52.607 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.607 DEBUG:boto:Data: 2023-10-17T19:09:52.608 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.608 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.609 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.609 DEBUG:boto:Params: {} 2023-10-17T19:09:52.610 DEBUG:boto:Token: None 2023-10-17T19:09:52.610 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.611 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14= 2023-10-17T19:09:52.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14=', 'Content-Length': '0'} 2023-10-17T19:09:52.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f58bdbe18dbfc1-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.654 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.655 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.655 DEBUG:boto:path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.656 DEBUG:boto:auth_path=/hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.656 DEBUG:boto:Method: GET 2023-10-17T19:09:52.657 DEBUG:boto:Path: /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.657 DEBUG:boto:Data: 2023-10-17T19:09:52.658 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.659 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.659 DEBUG:boto:Params: {} 2023-10-17T19:09:52.659 DEBUG:boto:Token: None 2023-10-17T19:09:52.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44?versionId=null 2023-10-17T19:09:52.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14= 2023-10-17T19:09:52.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFjs5oU7lf7wkwtt30BBvqXlZ14=', 'Content-Length': '0'} 2023-10-17T19:09:52.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989f22168c73c618-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.664 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.665 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.665 DEBUG:boto:path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.666 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.666 DEBUG:boto:Method: GET 2023-10-17T19:09:52.666 DEBUG:boto:Path: /hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.667 DEBUG:boto:Data: 2023-10-17T19:09:52.667 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.668 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.669 DEBUG:boto:Params: {} 2023-10-17T19:09:52.669 DEBUG:boto:Token: None 2023-10-17T19:09:52.670 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.670 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc6915b27c7e0fab-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.674 DEBUG:boto:b'hbmuog-61obj441000falseobj442023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.675 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.675 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.676 DEBUG:boto:path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.676 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.677 DEBUG:boto:Method: GET 2023-10-17T19:09:52.677 DEBUG:boto:Path: /hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.678 DEBUG:boto:Data: 2023-10-17T19:09:52.678 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.678 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.679 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.679 DEBUG:boto:Params: {} 2023-10-17T19:09:52.680 DEBUG:boto:Token: None 2023-10-17T19:09:52.680 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.681 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087505f4e3e02cce2-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.687 DEBUG:boto:b'hbmuog-61obj441000falseobj442023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.688 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.688 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.689 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.689 DEBUG:boto:Path: /hbmuog-61/obj44 2023-10-17T19:09:52.690 DEBUG:boto:Data: 2023-10-17T19:09:52.690 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.691 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.691 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.692 DEBUG:boto:Params: {} 2023-10-17T19:09:52.692 DEBUG:boto:Token: None 2023-10-17T19:09:52.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44 2023-10-17T19:09:52.693 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:si9AWEiMX8mVHmSklevIFOpfK5Y= 2023-10-17T19:09:52.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:si9AWEiMX8mVHmSklevIFOpfK5Y=', 'Content-Length': '0'} 2023-10-17T19:09:52.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa33e4f2e1520c5-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.698 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.698 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.699 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.699 DEBUG:boto:Path: /hbmuog-61/obj44 2023-10-17T19:09:52.700 DEBUG:boto:Data: 2023-10-17T19:09:52.700 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.701 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.702 DEBUG:boto:Params: {} 2023-10-17T19:09:52.702 DEBUG:boto:Token: None 2023-10-17T19:09:52.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44 2023-10-17T19:09:52.703 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:si9AWEiMX8mVHmSklevIFOpfK5Y= 2023-10-17T19:09:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:si9AWEiMX8mVHmSklevIFOpfK5Y=', 'Content-Length': '0'} 2023-10-17T19:09:52.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509ebde3890fa78d-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:09:52.707 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.707 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.708 DEBUG:boto:Method: GET 2023-10-17T19:09:52.708 DEBUG:boto:Path: /hbmuog-61/obj44 2023-10-17T19:09:52.709 DEBUG:boto:Data: 2023-10-17T19:09:52.709 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.710 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.711 DEBUG:boto:Params: {} 2023-10-17T19:09:52.711 DEBUG:boto:Token: None 2023-10-17T19:09:52.712 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44 2023-10-17T19:09:52.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAfJNzGXmYuiWOV98KHWby/jLIg= 2023-10-17T19:09:52.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAfJNzGXmYuiWOV98KHWby/jLIg=', 'Content-Length': '0'} 2023-10-17T19:09:52.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c1eb61a5829f4d-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.716 DEBUG:boto:path=/hbmuog-61/obj44 2023-10-17T19:09:52.717 DEBUG:boto:auth_path=/hbmuog-61/obj44 2023-10-17T19:09:52.717 DEBUG:boto:Method: GET 2023-10-17T19:09:52.718 DEBUG:boto:Path: /hbmuog-61/obj44 2023-10-17T19:09:52.718 DEBUG:boto:Data: 2023-10-17T19:09:52.719 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.720 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.720 DEBUG:boto:Params: {} 2023-10-17T19:09:52.721 DEBUG:boto:Token: None 2023-10-17T19:09:52.721 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj44 2023-10-17T19:09:52.722 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cAfJNzGXmYuiWOV98KHWby/jLIg= 2023-10-17T19:09:52.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cAfJNzGXmYuiWOV98KHWby/jLIg=', 'Content-Length': '0'} 2023-10-17T19:09:52.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3b1b5368151306-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.770 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.771 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.772 DEBUG:boto:path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.772 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.772 DEBUG:boto:Method: GET 2023-10-17T19:09:52.773 DEBUG:boto:Path: /hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.773 DEBUG:boto:Data: 2023-10-17T19:09:52.774 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.775 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.775 DEBUG:boto:Params: {} 2023-10-17T19:09:52.775 DEBUG:boto:Token: None 2023-10-17T19:09:52.776 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce72cf426d58211-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.780 DEBUG:boto:b'hbmuog-61obj441000falseobj442023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.781 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.781 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.782 DEBUG:boto:path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.782 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.783 DEBUG:boto:Method: GET 2023-10-17T19:09:52.783 DEBUG:boto:Path: /hbmuog-61/?prefix=obj44 2023-10-17T19:09:52.784 DEBUG:boto:Data: 2023-10-17T19:09:52.784 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.785 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.785 DEBUG:boto:Params: {} 2023-10-17T19:09:52.786 DEBUG:boto:Token: None 2023-10-17T19:09:52.786 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f574f547cf32c1-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.793 DEBUG:boto:b'hbmuog-61obj441000falseobj442023-10-17T19:06:19.764Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-17T19:09:52.794 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.794 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.795 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.795 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.796 DEBUG:boto:Method: GET 2023-10-17T19:09:52.796 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.797 DEBUG:boto:Data: 2023-10-17T19:09:52.797 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.798 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.798 DEBUG:boto:Params: {} 2023-10-17T19:09:52.799 DEBUG:boto:Token: None 2023-10-17T19:09:52.799 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90= 2023-10-17T19:09:52.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90=', 'Content-Length': '0'} 2023-10-17T19:09:52.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaafa735395a372e-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.803 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.803 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.804 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.804 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.805 DEBUG:boto:Method: GET 2023-10-17T19:09:52.805 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.806 DEBUG:boto:Data: 2023-10-17T19:09:52.806 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.807 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.807 DEBUG:boto:Params: {} 2023-10-17T19:09:52.808 DEBUG:boto:Token: None 2023-10-17T19:09:52.808 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90= 2023-10-17T19:09:52.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90=', 'Content-Length': '0'} 2023-10-17T19:09:52.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01839052c30bbd9-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.814 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.814 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.815 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.815 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.815 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.816 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.816 DEBUG:boto:Data: 2023-10-17T19:09:52.817 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.817 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.818 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.818 DEBUG:boto:Params: {} 2023-10-17T19:09:52.818 DEBUG:boto:Token: None 2023-10-17T19:09:52.819 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.819 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CS+Vy/PiGMX1vcLGmhAhXdC9+oY= 2023-10-17T19:09:52.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CS+Vy/PiGMX1vcLGmhAhXdC9+oY=', 'Content-Length': '0'} 2023-10-17T19:09:52.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a18820575a0bf48-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.822 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.823 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.823 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.824 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.824 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.824 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.825 DEBUG:boto:Data: 2023-10-17T19:09:52.825 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.826 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.826 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.827 DEBUG:boto:Params: {} 2023-10-17T19:09:52.827 DEBUG:boto:Token: None 2023-10-17T19:09:52.827 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.828 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CS+Vy/PiGMX1vcLGmhAhXdC9+oY= 2023-10-17T19:09:52.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CS+Vy/PiGMX1vcLGmhAhXdC9+oY=', 'Content-Length': '0'} 2023-10-17T19:09:52.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bee7de13d5ad3eca-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-17T19:09:52.831 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.832 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.832 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.833 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.833 DEBUG:boto:Method: GET 2023-10-17T19:09:52.834 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.834 DEBUG:boto:Data: 2023-10-17T19:09:52.834 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.835 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.836 DEBUG:boto:Params: {} 2023-10-17T19:09:52.836 DEBUG:boto:Token: None 2023-10-17T19:09:52.837 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90= 2023-10-17T19:09:52.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90=', 'Content-Length': '0'} 2023-10-17T19:09:52.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d19c04356d4dffb-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.882 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.883 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.883 DEBUG:boto:path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.884 DEBUG:boto:auth_path=/hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.884 DEBUG:boto:Method: GET 2023-10-17T19:09:52.885 DEBUG:boto:Path: /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.885 DEBUG:boto:Data: 2023-10-17T19:09:52.886 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.886 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.887 DEBUG:boto:Params: {} 2023-10-17T19:09:52.887 DEBUG:boto:Token: None 2023-10-17T19:09:52.888 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46?versionId=null 2023-10-17T19:09:52.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90= 2023-10-17T19:09:52.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2jO07g5s6YxpWAk3fxBFAiYIi90=', 'Content-Length': '0'} 2023-10-17T19:09:52.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643ca24380912112-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.892 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.892 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.893 DEBUG:boto:path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.893 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.893 DEBUG:boto:Method: GET 2023-10-17T19:09:52.894 DEBUG:boto:Path: /hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.894 DEBUG:boto:Data: 2023-10-17T19:09:52.895 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.895 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.896 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.896 DEBUG:boto:Params: {} 2023-10-17T19:09:52.897 DEBUG:boto:Token: None 2023-10-17T19:09:52.897 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.898 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b8125952991d295-00652edc00-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.901 DEBUG:boto:b'hbmuog-61obj461000falseobj462023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.902 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:52.903 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:52.903 DEBUG:boto:path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.903 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.904 DEBUG:boto:Method: GET 2023-10-17T19:09:52.904 DEBUG:boto:Path: /hbmuog-61/?prefix=obj46 2023-10-17T19:09:52.905 DEBUG:boto:Data: 2023-10-17T19:09:52.905 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.905 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.906 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.906 DEBUG:boto:Params: {} 2023-10-17T19:09:52.907 DEBUG:boto:Token: None 2023-10-17T19:09:52.907 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/ 2023-10-17T19:09:52.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU= 2023-10-17T19:09:52.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q3B5o+fWEleFF//TKklllJFOHSU=', 'Content-Length': '0'} 2023-10-17T19:09:52.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000785519c754cc4dd7-00652edc00-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.958 DEBUG:boto:b'hbmuog-61obj461000falseobj462023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:52.959 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.960 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.960 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.960 DEBUG:boto:Path: /hbmuog-61/obj46 2023-10-17T19:09:52.961 DEBUG:boto:Data: 2023-10-17T19:09:52.961 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.962 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.963 DEBUG:boto:Params: {} 2023-10-17T19:09:52.963 DEBUG:boto:Token: None 2023-10-17T19:09:52.964 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46 2023-10-17T19:09:52.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q6Xwoy4o1df+hQKoMaucVfUvI84= 2023-10-17T19:09:52.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q6Xwoy4o1df+hQKoMaucVfUvI84=', 'Content-Length': '0'} 2023-10-17T19:09:52.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f499e7b1769884-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.967 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.967 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.968 DEBUG:boto:Method: HEAD 2023-10-17T19:09:52.968 DEBUG:boto:Path: /hbmuog-61/obj46 2023-10-17T19:09:52.969 DEBUG:boto:Data: 2023-10-17T19:09:52.969 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.970 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.970 DEBUG:boto:Params: {} 2023-10-17T19:09:52.971 DEBUG:boto:Token: None 2023-10-17T19:09:52.971 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46 2023-10-17T19:09:52.973 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q6Xwoy4o1df+hQKoMaucVfUvI84= 2023-10-17T19:09:52.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q6Xwoy4o1df+hQKoMaucVfUvI84=', 'Content-Length': '0'} 2023-10-17T19:09:52.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648436a0b62d9b3d-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-17T19:09:52.977 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.978 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.978 DEBUG:boto:Method: GET 2023-10-17T19:09:52.979 DEBUG:boto:Path: /hbmuog-61/obj46 2023-10-17T19:09:52.980 DEBUG:boto:Data: 2023-10-17T19:09:52.981 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.981 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:52.982 DEBUG:boto:Port: 8001 2023-10-17T19:09:52.983 DEBUG:boto:Params: {} 2023-10-17T19:09:52.983 DEBUG:boto:Token: None 2023-10-17T19:09:52.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46 2023-10-17T19:09:52.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+90PLGBEFpUVe7OYWogZK8JyVy8= 2023-10-17T19:09:52.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+90PLGBEFpUVe7OYWogZK8JyVy8=', 'Content-Length': '0'} 2023-10-17T19:09:52.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dbf2365c85d7b96-00652edc00-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:52.988 DEBUG:boto:path=/hbmuog-61/obj46 2023-10-17T19:09:52.989 DEBUG:boto:auth_path=/hbmuog-61/obj46 2023-10-17T19:09:52.989 DEBUG:boto:Method: GET 2023-10-17T19:09:52.990 DEBUG:boto:Path: /hbmuog-61/obj46 2023-10-17T19:09:52.991 DEBUG:boto:Data: 2023-10-17T19:09:52.992 DEBUG:boto:Headers: {} 2023-10-17T19:09:52.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:52.993 DEBUG:boto:Port: 8000 2023-10-17T19:09:52.994 DEBUG:boto:Params: {} 2023-10-17T19:09:52.995 DEBUG:boto:Token: None 2023-10-17T19:09:52.996 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:52 GMT /hbmuog-61/obj46 2023-10-17T19:09:52.997 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+90PLGBEFpUVe7OYWogZK8JyVy8= 2023-10-17T19:09:52.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+90PLGBEFpUVe7OYWogZK8JyVy8=', 'Content-Length': '0'} 2023-10-17T19:09:53.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005afc43a7e38bea7c-00652edc00-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.001 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.001 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.001 DEBUG:boto:path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.002 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.002 DEBUG:boto:Method: GET 2023-10-17T19:09:53.003 DEBUG:boto:Path: /hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.003 DEBUG:boto:Data: 2023-10-17T19:09:53.004 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.004 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.004 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.005 DEBUG:boto:Params: {} 2023-10-17T19:09:53.005 DEBUG:boto:Token: None 2023-10-17T19:09:53.006 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.006 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087b59496a5884dbf-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.050 DEBUG:boto:b'hbmuog-61obj461000falseobj462023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.052 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.053 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.053 DEBUG:boto:path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.054 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.055 DEBUG:boto:Method: GET 2023-10-17T19:09:53.055 DEBUG:boto:Path: /hbmuog-61/?prefix=obj46 2023-10-17T19:09:53.056 DEBUG:boto:Data: 2023-10-17T19:09:53.056 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.057 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.058 DEBUG:boto:Params: {} 2023-10-17T19:09:53.058 DEBUG:boto:Token: None 2023-10-17T19:09:53.059 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000007f4c4ec8d87c59-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.065 DEBUG:boto:b'hbmuog-61obj461000falseobj462023-10-17T19:06:20.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T19:09:53.066 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.067 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.068 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.068 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.068 DEBUG:boto:Method: GET 2023-10-17T19:09:53.069 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.069 DEBUG:boto:Data: 2023-10-17T19:09:53.070 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.070 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.071 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.072 DEBUG:boto:Params: {} 2023-10-17T19:09:53.072 DEBUG:boto:Token: None 2023-10-17T19:09:53.073 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.074 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec= 2023-10-17T19:09:53.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec=', 'Content-Length': '0'} 2023-10-17T19:09:53.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9314016fc34c2a-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.078 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.078 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.079 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.079 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.080 DEBUG:boto:Method: GET 2023-10-17T19:09:53.080 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.081 DEBUG:boto:Data: 2023-10-17T19:09:53.081 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.082 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.082 DEBUG:boto:Params: {} 2023-10-17T19:09:53.083 DEBUG:boto:Token: None 2023-10-17T19:09:53.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.085 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec= 2023-10-17T19:09:53.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec=', 'Content-Length': '0'} 2023-10-17T19:09:53.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99284ff7e6cf213-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.130 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.131 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.132 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.132 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.132 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.133 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.133 DEBUG:boto:Data: 2023-10-17T19:09:53.134 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.136 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.136 DEBUG:boto:Params: {} 2023-10-17T19:09:53.137 DEBUG:boto:Token: None 2023-10-17T19:09:53.138 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbOvc97bg/dyuLR/hqA0SJyErlM= 2023-10-17T19:09:53.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbOvc97bg/dyuLR/hqA0SJyErlM=', 'Content-Length': '0'} 2023-10-17T19:09:53.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dbd567e5bff1267-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.143 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.144 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.145 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.145 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.146 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.147 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.148 DEBUG:boto:Data: 2023-10-17T19:09:53.148 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.149 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.150 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.151 DEBUG:boto:Params: {} 2023-10-17T19:09:53.152 DEBUG:boto:Token: None 2023-10-17T19:09:53.152 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.153 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbOvc97bg/dyuLR/hqA0SJyErlM= 2023-10-17T19:09:53.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbOvc97bg/dyuLR/hqA0SJyErlM=', 'Content-Length': '0'} 2023-10-17T19:09:53.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8df0ec9950f3428-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T19:09:53.157 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.157 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.158 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.158 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.159 DEBUG:boto:Method: GET 2023-10-17T19:09:53.159 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.160 DEBUG:boto:Data: 2023-10-17T19:09:53.160 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.160 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.161 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.161 DEBUG:boto:Params: {} 2023-10-17T19:09:53.162 DEBUG:boto:Token: None 2023-10-17T19:09:53.162 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.164 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec= 2023-10-17T19:09:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec=', 'Content-Length': '0'} 2023-10-17T19:09:53.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d114f8c52b601ee5-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.167 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.167 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.168 DEBUG:boto:path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.168 DEBUG:boto:auth_path=/hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.169 DEBUG:boto:Method: GET 2023-10-17T19:09:53.169 DEBUG:boto:Path: /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.169 DEBUG:boto:Data: 2023-10-17T19:09:53.170 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.171 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.171 DEBUG:boto:Params: {} 2023-10-17T19:09:53.172 DEBUG:boto:Token: None 2023-10-17T19:09:53.172 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51?versionId=null 2023-10-17T19:09:53.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec= 2023-10-17T19:09:53.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XO3/zJ20RTS0TNxg3mujCNuH7ec=', 'Content-Length': '0'} 2023-10-17T19:09:53.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047c6f4578e900d7-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.176 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.177 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.177 DEBUG:boto:path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.178 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.178 DEBUG:boto:Method: GET 2023-10-17T19:09:53.179 DEBUG:boto:Path: /hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.179 DEBUG:boto:Data: 2023-10-17T19:09:53.180 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.180 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.180 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.181 DEBUG:boto:Params: {} 2023-10-17T19:09:53.181 DEBUG:boto:Token: None 2023-10-17T19:09:53.182 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d8cf51e4c64c77-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.226 DEBUG:boto:b'hbmuog-61obj511000falseobj512023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.228 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.229 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.230 DEBUG:boto:path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.230 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.231 DEBUG:boto:Method: GET 2023-10-17T19:09:53.232 DEBUG:boto:Path: /hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.233 DEBUG:boto:Data: 2023-10-17T19:09:53.233 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.234 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.235 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.236 DEBUG:boto:Params: {} 2023-10-17T19:09:53.236 DEBUG:boto:Token: None 2023-10-17T19:09:53.237 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000458f757d04983b14-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.244 DEBUG:boto:b'hbmuog-61obj511000falseobj512023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.244 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.245 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.246 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.246 DEBUG:boto:Path: /hbmuog-61/obj51 2023-10-17T19:09:53.247 DEBUG:boto:Data: 2023-10-17T19:09:53.248 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.248 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.249 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.250 DEBUG:boto:Params: {} 2023-10-17T19:09:53.251 DEBUG:boto:Token: None 2023-10-17T19:09:53.252 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51 2023-10-17T19:09:53.253 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jQLqBO+OJJxLwtrCGAj3X6rkQpY= 2023-10-17T19:09:53.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jQLqBO+OJJxLwtrCGAj3X6rkQpY=', 'Content-Length': '0'} 2023-10-17T19:09:53.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4442cdc9ead270b-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.256 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.257 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.257 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.258 DEBUG:boto:Path: /hbmuog-61/obj51 2023-10-17T19:09:53.258 DEBUG:boto:Data: 2023-10-17T19:09:53.258 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.259 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.259 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.260 DEBUG:boto:Params: {} 2023-10-17T19:09:53.260 DEBUG:boto:Token: None 2023-10-17T19:09:53.260 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51 2023-10-17T19:09:53.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jQLqBO+OJJxLwtrCGAj3X6rkQpY= 2023-10-17T19:09:53.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jQLqBO+OJJxLwtrCGAj3X6rkQpY=', 'Content-Length': '0'} 2023-10-17T19:09:53.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bea03753182894-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-17T19:09:53.264 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.265 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.265 DEBUG:boto:Method: GET 2023-10-17T19:09:53.266 DEBUG:boto:Path: /hbmuog-61/obj51 2023-10-17T19:09:53.266 DEBUG:boto:Data: 2023-10-17T19:09:53.267 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.267 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.268 DEBUG:boto:Params: {} 2023-10-17T19:09:53.268 DEBUG:boto:Token: None 2023-10-17T19:09:53.269 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51 2023-10-17T19:09:53.269 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e8E6y73ZVTaYomyPFBEnuMpKvRM= 2023-10-17T19:09:53.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e8E6y73ZVTaYomyPFBEnuMpKvRM=', 'Content-Length': '0'} 2023-10-17T19:09:53.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7c6888d975a9b4-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.272 DEBUG:boto:path=/hbmuog-61/obj51 2023-10-17T19:09:53.273 DEBUG:boto:auth_path=/hbmuog-61/obj51 2023-10-17T19:09:53.273 DEBUG:boto:Method: GET 2023-10-17T19:09:53.274 DEBUG:boto:Path: /hbmuog-61/obj51 2023-10-17T19:09:53.274 DEBUG:boto:Data: 2023-10-17T19:09:53.275 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.275 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.275 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.276 DEBUG:boto:Params: {} 2023-10-17T19:09:53.276 DEBUG:boto:Token: None 2023-10-17T19:09:53.277 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj51 2023-10-17T19:09:53.277 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:e8E6y73ZVTaYomyPFBEnuMpKvRM= 2023-10-17T19:09:53.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:e8E6y73ZVTaYomyPFBEnuMpKvRM=', 'Content-Length': '0'} 2023-10-17T19:09:53.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd50d55116c0d32-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.322 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.323 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.323 DEBUG:boto:path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.324 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.324 DEBUG:boto:Method: GET 2023-10-17T19:09:53.325 DEBUG:boto:Path: /hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.325 DEBUG:boto:Data: 2023-10-17T19:09:53.326 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.326 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.327 DEBUG:boto:Params: {} 2023-10-17T19:09:53.327 DEBUG:boto:Token: None 2023-10-17T19:09:53.328 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfda8dbf6bb95d8c-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.332 DEBUG:boto:b'hbmuog-61obj511000falseobj512023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.333 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.333 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.334 DEBUG:boto:path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.334 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.335 DEBUG:boto:Method: GET 2023-10-17T19:09:53.335 DEBUG:boto:Path: /hbmuog-61/?prefix=obj51 2023-10-17T19:09:53.335 DEBUG:boto:Data: 2023-10-17T19:09:53.336 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.336 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.337 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.337 DEBUG:boto:Params: {} 2023-10-17T19:09:53.338 DEBUG:boto:Token: None 2023-10-17T19:09:53.338 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.339 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0d49f646b2f371-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.344 DEBUG:boto:b'hbmuog-61obj511000falseobj512023-10-17T19:06:20.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:09:53.345 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.345 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.346 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.346 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.347 DEBUG:boto:Method: GET 2023-10-17T19:09:53.347 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.347 DEBUG:boto:Data: 2023-10-17T19:09:53.348 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.349 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.349 DEBUG:boto:Params: {} 2023-10-17T19:09:53.350 DEBUG:boto:Token: None 2023-10-17T19:09:53.350 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.351 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q= 2023-10-17T19:09:53.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q=', 'Content-Length': '0'} 2023-10-17T19:09:53.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009191002ac7026e49-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.354 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.354 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.355 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.355 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.356 DEBUG:boto:Method: GET 2023-10-17T19:09:53.356 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.356 DEBUG:boto:Data: 2023-10-17T19:09:53.357 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.358 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.358 DEBUG:boto:Params: {} 2023-10-17T19:09:53.359 DEBUG:boto:Token: None 2023-10-17T19:09:53.359 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q= 2023-10-17T19:09:53.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q=', 'Content-Length': '0'} 2023-10-17T19:09:53.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575c87ed9a4f335b-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.363 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.363 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.364 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.364 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.364 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.365 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.365 DEBUG:boto:Data: 2023-10-17T19:09:53.365 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.366 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.367 DEBUG:boto:Params: {} 2023-10-17T19:09:53.367 DEBUG:boto:Token: None 2023-10-17T19:09:53.368 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.368 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zlbrFaPvznhkhcpTubn/e/VnW7o= 2023-10-17T19:09:53.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zlbrFaPvznhkhcpTubn/e/VnW7o=', 'Content-Length': '0'} 2023-10-17T19:09:53.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013abcec31a77e331-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.371 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.371 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.372 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.372 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.372 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.373 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.373 DEBUG:boto:Data: 2023-10-17T19:09:53.373 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.374 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.375 DEBUG:boto:Params: {} 2023-10-17T19:09:53.375 DEBUG:boto:Token: None 2023-10-17T19:09:53.376 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zlbrFaPvznhkhcpTubn/e/VnW7o= 2023-10-17T19:09:53.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zlbrFaPvznhkhcpTubn/e/VnW7o=', 'Content-Length': '0'} 2023-10-17T19:09:53.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e1c1d21773db3e-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:09:53.379 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.380 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.380 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.381 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.381 DEBUG:boto:Method: GET 2023-10-17T19:09:53.381 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.382 DEBUG:boto:Data: 2023-10-17T19:09:53.382 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.383 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.383 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.383 DEBUG:boto:Params: {} 2023-10-17T19:09:53.384 DEBUG:boto:Token: None 2023-10-17T19:09:53.384 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.385 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q= 2023-10-17T19:09:53.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q=', 'Content-Length': '0'} 2023-10-17T19:09:53.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002062fb4303308ee2-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.434 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.435 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.435 DEBUG:boto:path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.436 DEBUG:boto:auth_path=/hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.436 DEBUG:boto:Method: GET 2023-10-17T19:09:53.437 DEBUG:boto:Path: /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.437 DEBUG:boto:Data: 2023-10-17T19:09:53.438 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.438 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.439 DEBUG:boto:Params: {} 2023-10-17T19:09:53.439 DEBUG:boto:Token: None 2023-10-17T19:09:53.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55?versionId=null 2023-10-17T19:09:53.440 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q= 2023-10-17T19:09:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DeJl6BTLSqJMyH++lko6oaiWZ+Q=', 'Content-Length': '0'} 2023-10-17T19:09:53.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a579a32c93aba749-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.443 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.444 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.444 DEBUG:boto:path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.445 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.445 DEBUG:boto:Method: GET 2023-10-17T19:09:53.446 DEBUG:boto:Path: /hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.446 DEBUG:boto:Data: 2023-10-17T19:09:53.447 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.447 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.447 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.448 DEBUG:boto:Params: {} 2023-10-17T19:09:53.448 DEBUG:boto:Token: None 2023-10-17T19:09:53.449 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.449 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7332b76240ed59-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.453 DEBUG:boto:b'hbmuog-61obj551000falseobj552023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.454 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.454 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.455 DEBUG:boto:path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.455 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.456 DEBUG:boto:Method: GET 2023-10-17T19:09:53.456 DEBUG:boto:Path: /hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.456 DEBUG:boto:Data: 2023-10-17T19:09:53.457 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.458 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.458 DEBUG:boto:Params: {} 2023-10-17T19:09:53.459 DEBUG:boto:Token: None 2023-10-17T19:09:53.459 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.460 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa60b63b5d38f2a8-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.465 DEBUG:boto:b'hbmuog-61obj551000falseobj552023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.466 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.466 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.466 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.467 DEBUG:boto:Path: /hbmuog-61/obj55 2023-10-17T19:09:53.467 DEBUG:boto:Data: 2023-10-17T19:09:53.468 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.468 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.469 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.469 DEBUG:boto:Params: {} 2023-10-17T19:09:53.469 DEBUG:boto:Token: None 2023-10-17T19:09:53.470 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55 2023-10-17T19:09:53.470 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UWHr9FJ8ndERtuiXnSnYTTQG03U= 2023-10-17T19:09:53.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UWHr9FJ8ndERtuiXnSnYTTQG03U=', 'Content-Length': '0'} 2023-10-17T19:09:53.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5099d9ec073ba71-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.474 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.474 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.475 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.475 DEBUG:boto:Path: /hbmuog-61/obj55 2023-10-17T19:09:53.475 DEBUG:boto:Data: 2023-10-17T19:09:53.476 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.476 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.477 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.477 DEBUG:boto:Params: {} 2023-10-17T19:09:53.477 DEBUG:boto:Token: None 2023-10-17T19:09:53.478 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55 2023-10-17T19:09:53.478 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UWHr9FJ8ndERtuiXnSnYTTQG03U= 2023-10-17T19:09:53.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UWHr9FJ8ndERtuiXnSnYTTQG03U=', 'Content-Length': '0'} 2023-10-17T19:09:53.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079bca99e3b56e86e-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:09:53.482 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.482 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.483 DEBUG:boto:Method: GET 2023-10-17T19:09:53.483 DEBUG:boto:Path: /hbmuog-61/obj55 2023-10-17T19:09:53.484 DEBUG:boto:Data: 2023-10-17T19:09:53.484 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.484 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.485 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.485 DEBUG:boto:Params: {} 2023-10-17T19:09:53.486 DEBUG:boto:Token: None 2023-10-17T19:09:53.486 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55 2023-10-17T19:09:53.487 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AY1DgIH+zvRQmPIVPnmbaOaxbYA= 2023-10-17T19:09:53.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AY1DgIH+zvRQmPIVPnmbaOaxbYA=', 'Content-Length': '0'} 2023-10-17T19:09:53.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84c306ce2d12473-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.490 DEBUG:boto:path=/hbmuog-61/obj55 2023-10-17T19:09:53.490 DEBUG:boto:auth_path=/hbmuog-61/obj55 2023-10-17T19:09:53.491 DEBUG:boto:Method: GET 2023-10-17T19:09:53.491 DEBUG:boto:Path: /hbmuog-61/obj55 2023-10-17T19:09:53.491 DEBUG:boto:Data: 2023-10-17T19:09:53.492 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.493 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.493 DEBUG:boto:Params: {} 2023-10-17T19:09:53.494 DEBUG:boto:Token: None 2023-10-17T19:09:53.494 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj55 2023-10-17T19:09:53.495 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AY1DgIH+zvRQmPIVPnmbaOaxbYA= 2023-10-17T19:09:53.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AY1DgIH+zvRQmPIVPnmbaOaxbYA=', 'Content-Length': '0'} 2023-10-17T19:09:53.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce11fe66c68882d-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.498 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.498 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.499 DEBUG:boto:path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.499 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.499 DEBUG:boto:Method: GET 2023-10-17T19:09:53.500 DEBUG:boto:Path: /hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.500 DEBUG:boto:Data: 2023-10-17T19:09:53.501 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.501 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.502 DEBUG:boto:Params: {} 2023-10-17T19:09:53.502 DEBUG:boto:Token: None 2023-10-17T19:09:53.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.503 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17871bd6f079c81-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.507 DEBUG:boto:b'hbmuog-61obj551000falseobj552023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.507 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.508 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.508 DEBUG:boto:path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.509 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.509 DEBUG:boto:Method: GET 2023-10-17T19:09:53.509 DEBUG:boto:Path: /hbmuog-61/?prefix=obj55 2023-10-17T19:09:53.510 DEBUG:boto:Data: 2023-10-17T19:09:53.510 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.510 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.511 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.511 DEBUG:boto:Params: {} 2023-10-17T19:09:53.512 DEBUG:boto:Token: None 2023-10-17T19:09:53.512 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.513 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003013a733743e63ff-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.518 DEBUG:boto:b'hbmuog-61obj551000falseobj552023-10-17T19:06:19.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-17T19:09:53.519 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.519 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.520 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.520 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.521 DEBUG:boto:Method: GET 2023-10-17T19:09:53.521 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.522 DEBUG:boto:Data: 2023-10-17T19:09:53.522 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.523 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.523 DEBUG:boto:Params: {} 2023-10-17T19:09:53.524 DEBUG:boto:Token: None 2023-10-17T19:09:53.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY= 2023-10-17T19:09:53.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY=', 'Content-Length': '0'} 2023-10-17T19:09:53.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f777d85bdc63ca9-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.528 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.528 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.529 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.529 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.530 DEBUG:boto:Method: GET 2023-10-17T19:09:53.530 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.530 DEBUG:boto:Data: 2023-10-17T19:09:53.531 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.532 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.532 DEBUG:boto:Params: {} 2023-10-17T19:09:53.533 DEBUG:boto:Token: None 2023-10-17T19:09:53.533 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.534 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY= 2023-10-17T19:09:53.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY=', 'Content-Length': '0'} 2023-10-17T19:09:53.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225c3b1220e3efbe-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.578 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.579 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.579 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.580 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.580 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.580 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.581 DEBUG:boto:Data: 2023-10-17T19:09:53.581 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.582 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.582 DEBUG:boto:Params: {} 2023-10-17T19:09:53.583 DEBUG:boto:Token: None 2023-10-17T19:09:53.583 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.584 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jVfpnDHD0vQdfDLOsJZsMpZCuns= 2023-10-17T19:09:53.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jVfpnDHD0vQdfDLOsJZsMpZCuns=', 'Content-Length': '0'} 2023-10-17T19:09:53.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000450b2fc1aead1fd9-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.587 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.587 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.588 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.588 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.588 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.589 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.589 DEBUG:boto:Data: 2023-10-17T19:09:53.589 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.590 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.590 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.591 DEBUG:boto:Params: {} 2023-10-17T19:09:53.591 DEBUG:boto:Token: None 2023-10-17T19:09:53.591 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.592 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jVfpnDHD0vQdfDLOsJZsMpZCuns= 2023-10-17T19:09:53.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jVfpnDHD0vQdfDLOsJZsMpZCuns=', 'Content-Length': '0'} 2023-10-17T19:09:53.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0521636f91c598a-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-17T19:09:53.595 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.595 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.596 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.596 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.597 DEBUG:boto:Method: GET 2023-10-17T19:09:53.597 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.597 DEBUG:boto:Data: 2023-10-17T19:09:53.598 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.599 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.599 DEBUG:boto:Params: {} 2023-10-17T19:09:53.599 DEBUG:boto:Token: None 2023-10-17T19:09:53.600 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY= 2023-10-17T19:09:53.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY=', 'Content-Length': '0'} 2023-10-17T19:09:53.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9b083b5a06f5255-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.604 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.604 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.604 DEBUG:boto:path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.605 DEBUG:boto:auth_path=/hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.605 DEBUG:boto:Method: GET 2023-10-17T19:09:53.606 DEBUG:boto:Path: /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.606 DEBUG:boto:Data: 2023-10-17T19:09:53.606 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.607 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.608 DEBUG:boto:Params: {} 2023-10-17T19:09:53.608 DEBUG:boto:Token: None 2023-10-17T19:09:53.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57?versionId=null 2023-10-17T19:09:53.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY= 2023-10-17T19:09:53.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2g8NetOyUwMg1PZnWCQlNm3RLAY=', 'Content-Length': '0'} 2023-10-17T19:09:53.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e3a50eb946289b-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.613 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.613 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.613 DEBUG:boto:path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.614 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.614 DEBUG:boto:Method: GET 2023-10-17T19:09:53.615 DEBUG:boto:Path: /hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.615 DEBUG:boto:Data: 2023-10-17T19:09:53.616 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.616 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.616 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.617 DEBUG:boto:Params: {} 2023-10-17T19:09:53.617 DEBUG:boto:Token: None 2023-10-17T19:09:53.618 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.618 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba6091bd4ab5779-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.666 DEBUG:boto:b'hbmuog-61obj571000falseobj572023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.667 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.668 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.668 DEBUG:boto:path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.669 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.669 DEBUG:boto:Method: GET 2023-10-17T19:09:53.669 DEBUG:boto:Path: /hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.670 DEBUG:boto:Data: 2023-10-17T19:09:53.670 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.671 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.671 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.672 DEBUG:boto:Params: {} 2023-10-17T19:09:53.672 DEBUG:boto:Token: None 2023-10-17T19:09:53.673 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.673 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0b62c8e2d7b928-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.679 DEBUG:boto:b'hbmuog-61obj571000falseobj572023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.680 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.680 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.681 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.681 DEBUG:boto:Path: /hbmuog-61/obj57 2023-10-17T19:09:53.682 DEBUG:boto:Data: 2023-10-17T19:09:53.682 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.683 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.683 DEBUG:boto:Params: {} 2023-10-17T19:09:53.684 DEBUG:boto:Token: None 2023-10-17T19:09:53.684 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57 2023-10-17T19:09:53.685 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8+RKlJ/n3X19WEUsoYO+AbuKFl8= 2023-10-17T19:09:53.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8+RKlJ/n3X19WEUsoYO+AbuKFl8=', 'Content-Length': '0'} 2023-10-17T19:09:53.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df48b3a23ef389fe-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.688 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.688 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.688 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.689 DEBUG:boto:Path: /hbmuog-61/obj57 2023-10-17T19:09:53.689 DEBUG:boto:Data: 2023-10-17T19:09:53.690 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.690 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.691 DEBUG:boto:Params: {} 2023-10-17T19:09:53.691 DEBUG:boto:Token: None 2023-10-17T19:09:53.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57 2023-10-17T19:09:53.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8+RKlJ/n3X19WEUsoYO+AbuKFl8= 2023-10-17T19:09:53.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8+RKlJ/n3X19WEUsoYO+AbuKFl8=', 'Content-Length': '0'} 2023-10-17T19:09:53.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001913b19553913116-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-17T19:09:53.696 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.696 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.697 DEBUG:boto:Method: GET 2023-10-17T19:09:53.697 DEBUG:boto:Path: /hbmuog-61/obj57 2023-10-17T19:09:53.697 DEBUG:boto:Data: 2023-10-17T19:09:53.698 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.698 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.699 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.699 DEBUG:boto:Params: {} 2023-10-17T19:09:53.700 DEBUG:boto:Token: None 2023-10-17T19:09:53.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57 2023-10-17T19:09:53.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TkXZbrA0KWwbGhR6LLHGMhc7AA0= 2023-10-17T19:09:53.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TkXZbrA0KWwbGhR6LLHGMhc7AA0=', 'Content-Length': '0'} 2023-10-17T19:09:53.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b0024f2c11cdf3-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.704 DEBUG:boto:path=/hbmuog-61/obj57 2023-10-17T19:09:53.704 DEBUG:boto:auth_path=/hbmuog-61/obj57 2023-10-17T19:09:53.705 DEBUG:boto:Method: GET 2023-10-17T19:09:53.705 DEBUG:boto:Path: /hbmuog-61/obj57 2023-10-17T19:09:53.706 DEBUG:boto:Data: 2023-10-17T19:09:53.706 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.707 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.707 DEBUG:boto:Params: {} 2023-10-17T19:09:53.708 DEBUG:boto:Token: None 2023-10-17T19:09:53.708 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj57 2023-10-17T19:09:53.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TkXZbrA0KWwbGhR6LLHGMhc7AA0= 2023-10-17T19:09:53.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TkXZbrA0KWwbGhR6LLHGMhc7AA0=', 'Content-Length': '0'} 2023-10-17T19:09:53.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4650a41d9a4a56-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.754 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.755 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.755 DEBUG:boto:path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.756 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.756 DEBUG:boto:Method: GET 2023-10-17T19:09:53.757 DEBUG:boto:Path: /hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.757 DEBUG:boto:Data: 2023-10-17T19:09:53.758 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.758 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.758 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.759 DEBUG:boto:Params: {} 2023-10-17T19:09:53.759 DEBUG:boto:Token: None 2023-10-17T19:09:53.760 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.760 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79a68da964ce82d-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.764 DEBUG:boto:b'hbmuog-61obj571000falseobj572023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.765 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.765 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.765 DEBUG:boto:path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.766 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.766 DEBUG:boto:Method: GET 2023-10-17T19:09:53.767 DEBUG:boto:Path: /hbmuog-61/?prefix=obj57 2023-10-17T19:09:53.767 DEBUG:boto:Data: 2023-10-17T19:09:53.768 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.768 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.768 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.769 DEBUG:boto:Params: {} 2023-10-17T19:09:53.769 DEBUG:boto:Token: None 2023-10-17T19:09:53.770 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.770 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6369f079f2e9d5a-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.776 DEBUG:boto:b'hbmuog-61obj571000falseobj572023-10-17T19:06:20.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-17T19:09:53.777 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.777 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.778 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.778 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.779 DEBUG:boto:Method: GET 2023-10-17T19:09:53.779 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.780 DEBUG:boto:Data: 2023-10-17T19:09:53.780 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.780 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.781 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.781 DEBUG:boto:Params: {} 2023-10-17T19:09:53.782 DEBUG:boto:Token: None 2023-10-17T19:09:53.782 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.783 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0= 2023-10-17T19:09:53.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0=', 'Content-Length': '0'} 2023-10-17T19:09:53.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003499b99bde6b3458-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.826 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.827 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.827 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.828 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.828 DEBUG:boto:Method: GET 2023-10-17T19:09:53.829 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.829 DEBUG:boto:Data: 2023-10-17T19:09:53.830 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.830 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.830 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.831 DEBUG:boto:Params: {} 2023-10-17T19:09:53.831 DEBUG:boto:Token: None 2023-10-17T19:09:53.832 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.832 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0= 2023-10-17T19:09:53.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0=', 'Content-Length': '0'} 2023-10-17T19:09:53.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9699a21e2a0441-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.835 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.836 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.836 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.837 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.837 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.838 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.838 DEBUG:boto:Data: 2023-10-17T19:09:53.838 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.839 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.839 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.841 DEBUG:boto:Params: {} 2023-10-17T19:09:53.841 DEBUG:boto:Token: None 2023-10-17T19:09:53.842 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.842 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ef6Cq/ECu/rWbYI3Yqjrfy0aACM= 2023-10-17T19:09:53.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ef6Cq/ECu/rWbYI3Yqjrfy0aACM=', 'Content-Length': '0'} 2023-10-17T19:09:53.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bae3c37692f95a6-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.845 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.846 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.846 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.847 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.847 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.848 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.848 DEBUG:boto:Data: 2023-10-17T19:09:53.848 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.849 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.849 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.850 DEBUG:boto:Params: {} 2023-10-17T19:09:53.850 DEBUG:boto:Token: None 2023-10-17T19:09:53.850 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.851 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ef6Cq/ECu/rWbYI3Yqjrfy0aACM= 2023-10-17T19:09:53.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ef6Cq/ECu/rWbYI3Yqjrfy0aACM=', 'Content-Length': '0'} 2023-10-17T19:09:53.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d8890b3cb93d6e-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-17T19:09:53.855 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.855 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.855 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.856 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.856 DEBUG:boto:Method: GET 2023-10-17T19:09:53.857 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.857 DEBUG:boto:Data: 2023-10-17T19:09:53.858 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.858 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.858 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.859 DEBUG:boto:Params: {} 2023-10-17T19:09:53.859 DEBUG:boto:Token: None 2023-10-17T19:09:53.860 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.860 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0= 2023-10-17T19:09:53.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0=', 'Content-Length': '0'} 2023-10-17T19:09:53.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e1bb9fa0443a51-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.863 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.864 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.864 DEBUG:boto:path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.864 DEBUG:boto:auth_path=/hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.865 DEBUG:boto:Method: GET 2023-10-17T19:09:53.865 DEBUG:boto:Path: /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.866 DEBUG:boto:Data: 2023-10-17T19:09:53.866 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.867 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.867 DEBUG:boto:Params: {} 2023-10-17T19:09:53.868 DEBUG:boto:Token: None 2023-10-17T19:09:53.868 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58?versionId=null 2023-10-17T19:09:53.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0= 2023-10-17T19:09:53.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:thfIFsh3XnndejFyhy6kpuNYqF0=', 'Content-Length': '0'} 2023-10-17T19:09:53.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffb30c2489108c5-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.872 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.872 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.873 DEBUG:boto:path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.873 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.874 DEBUG:boto:Method: GET 2023-10-17T19:09:53.874 DEBUG:boto:Path: /hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.875 DEBUG:boto:Data: 2023-10-17T19:09:53.875 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.876 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.876 DEBUG:boto:Params: {} 2023-10-17T19:09:53.877 DEBUG:boto:Token: None 2023-10-17T19:09:53.877 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb2881afb42f301-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.881 DEBUG:boto:b'hbmuog-61obj581000falseobj582023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.882 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.882 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.883 DEBUG:boto:path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.883 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.884 DEBUG:boto:Method: GET 2023-10-17T19:09:53.884 DEBUG:boto:Path: /hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.885 DEBUG:boto:Data: 2023-10-17T19:09:53.885 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.885 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.886 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.886 DEBUG:boto:Params: {} 2023-10-17T19:09:53.887 DEBUG:boto:Token: None 2023-10-17T19:09:53.887 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef2e0495afb2533-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.893 DEBUG:boto:b'hbmuog-61obj581000falseobj582023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.894 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.894 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.894 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.895 DEBUG:boto:Path: /hbmuog-61/obj58 2023-10-17T19:09:53.895 DEBUG:boto:Data: 2023-10-17T19:09:53.896 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.896 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.896 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.897 DEBUG:boto:Params: {} 2023-10-17T19:09:53.897 DEBUG:boto:Token: None 2023-10-17T19:09:53.898 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58 2023-10-17T19:09:53.898 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6S+uslP63OIcgzH9rEdb5hJH+qo= 2023-10-17T19:09:53.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6S+uslP63OIcgzH9rEdb5hJH+qo=', 'Content-Length': '0'} 2023-10-17T19:09:53.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7f7772e0fb1f65-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.901 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.902 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.902 DEBUG:boto:Method: HEAD 2023-10-17T19:09:53.902 DEBUG:boto:Path: /hbmuog-61/obj58 2023-10-17T19:09:53.903 DEBUG:boto:Data: 2023-10-17T19:09:53.903 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.904 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.904 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.905 DEBUG:boto:Params: {} 2023-10-17T19:09:53.905 DEBUG:boto:Token: None 2023-10-17T19:09:53.906 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58 2023-10-17T19:09:53.906 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6S+uslP63OIcgzH9rEdb5hJH+qo= 2023-10-17T19:09:53.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6S+uslP63OIcgzH9rEdb5hJH+qo=', 'Content-Length': '0'} 2023-10-17T19:09:53.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b102e1d9a46a5820-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-17T19:09:53.910 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.910 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.911 DEBUG:boto:Method: GET 2023-10-17T19:09:53.911 DEBUG:boto:Path: /hbmuog-61/obj58 2023-10-17T19:09:53.911 DEBUG:boto:Data: 2023-10-17T19:09:53.912 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.912 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.913 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.913 DEBUG:boto:Params: {} 2023-10-17T19:09:53.914 DEBUG:boto:Token: None 2023-10-17T19:09:53.914 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58 2023-10-17T19:09:53.915 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bjiIMhtuN0Wh2wZNUJBWxEWczJg= 2023-10-17T19:09:53.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bjiIMhtuN0Wh2wZNUJBWxEWczJg=', 'Content-Length': '0'} 2023-10-17T19:09:53.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056eac7e8c42b641a-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.918 DEBUG:boto:path=/hbmuog-61/obj58 2023-10-17T19:09:53.918 DEBUG:boto:auth_path=/hbmuog-61/obj58 2023-10-17T19:09:53.919 DEBUG:boto:Method: GET 2023-10-17T19:09:53.919 DEBUG:boto:Path: /hbmuog-61/obj58 2023-10-17T19:09:53.919 DEBUG:boto:Data: 2023-10-17T19:09:53.920 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.920 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.921 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.921 DEBUG:boto:Params: {} 2023-10-17T19:09:53.922 DEBUG:boto:Token: None 2023-10-17T19:09:53.922 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj58 2023-10-17T19:09:53.923 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bjiIMhtuN0Wh2wZNUJBWxEWczJg= 2023-10-17T19:09:53.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bjiIMhtuN0Wh2wZNUJBWxEWczJg=', 'Content-Length': '0'} 2023-10-17T19:09:53.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1400fcd049d3b42-00652edc01-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.966 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.967 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.967 DEBUG:boto:path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.968 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.968 DEBUG:boto:Method: GET 2023-10-17T19:09:53.969 DEBUG:boto:Path: /hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.969 DEBUG:boto:Data: 2023-10-17T19:09:53.969 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.970 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.971 DEBUG:boto:Params: {} 2023-10-17T19:09:53.971 DEBUG:boto:Token: None 2023-10-17T19:09:53.972 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c4abe786d4f18b-00652edc01-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.976 DEBUG:boto:b'hbmuog-61obj581000falseobj582023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.977 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:53.978 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:53.978 DEBUG:boto:path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.978 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.979 DEBUG:boto:Method: GET 2023-10-17T19:09:53.979 DEBUG:boto:Path: /hbmuog-61/?prefix=obj58 2023-10-17T19:09:53.980 DEBUG:boto:Data: 2023-10-17T19:09:53.980 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.981 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:53.981 DEBUG:boto:Port: 8000 2023-10-17T19:09:53.981 DEBUG:boto:Params: {} 2023-10-17T19:09:53.982 DEBUG:boto:Token: None 2023-10-17T19:09:53.982 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/ 2023-10-17T19:09:53.983 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos= 2023-10-17T19:09:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lq43ZzRXqVdQ0aVwHI101gjDHos=', 'Content-Length': '0'} 2023-10-17T19:09:53.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b09d44e38a1cb2-00652edc01-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:53.988 DEBUG:boto:b'hbmuog-61obj581000falseobj582023-10-17T19:06:21.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:53.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-17T19:09:53.989 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:53.990 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:53.990 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:53.991 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:53.991 DEBUG:boto:Method: GET 2023-10-17T19:09:53.991 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:53.992 DEBUG:boto:Data: 2023-10-17T19:09:53.992 DEBUG:boto:Headers: {} 2023-10-17T19:09:53.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:53.993 DEBUG:boto:Port: 8001 2023-10-17T19:09:53.994 DEBUG:boto:Params: {} 2023-10-17T19:09:53.994 DEBUG:boto:Token: None 2023-10-17T19:09:53.995 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:53 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:53.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eGFx+bZjrWzDc5TSZlxVeXthlKc= 2023-10-17T19:09:53.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eGFx+bZjrWzDc5TSZlxVeXthlKc=', 'Content-Length': '0'} 2023-10-17T19:09:53.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae2424aad570c19-00652edc01-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.042 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.043 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.043 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.044 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.044 DEBUG:boto:Method: GET 2023-10-17T19:09:54.045 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.045 DEBUG:boto:Data: 2023-10-17T19:09:54.045 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.046 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.047 DEBUG:boto:Params: {} 2023-10-17T19:09:54.047 DEBUG:boto:Token: None 2023-10-17T19:09:54.048 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y= 2023-10-17T19:09:54.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y=', 'Content-Length': '0'} 2023-10-17T19:09:54.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b28cd41bad57f2e-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.052 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.052 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.052 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.053 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.053 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.054 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.054 DEBUG:boto:Data: 2023-10-17T19:09:54.055 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.055 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.056 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.056 DEBUG:boto:Params: {} 2023-10-17T19:09:54.056 DEBUG:boto:Token: None 2023-10-17T19:09:54.057 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.057 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MJCd2N2SL2wbL3m+delJ1xV956U= 2023-10-17T19:09:54.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MJCd2N2SL2wbL3m+delJ1xV956U=', 'Content-Length': '0'} 2023-10-17T19:09:54.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc754093f9f43bdf-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.061 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.061 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.061 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.062 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.062 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.063 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.063 DEBUG:boto:Data: 2023-10-17T19:09:54.063 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.064 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.064 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.065 DEBUG:boto:Params: {} 2023-10-17T19:09:54.065 DEBUG:boto:Token: None 2023-10-17T19:09:54.065 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.066 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MJCd2N2SL2wbL3m+delJ1xV956U= 2023-10-17T19:09:54.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MJCd2N2SL2wbL3m+delJ1xV956U=', 'Content-Length': '0'} 2023-10-17T19:09:54.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009fdd76bc89c9378-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-17T19:09:54.069 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.070 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.070 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.070 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.071 DEBUG:boto:Method: GET 2023-10-17T19:09:54.071 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.072 DEBUG:boto:Data: 2023-10-17T19:09:54.072 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.073 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.073 DEBUG:boto:Params: {} 2023-10-17T19:09:54.074 DEBUG:boto:Token: None 2023-10-17T19:09:54.074 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.075 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y= 2023-10-17T19:09:54.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y=', 'Content-Length': '0'} 2023-10-17T19:09:54.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f71fcb6cb4609fc-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.078 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.078 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.079 DEBUG:boto:path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.079 DEBUG:boto:auth_path=/hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.079 DEBUG:boto:Method: GET 2023-10-17T19:09:54.080 DEBUG:boto:Path: /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.080 DEBUG:boto:Data: 2023-10-17T19:09:54.081 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.081 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.082 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.082 DEBUG:boto:Params: {} 2023-10-17T19:09:54.082 DEBUG:boto:Token: None 2023-10-17T19:09:54.083 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62?versionId=null 2023-10-17T19:09:54.083 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y= 2023-10-17T19:09:54.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lAI74iIHon1sUgDsoi3dVz8u26Y=', 'Content-Length': '0'} 2023-10-17T19:09:54.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042245815b795c127-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.086 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.087 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.087 DEBUG:boto:path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.088 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.088 DEBUG:boto:Method: GET 2023-10-17T19:09:54.088 DEBUG:boto:Path: /hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.089 DEBUG:boto:Data: 2023-10-17T19:09:54.089 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.090 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.091 DEBUG:boto:Params: {} 2023-10-17T19:09:54.091 DEBUG:boto:Token: None 2023-10-17T19:09:54.092 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d1371eb3b14090-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.096 DEBUG:boto:b'hbmuog-61obj621000falseobj622023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.096 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.097 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.097 DEBUG:boto:path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.098 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.098 DEBUG:boto:Method: GET 2023-10-17T19:09:54.099 DEBUG:boto:Path: /hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.099 DEBUG:boto:Data: 2023-10-17T19:09:54.099 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.100 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.101 DEBUG:boto:Params: {} 2023-10-17T19:09:54.101 DEBUG:boto:Token: None 2023-10-17T19:09:54.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909c8b16e86f5986-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.107 DEBUG:boto:b'hbmuog-61obj621000falseobj622023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.108 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.108 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.109 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.109 DEBUG:boto:Path: /hbmuog-61/obj62 2023-10-17T19:09:54.110 DEBUG:boto:Data: 2023-10-17T19:09:54.110 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.111 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.111 DEBUG:boto:Params: {} 2023-10-17T19:09:54.112 DEBUG:boto:Token: None 2023-10-17T19:09:54.112 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62 2023-10-17T19:09:54.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9UkB3gnmmryJn6KIn9bimw0bSH4= 2023-10-17T19:09:54.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9UkB3gnmmryJn6KIn9bimw0bSH4=', 'Content-Length': '0'} 2023-10-17T19:09:54.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000458b585ee88134be-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.116 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.116 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.116 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.117 DEBUG:boto:Path: /hbmuog-61/obj62 2023-10-17T19:09:54.117 DEBUG:boto:Data: 2023-10-17T19:09:54.118 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.118 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.118 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.119 DEBUG:boto:Params: {} 2023-10-17T19:09:54.119 DEBUG:boto:Token: None 2023-10-17T19:09:54.120 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62 2023-10-17T19:09:54.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9UkB3gnmmryJn6KIn9bimw0bSH4= 2023-10-17T19:09:54.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9UkB3gnmmryJn6KIn9bimw0bSH4=', 'Content-Length': '0'} 2023-10-17T19:09:54.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a686f55736122e-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-17T19:09:54.123 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.124 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.124 DEBUG:boto:Method: GET 2023-10-17T19:09:54.125 DEBUG:boto:Path: /hbmuog-61/obj62 2023-10-17T19:09:54.125 DEBUG:boto:Data: 2023-10-17T19:09:54.126 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.126 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.127 DEBUG:boto:Params: {} 2023-10-17T19:09:54.127 DEBUG:boto:Token: None 2023-10-17T19:09:54.128 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62 2023-10-17T19:09:54.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ocBSp5Q5DEChbGnqSri33eMz62Q= 2023-10-17T19:09:54.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ocBSp5Q5DEChbGnqSri33eMz62Q=', 'Content-Length': '0'} 2023-10-17T19:09:54.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc67e87be2041a69-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.174 DEBUG:boto:path=/hbmuog-61/obj62 2023-10-17T19:09:54.175 DEBUG:boto:auth_path=/hbmuog-61/obj62 2023-10-17T19:09:54.175 DEBUG:boto:Method: GET 2023-10-17T19:09:54.176 DEBUG:boto:Path: /hbmuog-61/obj62 2023-10-17T19:09:54.176 DEBUG:boto:Data: 2023-10-17T19:09:54.177 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.177 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.177 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.178 DEBUG:boto:Params: {} 2023-10-17T19:09:54.178 DEBUG:boto:Token: None 2023-10-17T19:09:54.179 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj62 2023-10-17T19:09:54.179 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ocBSp5Q5DEChbGnqSri33eMz62Q= 2023-10-17T19:09:54.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ocBSp5Q5DEChbGnqSri33eMz62Q=', 'Content-Length': '0'} 2023-10-17T19:09:54.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d59153611a6ad8-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.184 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.185 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.185 DEBUG:boto:path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.185 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.186 DEBUG:boto:Method: GET 2023-10-17T19:09:54.186 DEBUG:boto:Path: /hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.187 DEBUG:boto:Data: 2023-10-17T19:09:54.187 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.188 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.188 DEBUG:boto:Params: {} 2023-10-17T19:09:54.189 DEBUG:boto:Token: None 2023-10-17T19:09:54.189 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.190 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5341a0a6a655b88-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.193 DEBUG:boto:b'hbmuog-61obj621000falseobj622023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.194 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.195 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.195 DEBUG:boto:path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.196 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.196 DEBUG:boto:Method: GET 2023-10-17T19:09:54.196 DEBUG:boto:Path: /hbmuog-61/?prefix=obj62 2023-10-17T19:09:54.197 DEBUG:boto:Data: 2023-10-17T19:09:54.197 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.198 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.198 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.198 DEBUG:boto:Params: {} 2023-10-17T19:09:54.199 DEBUG:boto:Token: None 2023-10-17T19:09:54.199 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.200 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047bf325f940cbe4b-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.206 DEBUG:boto:b'hbmuog-61obj621000falseobj622023-10-17T19:06:21.401Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T19:09:54.207 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.207 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.208 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.208 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.208 DEBUG:boto:Method: GET 2023-10-17T19:09:54.209 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.209 DEBUG:boto:Data: 2023-10-17T19:09:54.210 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.210 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.211 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.211 DEBUG:boto:Params: {} 2023-10-17T19:09:54.211 DEBUG:boto:Token: None 2023-10-17T19:09:54.212 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.212 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE= 2023-10-17T19:09:54.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE=', 'Content-Length': '0'} 2023-10-17T19:09:54.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006019c115754139e2-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.216 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.216 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.216 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.217 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.217 DEBUG:boto:Method: GET 2023-10-17T19:09:54.218 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.218 DEBUG:boto:Data: 2023-10-17T19:09:54.219 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.219 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.219 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.220 DEBUG:boto:Params: {} 2023-10-17T19:09:54.220 DEBUG:boto:Token: None 2023-10-17T19:09:54.221 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.221 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE= 2023-10-17T19:09:54.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE=', 'Content-Length': '0'} 2023-10-17T19:09:54.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd39f62ffbf1e34-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.224 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.225 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.225 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.225 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.226 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.226 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.227 DEBUG:boto:Data: 2023-10-17T19:09:54.227 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.227 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.228 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.228 DEBUG:boto:Params: {} 2023-10-17T19:09:54.229 DEBUG:boto:Token: None 2023-10-17T19:09:54.229 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.230 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Io2Pzg4IeKjznwtQCHCNR/asBdY= 2023-10-17T19:09:54.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Io2Pzg4IeKjznwtQCHCNR/asBdY=', 'Content-Length': '0'} 2023-10-17T19:09:54.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cb47e4988cd7305-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.232 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.233 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.233 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.234 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.234 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.234 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.235 DEBUG:boto:Data: 2023-10-17T19:09:54.235 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.236 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.236 DEBUG:boto:Params: {} 2023-10-17T19:09:54.238 DEBUG:boto:Token: None 2023-10-17T19:09:54.238 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Io2Pzg4IeKjznwtQCHCNR/asBdY= 2023-10-17T19:09:54.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Io2Pzg4IeKjznwtQCHCNR/asBdY=', 'Content-Length': '0'} 2023-10-17T19:09:54.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836100989d5a87e2-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T19:09:54.242 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.243 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.243 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.243 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.244 DEBUG:boto:Method: GET 2023-10-17T19:09:54.244 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.245 DEBUG:boto:Data: 2023-10-17T19:09:54.245 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.246 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.246 DEBUG:boto:Params: {} 2023-10-17T19:09:54.247 DEBUG:boto:Token: None 2023-10-17T19:09:54.247 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.248 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE= 2023-10-17T19:09:54.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE=', 'Content-Length': '0'} 2023-10-17T19:09:54.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c47b7b4dfbf252a-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.251 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.251 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.252 DEBUG:boto:path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.253 DEBUG:boto:auth_path=/hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.253 DEBUG:boto:Method: GET 2023-10-17T19:09:54.254 DEBUG:boto:Path: /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.254 DEBUG:boto:Data: 2023-10-17T19:09:54.255 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.256 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.256 DEBUG:boto:Params: {} 2023-10-17T19:09:54.257 DEBUG:boto:Token: None 2023-10-17T19:09:54.257 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68?versionId=null 2023-10-17T19:09:54.258 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE= 2023-10-17T19:09:54.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GUE8nsd89qzDgoBQ2ckRyaiAmFE=', 'Content-Length': '0'} 2023-10-17T19:09:54.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea276297d6dc591-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.261 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.262 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.262 DEBUG:boto:path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.263 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.263 DEBUG:boto:Method: GET 2023-10-17T19:09:54.264 DEBUG:boto:Path: /hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.264 DEBUG:boto:Data: 2023-10-17T19:09:54.265 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.265 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.266 DEBUG:boto:Params: {} 2023-10-17T19:09:54.266 DEBUG:boto:Token: None 2023-10-17T19:09:54.267 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a254207bf8f38b8a-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.314 DEBUG:boto:b'hbmuog-61obj681000falseobj682023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.315 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.316 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.316 DEBUG:boto:path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.316 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.317 DEBUG:boto:Method: GET 2023-10-17T19:09:54.317 DEBUG:boto:Path: /hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.318 DEBUG:boto:Data: 2023-10-17T19:09:54.318 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.319 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.319 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.319 DEBUG:boto:Params: {} 2023-10-17T19:09:54.320 DEBUG:boto:Token: None 2023-10-17T19:09:54.320 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.321 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b03f439d51ea420-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.326 DEBUG:boto:b'hbmuog-61obj681000falseobj682023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.327 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.327 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.328 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.328 DEBUG:boto:Path: /hbmuog-61/obj68 2023-10-17T19:09:54.329 DEBUG:boto:Data: 2023-10-17T19:09:54.329 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.330 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.330 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.330 DEBUG:boto:Params: {} 2023-10-17T19:09:54.331 DEBUG:boto:Token: None 2023-10-17T19:09:54.331 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68 2023-10-17T19:09:54.332 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pi9vjMDpPaXveCtrlBaysY5vhY8= 2023-10-17T19:09:54.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pi9vjMDpPaXveCtrlBaysY5vhY8=', 'Content-Length': '0'} 2023-10-17T19:09:54.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000790b5433c95f7c6-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.335 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.335 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.335 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.336 DEBUG:boto:Path: /hbmuog-61/obj68 2023-10-17T19:09:54.336 DEBUG:boto:Data: 2023-10-17T19:09:54.337 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.338 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.338 DEBUG:boto:Params: {} 2023-10-17T19:09:54.338 DEBUG:boto:Token: None 2023-10-17T19:09:54.339 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68 2023-10-17T19:09:54.339 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pi9vjMDpPaXveCtrlBaysY5vhY8= 2023-10-17T19:09:54.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pi9vjMDpPaXveCtrlBaysY5vhY8=', 'Content-Length': '0'} 2023-10-17T19:09:54.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f6c79d577ca9c6-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-17T19:09:54.343 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.343 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.343 DEBUG:boto:Method: GET 2023-10-17T19:09:54.344 DEBUG:boto:Path: /hbmuog-61/obj68 2023-10-17T19:09:54.344 DEBUG:boto:Data: 2023-10-17T19:09:54.345 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.345 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.345 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.346 DEBUG:boto:Params: {} 2023-10-17T19:09:54.346 DEBUG:boto:Token: None 2023-10-17T19:09:54.347 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68 2023-10-17T19:09:54.347 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g/9D0rHvAHvVZSXpBrgUyAOY3Hs= 2023-10-17T19:09:54.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g/9D0rHvAHvVZSXpBrgUyAOY3Hs=', 'Content-Length': '0'} 2023-10-17T19:09:54.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000050401b40007e0d3-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.350 DEBUG:boto:path=/hbmuog-61/obj68 2023-10-17T19:09:54.351 DEBUG:boto:auth_path=/hbmuog-61/obj68 2023-10-17T19:09:54.351 DEBUG:boto:Method: GET 2023-10-17T19:09:54.352 DEBUG:boto:Path: /hbmuog-61/obj68 2023-10-17T19:09:54.352 DEBUG:boto:Data: 2023-10-17T19:09:54.353 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.353 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.354 DEBUG:boto:Params: {} 2023-10-17T19:09:54.354 DEBUG:boto:Token: None 2023-10-17T19:09:54.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj68 2023-10-17T19:09:54.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g/9D0rHvAHvVZSXpBrgUyAOY3Hs= 2023-10-17T19:09:54.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g/9D0rHvAHvVZSXpBrgUyAOY3Hs=', 'Content-Length': '0'} 2023-10-17T19:09:54.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064154e8bd55bc5b9-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.359 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.359 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.359 DEBUG:boto:path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.360 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.360 DEBUG:boto:Method: GET 2023-10-17T19:09:54.361 DEBUG:boto:Path: /hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.361 DEBUG:boto:Data: 2023-10-17T19:09:54.362 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.362 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.363 DEBUG:boto:Params: {} 2023-10-17T19:09:54.363 DEBUG:boto:Token: None 2023-10-17T19:09:54.364 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004caee4db29ab8d8e-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.368 DEBUG:boto:b'hbmuog-61obj681000falseobj682023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.369 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.369 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.369 DEBUG:boto:path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.370 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.370 DEBUG:boto:Method: GET 2023-10-17T19:09:54.371 DEBUG:boto:Path: /hbmuog-61/?prefix=obj68 2023-10-17T19:09:54.371 DEBUG:boto:Data: 2023-10-17T19:09:54.372 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.372 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.373 DEBUG:boto:Params: {} 2023-10-17T19:09:54.373 DEBUG:boto:Token: None 2023-10-17T19:09:54.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014cf407c11e86ed6-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.379 DEBUG:boto:b'hbmuog-61obj681000falseobj682023-10-17T19:06:20.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-17T19:09:54.380 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.381 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.381 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.382 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.382 DEBUG:boto:Method: GET 2023-10-17T19:09:54.383 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.383 DEBUG:boto:Data: 2023-10-17T19:09:54.384 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.385 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.385 DEBUG:boto:Params: {} 2023-10-17T19:09:54.386 DEBUG:boto:Token: None 2023-10-17T19:09:54.386 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw= 2023-10-17T19:09:54.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw=', 'Content-Length': '0'} 2023-10-17T19:09:54.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c2cc1c48c18f571-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.390 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.390 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.391 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.391 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.391 DEBUG:boto:Method: GET 2023-10-17T19:09:54.392 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.392 DEBUG:boto:Data: 2023-10-17T19:09:54.393 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.393 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.394 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.394 DEBUG:boto:Params: {} 2023-10-17T19:09:54.394 DEBUG:boto:Token: None 2023-10-17T19:09:54.395 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.395 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw= 2023-10-17T19:09:54.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw=', 'Content-Length': '0'} 2023-10-17T19:09:54.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d24bd009992b7c-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.438 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.439 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.439 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.440 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.440 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.441 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.441 DEBUG:boto:Data: 2023-10-17T19:09:54.442 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.442 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.443 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.443 DEBUG:boto:Params: {} 2023-10-17T19:09:54.444 DEBUG:boto:Token: None 2023-10-17T19:09:54.444 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.445 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uivX6YugHx6twzqDGZv/EhuDt0g= 2023-10-17T19:09:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uivX6YugHx6twzqDGZv/EhuDt0g=', 'Content-Length': '0'} 2023-10-17T19:09:54.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0fb8600fe4a95ba-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.450 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.450 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.451 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.453 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.453 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.454 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.454 DEBUG:boto:Data: 2023-10-17T19:09:54.455 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.456 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.456 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.457 DEBUG:boto:Params: {} 2023-10-17T19:09:54.458 DEBUG:boto:Token: None 2023-10-17T19:09:54.459 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.459 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uivX6YugHx6twzqDGZv/EhuDt0g= 2023-10-17T19:09:54.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uivX6YugHx6twzqDGZv/EhuDt0g=', 'Content-Length': '0'} 2023-10-17T19:09:54.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a8a80604bbed3d-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-17T19:09:54.463 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.463 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.464 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.464 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.465 DEBUG:boto:Method: GET 2023-10-17T19:09:54.465 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.466 DEBUG:boto:Data: 2023-10-17T19:09:54.467 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.467 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.468 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.468 DEBUG:boto:Params: {} 2023-10-17T19:09:54.469 DEBUG:boto:Token: None 2023-10-17T19:09:54.470 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.470 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw= 2023-10-17T19:09:54.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw=', 'Content-Length': '0'} 2023-10-17T19:09:54.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43328c184a83695-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.474 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.475 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.475 DEBUG:boto:path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.476 DEBUG:boto:auth_path=/hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.476 DEBUG:boto:Method: GET 2023-10-17T19:09:54.476 DEBUG:boto:Path: /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.477 DEBUG:boto:Data: 2023-10-17T19:09:54.478 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.478 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.479 DEBUG:boto:Params: {} 2023-10-17T19:09:54.480 DEBUG:boto:Token: None 2023-10-17T19:09:54.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69?versionId=null 2023-10-17T19:09:54.481 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw= 2023-10-17T19:09:54.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ulLXSeA5kK4Qasz//VB6EloETnw=', 'Content-Length': '0'} 2023-10-17T19:09:54.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6348dbd8fd59fd-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.484 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.484 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.485 DEBUG:boto:path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.485 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.486 DEBUG:boto:Method: GET 2023-10-17T19:09:54.486 DEBUG:boto:Path: /hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.486 DEBUG:boto:Data: 2023-10-17T19:09:54.487 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.488 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.488 DEBUG:boto:Params: {} 2023-10-17T19:09:54.489 DEBUG:boto:Token: None 2023-10-17T19:09:54.489 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.490 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e3f7b2c9a6e830-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.493 DEBUG:boto:b'hbmuog-61obj691000falseobj692023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.494 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.494 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.495 DEBUG:boto:path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.495 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.495 DEBUG:boto:Method: GET 2023-10-17T19:09:54.496 DEBUG:boto:Path: /hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.496 DEBUG:boto:Data: 2023-10-17T19:09:54.497 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.497 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.498 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.498 DEBUG:boto:Params: {} 2023-10-17T19:09:54.498 DEBUG:boto:Token: None 2023-10-17T19:09:54.499 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.499 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf8aa2908f7f115-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.504 DEBUG:boto:b'hbmuog-61obj691000falseobj692023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.505 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.505 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.506 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.506 DEBUG:boto:Path: /hbmuog-61/obj69 2023-10-17T19:09:54.507 DEBUG:boto:Data: 2023-10-17T19:09:54.507 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.508 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.508 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.508 DEBUG:boto:Params: {} 2023-10-17T19:09:54.509 DEBUG:boto:Token: None 2023-10-17T19:09:54.509 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69 2023-10-17T19:09:54.510 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T99qXMecKNnrzVXv22RBIzZ82Zg= 2023-10-17T19:09:54.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T99qXMecKNnrzVXv22RBIzZ82Zg=', 'Content-Length': '0'} 2023-10-17T19:09:54.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0faec83ef93616a-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.513 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.513 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.514 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.514 DEBUG:boto:Path: /hbmuog-61/obj69 2023-10-17T19:09:54.514 DEBUG:boto:Data: 2023-10-17T19:09:54.515 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.516 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.516 DEBUG:boto:Params: {} 2023-10-17T19:09:54.516 DEBUG:boto:Token: None 2023-10-17T19:09:54.517 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69 2023-10-17T19:09:54.517 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:T99qXMecKNnrzVXv22RBIzZ82Zg= 2023-10-17T19:09:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:T99qXMecKNnrzVXv22RBIzZ82Zg=', 'Content-Length': '0'} 2023-10-17T19:09:54.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dc9ae3dd08d4ad-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-17T19:09:54.521 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.521 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.522 DEBUG:boto:Method: GET 2023-10-17T19:09:54.522 DEBUG:boto:Path: /hbmuog-61/obj69 2023-10-17T19:09:54.523 DEBUG:boto:Data: 2023-10-17T19:09:54.523 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.524 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.524 DEBUG:boto:Params: {} 2023-10-17T19:09:54.525 DEBUG:boto:Token: None 2023-10-17T19:09:54.525 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69 2023-10-17T19:09:54.526 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/MAe2ApvdOoCpLxXYgnDuTz3YCo= 2023-10-17T19:09:54.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/MAe2ApvdOoCpLxXYgnDuTz3YCo=', 'Content-Length': '0'} 2023-10-17T19:09:54.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d586f4a5c837157f-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.529 DEBUG:boto:path=/hbmuog-61/obj69 2023-10-17T19:09:54.529 DEBUG:boto:auth_path=/hbmuog-61/obj69 2023-10-17T19:09:54.530 DEBUG:boto:Method: GET 2023-10-17T19:09:54.530 DEBUG:boto:Path: /hbmuog-61/obj69 2023-10-17T19:09:54.531 DEBUG:boto:Data: 2023-10-17T19:09:54.531 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.531 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.532 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.532 DEBUG:boto:Params: {} 2023-10-17T19:09:54.533 DEBUG:boto:Token: None 2023-10-17T19:09:54.533 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj69 2023-10-17T19:09:54.534 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/MAe2ApvdOoCpLxXYgnDuTz3YCo= 2023-10-17T19:09:54.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/MAe2ApvdOoCpLxXYgnDuTz3YCo=', 'Content-Length': '0'} 2023-10-17T19:09:54.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baddab9272e25cb6-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.578 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.579 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.580 DEBUG:boto:path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.580 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.580 DEBUG:boto:Method: GET 2023-10-17T19:09:54.581 DEBUG:boto:Path: /hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.581 DEBUG:boto:Data: 2023-10-17T19:09:54.582 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.582 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.583 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.583 DEBUG:boto:Params: {} 2023-10-17T19:09:54.584 DEBUG:boto:Token: None 2023-10-17T19:09:54.584 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.585 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c4ea072eaeeba7-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.589 DEBUG:boto:b'hbmuog-61obj691000falseobj692023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.590 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.591 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.591 DEBUG:boto:path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.592 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.593 DEBUG:boto:Method: GET 2023-10-17T19:09:54.593 DEBUG:boto:Path: /hbmuog-61/?prefix=obj69 2023-10-17T19:09:54.594 DEBUG:boto:Data: 2023-10-17T19:09:54.594 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.595 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.595 DEBUG:boto:Params: {} 2023-10-17T19:09:54.596 DEBUG:boto:Token: None 2023-10-17T19:09:54.596 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.597 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe3af283a72bd91-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.602 DEBUG:boto:b'hbmuog-61obj691000falseobj692023-10-17T19:06:20.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-17T19:09:54.604 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.604 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.604 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.605 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.605 DEBUG:boto:Method: GET 2023-10-17T19:09:54.606 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.606 DEBUG:boto:Data: 2023-10-17T19:09:54.607 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.607 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.608 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.608 DEBUG:boto:Params: {} 2023-10-17T19:09:54.608 DEBUG:boto:Token: None 2023-10-17T19:09:54.609 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.610 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc= 2023-10-17T19:09:54.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc=', 'Content-Length': '0'} 2023-10-17T19:09:54.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a7de430944dd01-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.614 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.614 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.615 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.615 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.615 DEBUG:boto:Method: GET 2023-10-17T19:09:54.616 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.616 DEBUG:boto:Data: 2023-10-17T19:09:54.617 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.618 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.618 DEBUG:boto:Params: {} 2023-10-17T19:09:54.619 DEBUG:boto:Token: None 2023-10-17T19:09:54.619 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc= 2023-10-17T19:09:54.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc=', 'Content-Length': '0'} 2023-10-17T19:09:54.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091927f4c8b7f5279-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.623 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.623 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.624 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.624 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.625 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.625 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.625 DEBUG:boto:Data: 2023-10-17T19:09:54.626 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.627 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.627 DEBUG:boto:Params: {} 2023-10-17T19:09:54.628 DEBUG:boto:Token: None 2023-10-17T19:09:54.628 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EYlbwlMJzcdtdqikQSP1POBLe8Q= 2023-10-17T19:09:54.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EYlbwlMJzcdtdqikQSP1POBLe8Q=', 'Content-Length': '0'} 2023-10-17T19:09:54.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b64ec95d04ed26-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.632 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.632 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.632 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.633 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.633 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.634 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.634 DEBUG:boto:Data: 2023-10-17T19:09:54.634 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.635 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.635 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.636 DEBUG:boto:Params: {} 2023-10-17T19:09:54.636 DEBUG:boto:Token: None 2023-10-17T19:09:54.637 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EYlbwlMJzcdtdqikQSP1POBLe8Q= 2023-10-17T19:09:54.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EYlbwlMJzcdtdqikQSP1POBLe8Q=', 'Content-Length': '0'} 2023-10-17T19:09:54.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e3b4ce675b55f2-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-17T19:09:54.641 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.641 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.641 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.642 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.642 DEBUG:boto:Method: GET 2023-10-17T19:09:54.643 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.643 DEBUG:boto:Data: 2023-10-17T19:09:54.644 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.644 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.644 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.645 DEBUG:boto:Params: {} 2023-10-17T19:09:54.645 DEBUG:boto:Token: None 2023-10-17T19:09:54.646 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.646 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc= 2023-10-17T19:09:54.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc=', 'Content-Length': '0'} 2023-10-17T19:09:54.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b091735378903f-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.690 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.691 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.691 DEBUG:boto:path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.692 DEBUG:boto:auth_path=/hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.692 DEBUG:boto:Method: GET 2023-10-17T19:09:54.693 DEBUG:boto:Path: /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.693 DEBUG:boto:Data: 2023-10-17T19:09:54.694 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.694 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.695 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.695 DEBUG:boto:Params: {} 2023-10-17T19:09:54.695 DEBUG:boto:Token: None 2023-10-17T19:09:54.696 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74?versionId=null 2023-10-17T19:09:54.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc= 2023-10-17T19:09:54.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3iW6cC8W4Qn6QZN3ezROH/B8ffc=', 'Content-Length': '0'} 2023-10-17T19:09:54.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bef6ccc57e3dffb-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.700 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.701 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.701 DEBUG:boto:path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.701 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.702 DEBUG:boto:Method: GET 2023-10-17T19:09:54.702 DEBUG:boto:Path: /hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.703 DEBUG:boto:Data: 2023-10-17T19:09:54.703 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.704 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.704 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.704 DEBUG:boto:Params: {} 2023-10-17T19:09:54.705 DEBUG:boto:Token: None 2023-10-17T19:09:54.705 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.706 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f523e16db8be4162-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.710 DEBUG:boto:b'hbmuog-61obj741000falseobj742023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.711 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.711 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.711 DEBUG:boto:path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.712 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.712 DEBUG:boto:Method: GET 2023-10-17T19:09:54.713 DEBUG:boto:Path: /hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.713 DEBUG:boto:Data: 2023-10-17T19:09:54.714 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.714 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.715 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.715 DEBUG:boto:Params: {} 2023-10-17T19:09:54.716 DEBUG:boto:Token: None 2023-10-17T19:09:54.716 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006502752d74add781-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.762 DEBUG:boto:b'hbmuog-61obj741000falseobj742023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.763 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.764 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.764 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.764 DEBUG:boto:Path: /hbmuog-61/obj74 2023-10-17T19:09:54.765 DEBUG:boto:Data: 2023-10-17T19:09:54.765 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.766 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.767 DEBUG:boto:Params: {} 2023-10-17T19:09:54.767 DEBUG:boto:Token: None 2023-10-17T19:09:54.768 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74 2023-10-17T19:09:54.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P3qJt93FratqgHMdVpfzfiDpAFY= 2023-10-17T19:09:54.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P3qJt93FratqgHMdVpfzfiDpAFY=', 'Content-Length': '0'} 2023-10-17T19:09:54.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076cebba621c9b07-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.771 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.772 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.772 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.772 DEBUG:boto:Path: /hbmuog-61/obj74 2023-10-17T19:09:54.773 DEBUG:boto:Data: 2023-10-17T19:09:54.773 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.774 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.774 DEBUG:boto:Params: {} 2023-10-17T19:09:54.775 DEBUG:boto:Token: None 2023-10-17T19:09:54.775 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74 2023-10-17T19:09:54.776 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:P3qJt93FratqgHMdVpfzfiDpAFY= 2023-10-17T19:09:54.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:P3qJt93FratqgHMdVpfzfiDpAFY=', 'Content-Length': '0'} 2023-10-17T19:09:54.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddc6380989fcc4dd-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-17T19:09:54.779 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.780 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.780 DEBUG:boto:Method: GET 2023-10-17T19:09:54.780 DEBUG:boto:Path: /hbmuog-61/obj74 2023-10-17T19:09:54.781 DEBUG:boto:Data: 2023-10-17T19:09:54.781 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.782 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.782 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.783 DEBUG:boto:Params: {} 2023-10-17T19:09:54.783 DEBUG:boto:Token: None 2023-10-17T19:09:54.784 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74 2023-10-17T19:09:54.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gY7tYNVkUYHxZdEZFR0jkBlc5Ic= 2023-10-17T19:09:54.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gY7tYNVkUYHxZdEZFR0jkBlc5Ic=', 'Content-Length': '0'} 2023-10-17T19:09:54.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e9f249b69bee7c-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.787 DEBUG:boto:path=/hbmuog-61/obj74 2023-10-17T19:09:54.788 DEBUG:boto:auth_path=/hbmuog-61/obj74 2023-10-17T19:09:54.788 DEBUG:boto:Method: GET 2023-10-17T19:09:54.789 DEBUG:boto:Path: /hbmuog-61/obj74 2023-10-17T19:09:54.789 DEBUG:boto:Data: 2023-10-17T19:09:54.790 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.790 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.790 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.791 DEBUG:boto:Params: {} 2023-10-17T19:09:54.791 DEBUG:boto:Token: None 2023-10-17T19:09:54.792 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj74 2023-10-17T19:09:54.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gY7tYNVkUYHxZdEZFR0jkBlc5Ic= 2023-10-17T19:09:54.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gY7tYNVkUYHxZdEZFR0jkBlc5Ic=', 'Content-Length': '0'} 2023-10-17T19:09:54.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e605335726ecb8c-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.796 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.796 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.797 DEBUG:boto:path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.797 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.798 DEBUG:boto:Method: GET 2023-10-17T19:09:54.798 DEBUG:boto:Path: /hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.799 DEBUG:boto:Data: 2023-10-17T19:09:54.799 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.799 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.800 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.800 DEBUG:boto:Params: {} 2023-10-17T19:09:54.801 DEBUG:boto:Token: None 2023-10-17T19:09:54.801 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.802 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f6a8174d5090f2a-00652edc02-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.850 DEBUG:boto:b'hbmuog-61obj741000falseobj742023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.851 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:54.852 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:54.852 DEBUG:boto:path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.853 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.853 DEBUG:boto:Method: GET 2023-10-17T19:09:54.853 DEBUG:boto:Path: /hbmuog-61/?prefix=obj74 2023-10-17T19:09:54.854 DEBUG:boto:Data: 2023-10-17T19:09:54.854 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.855 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.855 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.856 DEBUG:boto:Params: {} 2023-10-17T19:09:54.856 DEBUG:boto:Token: None 2023-10-17T19:09:54.857 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/ 2023-10-17T19:09:54.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs= 2023-10-17T19:09:54.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CLXzKgNLfi4QoH4dsruF/ECzCKs=', 'Content-Length': '0'} 2023-10-17T19:09:54.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce6e4f1a1e608fbf-00652edc02-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.866 DEBUG:boto:b'hbmuog-61obj741000falseobj742023-10-17T19:06:21.757Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:54.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-17T19:09:54.867 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:54.867 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:54.868 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.868 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.869 DEBUG:boto:Method: GET 2023-10-17T19:09:54.869 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.870 DEBUG:boto:Data: 2023-10-17T19:09:54.870 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.870 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.871 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.871 DEBUG:boto:Params: {} 2023-10-17T19:09:54.872 DEBUG:boto:Token: None 2023-10-17T19:09:54.872 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.873 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g59V8ORve+HLLyVG5CE64Hz2woc= 2023-10-17T19:09:54.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g59V8ORve+HLLyVG5CE64Hz2woc=', 'Content-Length': '0'} 2023-10-17T19:09:54.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d67ed412e98a84-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.876 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:54.877 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:54.877 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.877 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.878 DEBUG:boto:Method: GET 2023-10-17T19:09:54.878 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.879 DEBUG:boto:Data: 2023-10-17T19:09:54.879 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.880 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.880 DEBUG:boto:Params: {} 2023-10-17T19:09:54.881 DEBUG:boto:Token: None 2023-10-17T19:09:54.881 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g59V8ORve+HLLyVG5CE64Hz2woc= 2023-10-17T19:09:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g59V8ORve+HLLyVG5CE64Hz2woc=', 'Content-Length': '0'} 2023-10-17T19:09:54.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcf3467f9ba3697-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.926 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:54.927 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:54.927 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.928 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.928 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.929 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.929 DEBUG:boto:Data: 2023-10-17T19:09:54.929 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:54.930 DEBUG:boto:Port: 8001 2023-10-17T19:09:54.931 DEBUG:boto:Params: {} 2023-10-17T19:09:54.931 DEBUG:boto:Token: None 2023-10-17T19:09:54.932 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zCT5l+bcxb74kuQaw2SgdBaeLmI= 2023-10-17T19:09:54.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zCT5l+bcxb74kuQaw2SgdBaeLmI=', 'Content-Length': '0'} 2023-10-17T19:09:54.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb6e152d82e6f52-00652edc02-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.935 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:54.935 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:54.936 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.936 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.937 DEBUG:boto:Method: HEAD 2023-10-17T19:09:54.937 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.937 DEBUG:boto:Data: 2023-10-17T19:09:54.938 DEBUG:boto:Headers: {} 2023-10-17T19:09:54.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:54.939 DEBUG:boto:Port: 8000 2023-10-17T19:09:54.939 DEBUG:boto:Params: {} 2023-10-17T19:09:54.940 DEBUG:boto:Token: None 2023-10-17T19:09:54.940 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:54 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:54.941 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zCT5l+bcxb74kuQaw2SgdBaeLmI= 2023-10-17T19:09:54.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zCT5l+bcxb74kuQaw2SgdBaeLmI=', 'Content-Length': '0'} 2023-10-17T19:09:54.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb454524947fc02-00652edc02-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:54.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-17T19:09:54.944 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:54.945 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.014 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.015 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.015 DEBUG:boto:Method: GET 2023-10-17T19:09:55.016 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.016 DEBUG:boto:Data: 2023-10-17T19:09:55.016 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.017 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.018 DEBUG:boto:Params: {} 2023-10-17T19:09:55.018 DEBUG:boto:Token: None 2023-10-17T19:09:55.019 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UJBuotZVU0UjJmzpD2XShYjgzB4= 2023-10-17T19:09:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UJBuotZVU0UjJmzpD2XShYjgzB4=', 'Content-Length': '0'} 2023-10-17T19:09:55.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e647b244c0f653a4-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.023 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:55.023 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.023 DEBUG:boto:path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.024 DEBUG:boto:auth_path=/hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.024 DEBUG:boto:Method: GET 2023-10-17T19:09:55.025 DEBUG:boto:Path: /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.025 DEBUG:boto:Data: 2023-10-17T19:09:55.026 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.026 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.027 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.027 DEBUG:boto:Params: {} 2023-10-17T19:09:55.027 DEBUG:boto:Token: None 2023-10-17T19:09:55.028 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76?versionId=null 2023-10-17T19:09:55.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UJBuotZVU0UjJmzpD2XShYjgzB4= 2023-10-17T19:09:55.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UJBuotZVU0UjJmzpD2XShYjgzB4=', 'Content-Length': '0'} 2023-10-17T19:09:55.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c9a385e6f939de-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.032 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.033 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.033 DEBUG:boto:path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.033 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.034 DEBUG:boto:Method: GET 2023-10-17T19:09:55.034 DEBUG:boto:Path: /hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.035 DEBUG:boto:Data: 2023-10-17T19:09:55.035 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.036 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.036 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.036 DEBUG:boto:Params: {} 2023-10-17T19:09:55.037 DEBUG:boto:Token: None 2023-10-17T19:09:55.037 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.038 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305a86014cbe35d4-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.041 DEBUG:boto:b'hbmuog-61obj761000falseobj762023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.042 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.043 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.043 DEBUG:boto:path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.043 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.044 DEBUG:boto:Method: GET 2023-10-17T19:09:55.044 DEBUG:boto:Path: /hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.045 DEBUG:boto:Data: 2023-10-17T19:09:55.045 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.046 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.046 DEBUG:boto:Params: {} 2023-10-17T19:09:55.047 DEBUG:boto:Token: None 2023-10-17T19:09:55.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0136b459584f882-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.053 DEBUG:boto:b'hbmuog-61obj761000falseobj762023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.054 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:55.055 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.055 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.055 DEBUG:boto:Path: /hbmuog-61/obj76 2023-10-17T19:09:55.056 DEBUG:boto:Data: 2023-10-17T19:09:55.056 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.057 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.057 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.057 DEBUG:boto:Params: {} 2023-10-17T19:09:55.058 DEBUG:boto:Token: None 2023-10-17T19:09:55.058 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76 2023-10-17T19:09:55.059 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fJrAL7aIm3X4mLk/cIi/FGyDLDQ= 2023-10-17T19:09:55.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fJrAL7aIm3X4mLk/cIi/FGyDLDQ=', 'Content-Length': '0'} 2023-10-17T19:09:55.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f355837f99060d16-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.062 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:55.063 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.063 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.063 DEBUG:boto:Path: /hbmuog-61/obj76 2023-10-17T19:09:55.064 DEBUG:boto:Data: 2023-10-17T19:09:55.064 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.065 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.065 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.066 DEBUG:boto:Params: {} 2023-10-17T19:09:55.066 DEBUG:boto:Token: None 2023-10-17T19:09:55.067 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76 2023-10-17T19:09:55.067 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:fJrAL7aIm3X4mLk/cIi/FGyDLDQ= 2023-10-17T19:09:55.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:fJrAL7aIm3X4mLk/cIi/FGyDLDQ=', 'Content-Length': '0'} 2023-10-17T19:09:55.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002695b421c276b33-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-17T19:09:55.071 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:55.071 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.072 DEBUG:boto:Method: GET 2023-10-17T19:09:55.072 DEBUG:boto:Path: /hbmuog-61/obj76 2023-10-17T19:09:55.073 DEBUG:boto:Data: 2023-10-17T19:09:55.073 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.074 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.074 DEBUG:boto:Params: {} 2023-10-17T19:09:55.075 DEBUG:boto:Token: None 2023-10-17T19:09:55.075 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76 2023-10-17T19:09:55.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZvFp/X8guvC2myQpNDV6f3BkDw= 2023-10-17T19:09:55.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZvFp/X8guvC2myQpNDV6f3BkDw=', 'Content-Length': '0'} 2023-10-17T19:09:55.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6c1b2347245ad3-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.079 DEBUG:boto:path=/hbmuog-61/obj76 2023-10-17T19:09:55.079 DEBUG:boto:auth_path=/hbmuog-61/obj76 2023-10-17T19:09:55.080 DEBUG:boto:Method: GET 2023-10-17T19:09:55.080 DEBUG:boto:Path: /hbmuog-61/obj76 2023-10-17T19:09:55.081 DEBUG:boto:Data: 2023-10-17T19:09:55.081 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.082 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.082 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.083 DEBUG:boto:Params: {} 2023-10-17T19:09:55.083 DEBUG:boto:Token: None 2023-10-17T19:09:55.084 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj76 2023-10-17T19:09:55.084 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4ZvFp/X8guvC2myQpNDV6f3BkDw= 2023-10-17T19:09:55.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZvFp/X8guvC2myQpNDV6f3BkDw=', 'Content-Length': '0'} 2023-10-17T19:09:55.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002257e750e8c7b54a-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.134 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.135 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.135 DEBUG:boto:path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.136 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.136 DEBUG:boto:Method: GET 2023-10-17T19:09:55.137 DEBUG:boto:Path: /hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.137 DEBUG:boto:Data: 2023-10-17T19:09:55.138 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.138 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.139 DEBUG:boto:Params: {} 2023-10-17T19:09:55.139 DEBUG:boto:Token: None 2023-10-17T19:09:55.140 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8f72b9e72ca960-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.144 DEBUG:boto:b'hbmuog-61obj761000falseobj762023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.145 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.145 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.146 DEBUG:boto:path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.146 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.147 DEBUG:boto:Method: GET 2023-10-17T19:09:55.147 DEBUG:boto:Path: /hbmuog-61/?prefix=obj76 2023-10-17T19:09:55.147 DEBUG:boto:Data: 2023-10-17T19:09:55.148 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.148 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.149 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.149 DEBUG:boto:Params: {} 2023-10-17T19:09:55.150 DEBUG:boto:Token: None 2023-10-17T19:09:55.150 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.151 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb406cdb18579dce-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.156 DEBUG:boto:b'hbmuog-61obj761000falseobj762023-10-17T19:06:20.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T19:09:55.157 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.158 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.158 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.158 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.159 DEBUG:boto:Method: GET 2023-10-17T19:09:55.159 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.160 DEBUG:boto:Data: 2023-10-17T19:09:55.160 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.161 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.161 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.162 DEBUG:boto:Params: {} 2023-10-17T19:09:55.162 DEBUG:boto:Token: None 2023-10-17T19:09:55.163 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.163 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y= 2023-10-17T19:09:55.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y=', 'Content-Length': '0'} 2023-10-17T19:09:55.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b8608e6ab52da5-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.166 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.167 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.167 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.168 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.168 DEBUG:boto:Method: GET 2023-10-17T19:09:55.169 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.169 DEBUG:boto:Data: 2023-10-17T19:09:55.170 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.170 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.170 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.171 DEBUG:boto:Params: {} 2023-10-17T19:09:55.171 DEBUG:boto:Token: None 2023-10-17T19:09:55.172 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.172 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y= 2023-10-17T19:09:55.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y=', 'Content-Length': '0'} 2023-10-17T19:09:55.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3d775f29dde0dd-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.176 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.176 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.176 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.177 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.177 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.178 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.178 DEBUG:boto:Data: 2023-10-17T19:09:55.178 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.179 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.179 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.180 DEBUG:boto:Params: {} 2023-10-17T19:09:55.180 DEBUG:boto:Token: None 2023-10-17T19:09:55.181 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.181 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RTkdb8QW8j0199S/HLLX0+0Cqmg= 2023-10-17T19:09:55.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RTkdb8QW8j0199S/HLLX0+0Cqmg=', 'Content-Length': '0'} 2023-10-17T19:09:55.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef1056587729920-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.185 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.185 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.185 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.186 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.186 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.187 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.187 DEBUG:boto:Data: 2023-10-17T19:09:55.188 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.188 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.188 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.189 DEBUG:boto:Params: {} 2023-10-17T19:09:55.189 DEBUG:boto:Token: None 2023-10-17T19:09:55.190 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.190 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RTkdb8QW8j0199S/HLLX0+0Cqmg= 2023-10-17T19:09:55.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RTkdb8QW8j0199S/HLLX0+0Cqmg=', 'Content-Length': '0'} 2023-10-17T19:09:55.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8bb7b80b689cd7-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T19:09:55.194 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.194 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.195 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.195 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.196 DEBUG:boto:Method: GET 2023-10-17T19:09:55.196 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.197 DEBUG:boto:Data: 2023-10-17T19:09:55.197 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.197 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.198 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.198 DEBUG:boto:Params: {} 2023-10-17T19:09:55.199 DEBUG:boto:Token: None 2023-10-17T19:09:55.199 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.200 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y= 2023-10-17T19:09:55.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y=', 'Content-Length': '0'} 2023-10-17T19:09:55.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002062d8fc719c515b-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.203 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.204 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.204 DEBUG:boto:path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.204 DEBUG:boto:auth_path=/hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.205 DEBUG:boto:Method: GET 2023-10-17T19:09:55.205 DEBUG:boto:Path: /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.206 DEBUG:boto:Data: 2023-10-17T19:09:55.206 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.207 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.207 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.207 DEBUG:boto:Params: {} 2023-10-17T19:09:55.208 DEBUG:boto:Token: None 2023-10-17T19:09:55.209 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85?versionId=null 2023-10-17T19:09:55.209 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y= 2023-10-17T19:09:55.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j7LOD3eCcpNQi8hOtiDfVqQh83Y=', 'Content-Length': '0'} 2023-10-17T19:09:55.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a19b5c6737341dba-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.254 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.255 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.256 DEBUG:boto:path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.256 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.256 DEBUG:boto:Method: GET 2023-10-17T19:09:55.257 DEBUG:boto:Path: /hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.257 DEBUG:boto:Data: 2023-10-17T19:09:55.258 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.258 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.259 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.259 DEBUG:boto:Params: {} 2023-10-17T19:09:55.260 DEBUG:boto:Token: None 2023-10-17T19:09:55.260 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000386d6ca24e394ba6-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.265 DEBUG:boto:b'hbmuog-61obj851000falseobj852023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.266 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.266 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.270 DEBUG:boto:path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.270 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.271 DEBUG:boto:Method: GET 2023-10-17T19:09:55.271 DEBUG:boto:Path: /hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.272 DEBUG:boto:Data: 2023-10-17T19:09:55.272 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.273 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.274 DEBUG:boto:Params: {} 2023-10-17T19:09:55.274 DEBUG:boto:Token: None 2023-10-17T19:09:55.275 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000146dba720e20918e-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.281 DEBUG:boto:b'hbmuog-61obj851000falseobj852023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.282 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.283 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.283 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.283 DEBUG:boto:Path: /hbmuog-61/obj85 2023-10-17T19:09:55.284 DEBUG:boto:Data: 2023-10-17T19:09:55.284 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.285 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.285 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.286 DEBUG:boto:Params: {} 2023-10-17T19:09:55.286 DEBUG:boto:Token: None 2023-10-17T19:09:55.287 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85 2023-10-17T19:09:55.287 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CutYKbaQwxYy0X+6VsTDDGCIRLE= 2023-10-17T19:09:55.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CutYKbaQwxYy0X+6VsTDDGCIRLE=', 'Content-Length': '0'} 2023-10-17T19:09:55.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e8de469310bb3d-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.291 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.291 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.291 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.292 DEBUG:boto:Path: /hbmuog-61/obj85 2023-10-17T19:09:55.292 DEBUG:boto:Data: 2023-10-17T19:09:55.293 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.293 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.293 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.294 DEBUG:boto:Params: {} 2023-10-17T19:09:55.294 DEBUG:boto:Token: None 2023-10-17T19:09:55.295 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85 2023-10-17T19:09:55.295 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CutYKbaQwxYy0X+6VsTDDGCIRLE= 2023-10-17T19:09:55.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CutYKbaQwxYy0X+6VsTDDGCIRLE=', 'Content-Length': '0'} 2023-10-17T19:09:55.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007adc3b2ab5f1e1b2-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-17T19:09:55.299 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.299 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.300 DEBUG:boto:Method: GET 2023-10-17T19:09:55.300 DEBUG:boto:Path: /hbmuog-61/obj85 2023-10-17T19:09:55.301 DEBUG:boto:Data: 2023-10-17T19:09:55.301 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.302 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.302 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.303 DEBUG:boto:Params: {} 2023-10-17T19:09:55.303 DEBUG:boto:Token: None 2023-10-17T19:09:55.304 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85 2023-10-17T19:09:55.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yqgLheN7EH0I84W0Arz9eKbSJCI= 2023-10-17T19:09:55.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yqgLheN7EH0I84W0Arz9eKbSJCI=', 'Content-Length': '0'} 2023-10-17T19:09:55.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc42640184610ef0-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.350 DEBUG:boto:path=/hbmuog-61/obj85 2023-10-17T19:09:55.351 DEBUG:boto:auth_path=/hbmuog-61/obj85 2023-10-17T19:09:55.351 DEBUG:boto:Method: GET 2023-10-17T19:09:55.352 DEBUG:boto:Path: /hbmuog-61/obj85 2023-10-17T19:09:55.352 DEBUG:boto:Data: 2023-10-17T19:09:55.353 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.353 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.354 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.354 DEBUG:boto:Params: {} 2023-10-17T19:09:55.355 DEBUG:boto:Token: None 2023-10-17T19:09:55.355 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj85 2023-10-17T19:09:55.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:yqgLheN7EH0I84W0Arz9eKbSJCI= 2023-10-17T19:09:55.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yqgLheN7EH0I84W0Arz9eKbSJCI=', 'Content-Length': '0'} 2023-10-17T19:09:55.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cdd915af9768bc-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.359 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.359 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.359 DEBUG:boto:path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.360 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.360 DEBUG:boto:Method: GET 2023-10-17T19:09:55.361 DEBUG:boto:Path: /hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.361 DEBUG:boto:Data: 2023-10-17T19:09:55.362 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.363 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.363 DEBUG:boto:Params: {} 2023-10-17T19:09:55.363 DEBUG:boto:Token: None 2023-10-17T19:09:55.364 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.364 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39ece918f9fe62c-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.368 DEBUG:boto:b'hbmuog-61obj851000falseobj852023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.369 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.369 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.370 DEBUG:boto:path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.370 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.371 DEBUG:boto:Method: GET 2023-10-17T19:09:55.371 DEBUG:boto:Path: /hbmuog-61/?prefix=obj85 2023-10-17T19:09:55.372 DEBUG:boto:Data: 2023-10-17T19:09:55.372 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.373 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.373 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.373 DEBUG:boto:Params: {} 2023-10-17T19:09:55.374 DEBUG:boto:Token: None 2023-10-17T19:09:55.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb1ece275a932ca-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.381 DEBUG:boto:b'hbmuog-61obj851000falseobj852023-10-17T19:06:20.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-17T19:09:55.382 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.382 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.383 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.383 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.384 DEBUG:boto:Method: GET 2023-10-17T19:09:55.384 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.384 DEBUG:boto:Data: 2023-10-17T19:09:55.385 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.385 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.386 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.386 DEBUG:boto:Params: {} 2023-10-17T19:09:55.387 DEBUG:boto:Token: None 2023-10-17T19:09:55.387 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.388 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s= 2023-10-17T19:09:55.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s=', 'Content-Length': '0'} 2023-10-17T19:09:55.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bbb6f76ce38ee6-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.391 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.391 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.392 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.392 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.392 DEBUG:boto:Method: GET 2023-10-17T19:09:55.393 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.393 DEBUG:boto:Data: 2023-10-17T19:09:55.394 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.395 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.395 DEBUG:boto:Params: {} 2023-10-17T19:09:55.396 DEBUG:boto:Token: None 2023-10-17T19:09:55.396 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.397 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s= 2023-10-17T19:09:55.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s=', 'Content-Length': '0'} 2023-10-17T19:09:55.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cafb2551b0b4dec0-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.442 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.443 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.443 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.444 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.444 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.444 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.445 DEBUG:boto:Data: 2023-10-17T19:09:55.445 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.446 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.446 DEBUG:boto:Params: {} 2023-10-17T19:09:55.447 DEBUG:boto:Token: None 2023-10-17T19:09:55.447 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UUXcY6ZYdU79zsxQd+5yU+hWi/k= 2023-10-17T19:09:55.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UUXcY6ZYdU79zsxQd+5yU+hWi/k=', 'Content-Length': '0'} 2023-10-17T19:09:55.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff443ec3b611e12-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.451 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.451 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.452 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.452 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.453 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.453 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.453 DEBUG:boto:Data: 2023-10-17T19:09:55.454 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.454 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.455 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.455 DEBUG:boto:Params: {} 2023-10-17T19:09:55.455 DEBUG:boto:Token: None 2023-10-17T19:09:55.456 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UUXcY6ZYdU79zsxQd+5yU+hWi/k= 2023-10-17T19:09:55.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UUXcY6ZYdU79zsxQd+5yU+hWi/k=', 'Content-Length': '0'} 2023-10-17T19:09:55.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc875ea08490ee96-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-17T19:09:55.460 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.460 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.461 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.461 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.462 DEBUG:boto:Method: GET 2023-10-17T19:09:55.462 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.462 DEBUG:boto:Data: 2023-10-17T19:09:55.463 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.463 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.464 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.464 DEBUG:boto:Params: {} 2023-10-17T19:09:55.465 DEBUG:boto:Token: None 2023-10-17T19:09:55.465 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.465 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s= 2023-10-17T19:09:55.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s=', 'Content-Length': '0'} 2023-10-17T19:09:55.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3fd65ec126f109-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.514 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.515 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.515 DEBUG:boto:path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.516 DEBUG:boto:auth_path=/hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.516 DEBUG:boto:Method: GET 2023-10-17T19:09:55.517 DEBUG:boto:Path: /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.517 DEBUG:boto:Data: 2023-10-17T19:09:55.517 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.518 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.519 DEBUG:boto:Params: {} 2023-10-17T19:09:55.519 DEBUG:boto:Token: None 2023-10-17T19:09:55.520 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87?versionId=null 2023-10-17T19:09:55.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s= 2023-10-17T19:09:55.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mo7IWBgZk4GctGp4uW/5EeU+d9s=', 'Content-Length': '0'} 2023-10-17T19:09:55.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7335c6a3d52dfd-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.524 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.524 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.525 DEBUG:boto:path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.525 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.526 DEBUG:boto:Method: GET 2023-10-17T19:09:55.526 DEBUG:boto:Path: /hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.526 DEBUG:boto:Data: 2023-10-17T19:09:55.527 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.527 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.528 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.528 DEBUG:boto:Params: {} 2023-10-17T19:09:55.529 DEBUG:boto:Token: None 2023-10-17T19:09:55.529 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.530 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193783353230a97b-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.533 DEBUG:boto:b'hbmuog-61obj871000falseobj872023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.534 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.534 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.535 DEBUG:boto:path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.535 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.535 DEBUG:boto:Method: GET 2023-10-17T19:09:55.536 DEBUG:boto:Path: /hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.536 DEBUG:boto:Data: 2023-10-17T19:09:55.537 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.537 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.537 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.538 DEBUG:boto:Params: {} 2023-10-17T19:09:55.538 DEBUG:boto:Token: None 2023-10-17T19:09:55.539 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.540 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1ab7bc69bf5603-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.545 DEBUG:boto:b'hbmuog-61obj871000falseobj872023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.545 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.546 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.546 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.547 DEBUG:boto:Path: /hbmuog-61/obj87 2023-10-17T19:09:55.547 DEBUG:boto:Data: 2023-10-17T19:09:55.547 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.548 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.548 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.549 DEBUG:boto:Params: {} 2023-10-17T19:09:55.549 DEBUG:boto:Token: None 2023-10-17T19:09:55.550 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87 2023-10-17T19:09:55.550 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8rDRYKbYQtA/LGWg6FRr8uNG1sw= 2023-10-17T19:09:55.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8rDRYKbYQtA/LGWg6FRr8uNG1sw=', 'Content-Length': '0'} 2023-10-17T19:09:55.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fdefc9bd1b93b6f-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.553 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.553 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.554 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.554 DEBUG:boto:Path: /hbmuog-61/obj87 2023-10-17T19:09:55.555 DEBUG:boto:Data: 2023-10-17T19:09:55.555 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.555 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.556 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.556 DEBUG:boto:Params: {} 2023-10-17T19:09:55.557 DEBUG:boto:Token: None 2023-10-17T19:09:55.557 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87 2023-10-17T19:09:55.558 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8rDRYKbYQtA/LGWg6FRr8uNG1sw= 2023-10-17T19:09:55.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8rDRYKbYQtA/LGWg6FRr8uNG1sw=', 'Content-Length': '0'} 2023-10-17T19:09:55.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925a81c29e926b62-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-17T19:09:55.561 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.562 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.562 DEBUG:boto:Method: GET 2023-10-17T19:09:55.563 DEBUG:boto:Path: /hbmuog-61/obj87 2023-10-17T19:09:55.563 DEBUG:boto:Data: 2023-10-17T19:09:55.563 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.564 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.564 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.565 DEBUG:boto:Params: {} 2023-10-17T19:09:55.565 DEBUG:boto:Token: None 2023-10-17T19:09:55.566 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87 2023-10-17T19:09:55.566 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiTJWd+C5Tm50IMogSijtRFeQmw= 2023-10-17T19:09:55.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiTJWd+C5Tm50IMogSijtRFeQmw=', 'Content-Length': '0'} 2023-10-17T19:09:55.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d344b754bbb0e6c4-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.569 DEBUG:boto:path=/hbmuog-61/obj87 2023-10-17T19:09:55.570 DEBUG:boto:auth_path=/hbmuog-61/obj87 2023-10-17T19:09:55.570 DEBUG:boto:Method: GET 2023-10-17T19:09:55.571 DEBUG:boto:Path: /hbmuog-61/obj87 2023-10-17T19:09:55.571 DEBUG:boto:Data: 2023-10-17T19:09:55.571 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.572 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.572 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.573 DEBUG:boto:Params: {} 2023-10-17T19:09:55.573 DEBUG:boto:Token: None 2023-10-17T19:09:55.574 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj87 2023-10-17T19:09:55.574 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hiTJWd+C5Tm50IMogSijtRFeQmw= 2023-10-17T19:09:55.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiTJWd+C5Tm50IMogSijtRFeQmw=', 'Content-Length': '0'} 2023-10-17T19:09:55.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a93bab4474c084-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.622 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.623 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.624 DEBUG:boto:path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.624 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.625 DEBUG:boto:Method: GET 2023-10-17T19:09:55.625 DEBUG:boto:Path: /hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.625 DEBUG:boto:Data: 2023-10-17T19:09:55.626 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.627 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.627 DEBUG:boto:Params: {} 2023-10-17T19:09:55.628 DEBUG:boto:Token: None 2023-10-17T19:09:55.628 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000455da98dfad7facb-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.633 DEBUG:boto:b'hbmuog-61obj871000falseobj872023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.633 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.634 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.634 DEBUG:boto:path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.635 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.635 DEBUG:boto:Method: GET 2023-10-17T19:09:55.635 DEBUG:boto:Path: /hbmuog-61/?prefix=obj87 2023-10-17T19:09:55.636 DEBUG:boto:Data: 2023-10-17T19:09:55.636 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.637 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.637 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.638 DEBUG:boto:Params: {} 2023-10-17T19:09:55.638 DEBUG:boto:Token: None 2023-10-17T19:09:55.638 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c9b019836d4c29-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.644 DEBUG:boto:b'hbmuog-61obj871000falseobj872023-10-17T19:06:21.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-17T19:09:55.645 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.646 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.646 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.647 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.647 DEBUG:boto:Method: GET 2023-10-17T19:09:55.648 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.648 DEBUG:boto:Data: 2023-10-17T19:09:55.649 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.649 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.650 DEBUG:boto:Params: {} 2023-10-17T19:09:55.650 DEBUG:boto:Token: None 2023-10-17T19:09:55.651 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.651 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA= 2023-10-17T19:09:55.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA=', 'Content-Length': '0'} 2023-10-17T19:09:55.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ed89c7442e0408-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.655 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.655 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.655 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.656 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.656 DEBUG:boto:Method: GET 2023-10-17T19:09:55.657 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.657 DEBUG:boto:Data: 2023-10-17T19:09:55.658 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.658 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.659 DEBUG:boto:Params: {} 2023-10-17T19:09:55.659 DEBUG:boto:Token: None 2023-10-17T19:09:55.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.660 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA= 2023-10-17T19:09:55.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA=', 'Content-Length': '0'} 2023-10-17T19:09:55.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda071924a8291ba-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.664 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.665 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.665 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.665 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.666 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.667 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.667 DEBUG:boto:Data: 2023-10-17T19:09:55.667 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.668 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.669 DEBUG:boto:Params: {} 2023-10-17T19:09:55.669 DEBUG:boto:Token: None 2023-10-17T19:09:55.670 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.670 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XoY8AcwLePNRwQenqrMz4BtpdQ8= 2023-10-17T19:09:55.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XoY8AcwLePNRwQenqrMz4BtpdQ8=', 'Content-Length': '0'} 2023-10-17T19:09:55.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006781a0f9c0d9e708-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.674 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.674 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.675 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.675 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.676 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.676 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.677 DEBUG:boto:Data: 2023-10-17T19:09:55.677 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.678 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.678 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.678 DEBUG:boto:Params: {} 2023-10-17T19:09:55.681 DEBUG:boto:Token: None 2023-10-17T19:09:55.682 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.682 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XoY8AcwLePNRwQenqrMz4BtpdQ8= 2023-10-17T19:09:55.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XoY8AcwLePNRwQenqrMz4BtpdQ8=', 'Content-Length': '0'} 2023-10-17T19:09:55.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac52f32e42a911a-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-17T19:09:55.687 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.688 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.689 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.689 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.690 DEBUG:boto:Method: GET 2023-10-17T19:09:55.691 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.692 DEBUG:boto:Data: 2023-10-17T19:09:55.692 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.693 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.694 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.694 DEBUG:boto:Params: {} 2023-10-17T19:09:55.695 DEBUG:boto:Token: None 2023-10-17T19:09:55.696 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA= 2023-10-17T19:09:55.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA=', 'Content-Length': '0'} 2023-10-17T19:09:55.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdee2b44470065e7-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.700 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.701 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.701 DEBUG:boto:path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.702 DEBUG:boto:auth_path=/hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.702 DEBUG:boto:Method: GET 2023-10-17T19:09:55.703 DEBUG:boto:Path: /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.703 DEBUG:boto:Data: 2023-10-17T19:09:55.703 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.704 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.705 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.705 DEBUG:boto:Params: {} 2023-10-17T19:09:55.706 DEBUG:boto:Token: None 2023-10-17T19:09:55.707 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92?versionId=null 2023-10-17T19:09:55.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA= 2023-10-17T19:09:55.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UNvettIRw6zWYv762YGbppTEvHA=', 'Content-Length': '0'} 2023-10-17T19:09:55.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c757afa57e7b1d-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.755 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.756 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.756 DEBUG:boto:path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.757 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.757 DEBUG:boto:Method: GET 2023-10-17T19:09:55.758 DEBUG:boto:Path: /hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.758 DEBUG:boto:Data: 2023-10-17T19:09:55.759 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.759 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.760 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.760 DEBUG:boto:Params: {} 2023-10-17T19:09:55.761 DEBUG:boto:Token: None 2023-10-17T19:09:55.761 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.762 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b0bece16baa3b13-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.765 DEBUG:boto:b'hbmuog-61obj921000falseobj922023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.766 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.766 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.767 DEBUG:boto:path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.767 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.768 DEBUG:boto:Method: GET 2023-10-17T19:09:55.768 DEBUG:boto:Path: /hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.769 DEBUG:boto:Data: 2023-10-17T19:09:55.769 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.769 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.770 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.770 DEBUG:boto:Params: {} 2023-10-17T19:09:55.771 DEBUG:boto:Token: None 2023-10-17T19:09:55.771 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.772 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092bf12de31e5703c-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.777 DEBUG:boto:b'hbmuog-61obj921000falseobj922023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.778 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.778 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.779 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.779 DEBUG:boto:Path: /hbmuog-61/obj92 2023-10-17T19:09:55.779 DEBUG:boto:Data: 2023-10-17T19:09:55.780 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.780 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.780 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.781 DEBUG:boto:Params: {} 2023-10-17T19:09:55.781 DEBUG:boto:Token: None 2023-10-17T19:09:55.782 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92 2023-10-17T19:09:55.782 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B/34r69b0ghZcbbWDK20A9Vyd+8= 2023-10-17T19:09:55.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B/34r69b0ghZcbbWDK20A9Vyd+8=', 'Content-Length': '0'} 2023-10-17T19:09:55.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001712d9ad50d70df8-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.785 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.786 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.786 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.786 DEBUG:boto:Path: /hbmuog-61/obj92 2023-10-17T19:09:55.787 DEBUG:boto:Data: 2023-10-17T19:09:55.787 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.788 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.789 DEBUG:boto:Params: {} 2023-10-17T19:09:55.789 DEBUG:boto:Token: None 2023-10-17T19:09:55.790 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92 2023-10-17T19:09:55.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:B/34r69b0ghZcbbWDK20A9Vyd+8= 2023-10-17T19:09:55.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B/34r69b0ghZcbbWDK20A9Vyd+8=', 'Content-Length': '0'} 2023-10-17T19:09:55.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cebbdfe6e5e55251-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-17T19:09:55.794 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.794 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.794 DEBUG:boto:Method: GET 2023-10-17T19:09:55.795 DEBUG:boto:Path: /hbmuog-61/obj92 2023-10-17T19:09:55.795 DEBUG:boto:Data: 2023-10-17T19:09:55.796 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.797 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.797 DEBUG:boto:Params: {} 2023-10-17T19:09:55.797 DEBUG:boto:Token: None 2023-10-17T19:09:55.798 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92 2023-10-17T19:09:55.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AmTurXyv2wpb+ZMaj8NWnHPFkRo= 2023-10-17T19:09:55.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AmTurXyv2wpb+ZMaj8NWnHPFkRo=', 'Content-Length': '0'} 2023-10-17T19:09:55.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071fde0a6dd5a2271-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.802 DEBUG:boto:path=/hbmuog-61/obj92 2023-10-17T19:09:55.802 DEBUG:boto:auth_path=/hbmuog-61/obj92 2023-10-17T19:09:55.803 DEBUG:boto:Method: GET 2023-10-17T19:09:55.803 DEBUG:boto:Path: /hbmuog-61/obj92 2023-10-17T19:09:55.804 DEBUG:boto:Data: 2023-10-17T19:09:55.804 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.804 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.805 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.805 DEBUG:boto:Params: {} 2023-10-17T19:09:55.806 DEBUG:boto:Token: None 2023-10-17T19:09:55.806 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj92 2023-10-17T19:09:55.807 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AmTurXyv2wpb+ZMaj8NWnHPFkRo= 2023-10-17T19:09:55.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AmTurXyv2wpb+ZMaj8NWnHPFkRo=', 'Content-Length': '0'} 2023-10-17T19:09:55.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2acf844c929168-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.810 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.810 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.811 DEBUG:boto:path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.811 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.812 DEBUG:boto:Method: GET 2023-10-17T19:09:55.812 DEBUG:boto:Path: /hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.812 DEBUG:boto:Data: 2023-10-17T19:09:55.813 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.813 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.814 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.814 DEBUG:boto:Params: {} 2023-10-17T19:09:55.815 DEBUG:boto:Token: None 2023-10-17T19:09:55.815 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da216f6829c2d64-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.819 DEBUG:boto:b'hbmuog-61obj921000falseobj922023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.820 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.820 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.820 DEBUG:boto:path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.821 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.821 DEBUG:boto:Method: GET 2023-10-17T19:09:55.822 DEBUG:boto:Path: /hbmuog-61/?prefix=obj92 2023-10-17T19:09:55.822 DEBUG:boto:Data: 2023-10-17T19:09:55.823 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.823 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.823 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.824 DEBUG:boto:Params: {} 2023-10-17T19:09:55.825 DEBUG:boto:Token: None 2023-10-17T19:09:55.825 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.825 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94b7822ee4e127e-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.831 DEBUG:boto:b'hbmuog-61obj921000falseobj922023-10-17T19:06:20.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-17T19:09:55.832 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.832 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.833 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.833 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.833 DEBUG:boto:Method: GET 2023-10-17T19:09:55.834 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.834 DEBUG:boto:Data: 2023-10-17T19:09:55.835 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.835 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.836 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.836 DEBUG:boto:Params: {} 2023-10-17T19:09:55.836 DEBUG:boto:Token: None 2023-10-17T19:09:55.837 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.837 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg= 2023-10-17T19:09:55.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg=', 'Content-Length': '0'} 2023-10-17T19:09:55.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007962558ae8349fee-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.841 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.841 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.842 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.842 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.842 DEBUG:boto:Method: GET 2023-10-17T19:09:55.843 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.843 DEBUG:boto:Data: 2023-10-17T19:09:55.844 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.844 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.844 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.845 DEBUG:boto:Params: {} 2023-10-17T19:09:55.845 DEBUG:boto:Token: None 2023-10-17T19:09:55.846 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.847 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg= 2023-10-17T19:09:55.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg=', 'Content-Length': '0'} 2023-10-17T19:09:55.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b0ab7096cdd198-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.890 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.891 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.891 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.892 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.892 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.893 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.893 DEBUG:boto:Data: 2023-10-17T19:09:55.894 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.894 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.895 DEBUG:boto:Params: {} 2023-10-17T19:09:55.895 DEBUG:boto:Token: None 2023-10-17T19:09:55.896 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nR9kVKobPBXn/6W0TKvUL63iHc0= 2023-10-17T19:09:55.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nR9kVKobPBXn/6W0TKvUL63iHc0=', 'Content-Length': '0'} 2023-10-17T19:09:55.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6ae33e4931cc16-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.900 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.900 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.901 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.901 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.901 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.902 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.902 DEBUG:boto:Data: 2023-10-17T19:09:55.903 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.903 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.903 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.904 DEBUG:boto:Params: {} 2023-10-17T19:09:55.904 DEBUG:boto:Token: None 2023-10-17T19:09:55.905 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.905 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nR9kVKobPBXn/6W0TKvUL63iHc0= 2023-10-17T19:09:55.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nR9kVKobPBXn/6W0TKvUL63iHc0=', 'Content-Length': '0'} 2023-10-17T19:09:55.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2479349a912fa65-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-17T19:09:55.909 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.909 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.910 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.910 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.911 DEBUG:boto:Method: GET 2023-10-17T19:09:55.911 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.911 DEBUG:boto:Data: 2023-10-17T19:09:55.912 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.912 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.913 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.913 DEBUG:boto:Params: {} 2023-10-17T19:09:55.914 DEBUG:boto:Token: None 2023-10-17T19:09:55.914 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.915 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg= 2023-10-17T19:09:55.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg=', 'Content-Length': '0'} 2023-10-17T19:09:55.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068198147a488a830-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.958 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.959 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.959 DEBUG:boto:path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.960 DEBUG:boto:auth_path=/hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.960 DEBUG:boto:Method: GET 2023-10-17T19:09:55.961 DEBUG:boto:Path: /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.961 DEBUG:boto:Data: 2023-10-17T19:09:55.962 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.962 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.962 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.963 DEBUG:boto:Params: {} 2023-10-17T19:09:55.963 DEBUG:boto:Token: None 2023-10-17T19:09:55.964 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93?versionId=null 2023-10-17T19:09:55.964 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg= 2023-10-17T19:09:55.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:spH5dEpbMKDQ9f5IEreu2gfyFYg=', 'Content-Length': '0'} 2023-10-17T19:09:55.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e228aada2f4e9ec-00652edc03-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.968 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.968 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.969 DEBUG:boto:path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.969 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.970 DEBUG:boto:Method: GET 2023-10-17T19:09:55.970 DEBUG:boto:Path: /hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.971 DEBUG:boto:Data: 2023-10-17T19:09:55.971 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.972 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.972 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.973 DEBUG:boto:Params: {} 2023-10-17T19:09:55.973 DEBUG:boto:Token: None 2023-10-17T19:09:55.974 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.974 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998ad3ecec95f28b-00652edc03-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.977 DEBUG:boto:b'hbmuog-61obj931000falseobj932023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.978 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:55.979 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:55.979 DEBUG:boto:path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.980 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.980 DEBUG:boto:Method: GET 2023-10-17T19:09:55.980 DEBUG:boto:Path: /hbmuog-61/?prefix=obj93 2023-10-17T19:09:55.981 DEBUG:boto:Data: 2023-10-17T19:09:55.981 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.982 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:55.982 DEBUG:boto:Port: 8000 2023-10-17T19:09:55.983 DEBUG:boto:Params: {} 2023-10-17T19:09:55.983 DEBUG:boto:Token: None 2023-10-17T19:09:55.984 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/ 2023-10-17T19:09:55.984 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4= 2023-10-17T19:09:55.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:t/cxkCVN5G2XB4dS7w12PZUjJo4=', 'Content-Length': '0'} 2023-10-17T19:09:55.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc7fd5b3e7621a1-00652edc03-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.989 DEBUG:boto:b'hbmuog-61obj931000falseobj932023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:55.990 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.991 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.991 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.991 DEBUG:boto:Path: /hbmuog-61/obj93 2023-10-17T19:09:55.992 DEBUG:boto:Data: 2023-10-17T19:09:55.992 DEBUG:boto:Headers: {} 2023-10-17T19:09:55.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:55.993 DEBUG:boto:Port: 8001 2023-10-17T19:09:55.993 DEBUG:boto:Params: {} 2023-10-17T19:09:55.994 DEBUG:boto:Token: None 2023-10-17T19:09:55.994 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:55 GMT /hbmuog-61/obj93 2023-10-17T19:09:55.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BHXmGnvd1lZB8p9NP2PxfWzazug= 2023-10-17T19:09:55.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BHXmGnvd1lZB8p9NP2PxfWzazug=', 'Content-Length': '0'} 2023-10-17T19:09:55.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f5ada3c8e12c9b-00652edc03-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:55.998 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:55.998 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:55.999 DEBUG:boto:Method: HEAD 2023-10-17T19:09:55.999 DEBUG:boto:Path: /hbmuog-61/obj93 2023-10-17T19:09:56.000 DEBUG:boto:Data: 2023-10-17T19:09:56.000 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.001 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.001 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.001 DEBUG:boto:Params: {} 2023-10-17T19:09:56.002 DEBUG:boto:Token: None 2023-10-17T19:09:56.002 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj93 2023-10-17T19:09:56.003 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4/ghl0ag8/jFYUj1Fivq1FVD44o= 2023-10-17T19:09:56.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4/ghl0ag8/jFYUj1Fivq1FVD44o=', 'Content-Length': '0'} 2023-10-17T19:09:56.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772ff377fd748b2f-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-17T19:09:56.006 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:56.007 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:56.007 DEBUG:boto:Method: GET 2023-10-17T19:09:56.007 DEBUG:boto:Path: /hbmuog-61/obj93 2023-10-17T19:09:56.008 DEBUG:boto:Data: 2023-10-17T19:09:56.008 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.009 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.009 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.009 DEBUG:boto:Params: {} 2023-10-17T19:09:56.010 DEBUG:boto:Token: None 2023-10-17T19:09:56.010 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj93 2023-10-17T19:09:56.011 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qjyk3cmukA2UgsA9MmlYLnH5t6I= 2023-10-17T19:09:56.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qjyk3cmukA2UgsA9MmlYLnH5t6I=', 'Content-Length': '0'} 2023-10-17T19:09:56.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9a63fedbcc8106a-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.014 DEBUG:boto:path=/hbmuog-61/obj93 2023-10-17T19:09:56.015 DEBUG:boto:auth_path=/hbmuog-61/obj93 2023-10-17T19:09:56.015 DEBUG:boto:Method: GET 2023-10-17T19:09:56.015 DEBUG:boto:Path: /hbmuog-61/obj93 2023-10-17T19:09:56.016 DEBUG:boto:Data: 2023-10-17T19:09:56.016 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.017 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.017 DEBUG:boto:Params: {} 2023-10-17T19:09:56.018 DEBUG:boto:Token: None 2023-10-17T19:09:56.018 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj93 2023-10-17T19:09:56.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qjyk3cmukA2UgsA9MmlYLnH5t6I= 2023-10-17T19:09:56.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qjyk3cmukA2UgsA9MmlYLnH5t6I=', 'Content-Length': '0'} 2023-10-17T19:09:56.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a08e38312fe2938-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.062 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.063 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.063 DEBUG:boto:path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.064 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.064 DEBUG:boto:Method: GET 2023-10-17T19:09:56.065 DEBUG:boto:Path: /hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.065 DEBUG:boto:Data: 2023-10-17T19:09:56.065 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.066 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.067 DEBUG:boto:Params: {} 2023-10-17T19:09:56.067 DEBUG:boto:Token: None 2023-10-17T19:09:56.068 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.068 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008505c2486619a3a-00652edc04-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.072 DEBUG:boto:b'hbmuog-61obj931000falseobj932023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.072 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.073 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.073 DEBUG:boto:path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.074 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.074 DEBUG:boto:Method: GET 2023-10-17T19:09:56.075 DEBUG:boto:Path: /hbmuog-61/?prefix=obj93 2023-10-17T19:09:56.075 DEBUG:boto:Data: 2023-10-17T19:09:56.075 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.076 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.077 DEBUG:boto:Params: {} 2023-10-17T19:09:56.077 DEBUG:boto:Token: None 2023-10-17T19:09:56.078 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.078 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001299a6ea80d49403-00652edc04-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.083 DEBUG:boto:b'hbmuog-61obj931000falseobj932023-10-17T19:06:21.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-17T19:09:56.085 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.085 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.086 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.086 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.090 DEBUG:boto:Method: GET 2023-10-17T19:09:56.090 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.091 DEBUG:boto:Data: 2023-10-17T19:09:56.091 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.091 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.092 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.092 DEBUG:boto:Params: {} 2023-10-17T19:09:56.093 DEBUG:boto:Token: None 2023-10-17T19:09:56.093 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.094 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU= 2023-10-17T19:09:56.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU=', 'Content-Length': '0'} 2023-10-17T19:09:56.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c355ecb7eefbac64-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.097 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.097 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.098 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.098 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.099 DEBUG:boto:Method: GET 2023-10-17T19:09:56.099 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.099 DEBUG:boto:Data: 2023-10-17T19:09:56.100 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.101 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.101 DEBUG:boto:Params: {} 2023-10-17T19:09:56.102 DEBUG:boto:Token: None 2023-10-17T19:09:56.102 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.103 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU= 2023-10-17T19:09:56.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU=', 'Content-Length': '0'} 2023-10-17T19:09:56.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8c2f7de8d59e7d-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.106 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.106 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.107 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.107 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.107 DEBUG:boto:Method: HEAD 2023-10-17T19:09:56.108 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.108 DEBUG:boto:Data: 2023-10-17T19:09:56.109 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.109 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.109 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.110 DEBUG:boto:Params: {} 2023-10-17T19:09:56.110 DEBUG:boto:Token: None 2023-10-17T19:09:56.111 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3L5Q7y0LT8Ed9BGdV8IyhC+eaGQ= 2023-10-17T19:09:56.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3L5Q7y0LT8Ed9BGdV8IyhC+eaGQ=', 'Content-Length': '0'} 2023-10-17T19:09:56.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067efe33ccca157d7-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.114 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.115 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.115 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.115 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.116 DEBUG:boto:Method: HEAD 2023-10-17T19:09:56.116 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.117 DEBUG:boto:Data: 2023-10-17T19:09:56.117 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.117 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.118 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.118 DEBUG:boto:Params: {} 2023-10-17T19:09:56.119 DEBUG:boto:Token: None 2023-10-17T19:09:56.119 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.120 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3L5Q7y0LT8Ed9BGdV8IyhC+eaGQ= 2023-10-17T19:09:56.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3L5Q7y0LT8Ed9BGdV8IyhC+eaGQ=', 'Content-Length': '0'} 2023-10-17T19:09:56.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f22340ee0870a82-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-17T19:09:56.123 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.124 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.124 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.124 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.125 DEBUG:boto:Method: GET 2023-10-17T19:09:56.125 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.126 DEBUG:boto:Data: 2023-10-17T19:09:56.126 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.127 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.127 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.127 DEBUG:boto:Params: {} 2023-10-17T19:09:56.128 DEBUG:boto:Token: None 2023-10-17T19:09:56.128 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.129 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU= 2023-10-17T19:09:56.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU=', 'Content-Length': '0'} 2023-10-17T19:09:56.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25cbdeb0401f1aa-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.132 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.132 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.133 DEBUG:boto:path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.133 DEBUG:boto:auth_path=/hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.134 DEBUG:boto:Method: GET 2023-10-17T19:09:56.134 DEBUG:boto:Path: /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.135 DEBUG:boto:Data: 2023-10-17T19:09:56.135 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.136 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.136 DEBUG:boto:Params: {} 2023-10-17T19:09:56.137 DEBUG:boto:Token: None 2023-10-17T19:09:56.137 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95?versionId=null 2023-10-17T19:09:56.138 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU= 2023-10-17T19:09:56.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+LWeqd8JmL7eFP77LG6pMSAN3uU=', 'Content-Length': '0'} 2023-10-17T19:09:56.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45cec8c2feb0c4f-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.141 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.141 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.142 DEBUG:boto:path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.142 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.143 DEBUG:boto:Method: GET 2023-10-17T19:09:56.143 DEBUG:boto:Path: /hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.144 DEBUG:boto:Data: 2023-10-17T19:09:56.144 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.144 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.145 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.145 DEBUG:boto:Params: {} 2023-10-17T19:09:56.146 DEBUG:boto:Token: None 2023-10-17T19:09:56.146 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.147 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81fcf56804dfc8e-00652edc04-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.190 DEBUG:boto:b'hbmuog-61obj951000falseobj952023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.191 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.192 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.192 DEBUG:boto:path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.192 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.193 DEBUG:boto:Method: GET 2023-10-17T19:09:56.193 DEBUG:boto:Path: /hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.194 DEBUG:boto:Data: 2023-10-17T19:09:56.194 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.195 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.195 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.195 DEBUG:boto:Params: {} 2023-10-17T19:09:56.196 DEBUG:boto:Token: None 2023-10-17T19:09:56.196 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.197 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4fd8c22b03b29b-00652edc04-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.202 DEBUG:boto:b'hbmuog-61obj951000falseobj952023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.203 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.203 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.204 DEBUG:boto:Method: HEAD 2023-10-17T19:09:56.204 DEBUG:boto:Path: /hbmuog-61/obj95 2023-10-17T19:09:56.205 DEBUG:boto:Data: 2023-10-17T19:09:56.205 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.206 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.206 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.206 DEBUG:boto:Params: {} 2023-10-17T19:09:56.207 DEBUG:boto:Token: None 2023-10-17T19:09:56.207 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95 2023-10-17T19:09:56.208 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PsqlnFu/vDVIL+fnPLQV8lhLQ1s= 2023-10-17T19:09:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PsqlnFu/vDVIL+fnPLQV8lhLQ1s=', 'Content-Length': '0'} 2023-10-17T19:09:56.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be19fe9ec0d1c725-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.211 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.211 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.212 DEBUG:boto:Method: HEAD 2023-10-17T19:09:56.212 DEBUG:boto:Path: /hbmuog-61/obj95 2023-10-17T19:09:56.212 DEBUG:boto:Data: 2023-10-17T19:09:56.213 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.214 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.214 DEBUG:boto:Params: {} 2023-10-17T19:09:56.214 DEBUG:boto:Token: None 2023-10-17T19:09:56.215 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95 2023-10-17T19:09:56.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PsqlnFu/vDVIL+fnPLQV8lhLQ1s= 2023-10-17T19:09:56.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PsqlnFu/vDVIL+fnPLQV8lhLQ1s=', 'Content-Length': '0'} 2023-10-17T19:09:56.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a97460918ac239-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-17T19:09:56.219 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.219 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.219 DEBUG:boto:Method: GET 2023-10-17T19:09:56.220 DEBUG:boto:Path: /hbmuog-61/obj95 2023-10-17T19:09:56.220 DEBUG:boto:Data: 2023-10-17T19:09:56.221 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.221 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.222 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.222 DEBUG:boto:Params: {} 2023-10-17T19:09:56.223 DEBUG:boto:Token: None 2023-10-17T19:09:56.223 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95 2023-10-17T19:09:56.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:39lb/nWdlCJrdjjMcP3OT8CCKQY= 2023-10-17T19:09:56.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:39lb/nWdlCJrdjjMcP3OT8CCKQY=', 'Content-Length': '0'} 2023-10-17T19:09:56.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-61:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a95ef3a68c48084-00652edc04-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.227 DEBUG:boto:path=/hbmuog-61/obj95 2023-10-17T19:09:56.227 DEBUG:boto:auth_path=/hbmuog-61/obj95 2023-10-17T19:09:56.228 DEBUG:boto:Method: GET 2023-10-17T19:09:56.228 DEBUG:boto:Path: /hbmuog-61/obj95 2023-10-17T19:09:56.229 DEBUG:boto:Data: 2023-10-17T19:09:56.229 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.230 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.230 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.231 DEBUG:boto:Params: {} 2023-10-17T19:09:56.231 DEBUG:boto:Token: None 2023-10-17T19:09:56.232 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/obj95 2023-10-17T19:09:56.232 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:39lb/nWdlCJrdjjMcP3OT8CCKQY= 2023-10-17T19:09:56.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:39lb/nWdlCJrdjjMcP3OT8CCKQY=', 'Content-Length': '0'} 2023-10-17T19:09:56.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:06:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002005f2dc4cff6bbc-00652edc04-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.235 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.236 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.236 DEBUG:boto:path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.237 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.237 DEBUG:boto:Method: GET 2023-10-17T19:09:56.237 DEBUG:boto:Path: /hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.238 DEBUG:boto:Data: 2023-10-17T19:09:56.238 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.239 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:09:56.239 DEBUG:boto:Port: 8001 2023-10-17T19:09:56.240 DEBUG:boto:Params: {} 2023-10-17T19:09:56.240 DEBUG:boto:Token: None 2023-10-17T19:09:56.241 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.241 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ff67c1fbee524a-00652edc04-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.245 DEBUG:boto:b'hbmuog-61obj951000falseobj952023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.245 DEBUG:boto:path=/hbmuog-61/ 2023-10-17T19:09:56.246 DEBUG:boto:auth_path=/hbmuog-61/ 2023-10-17T19:09:56.246 DEBUG:boto:path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.247 DEBUG:boto:auth_path=/hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.247 DEBUG:boto:Method: GET 2023-10-17T19:09:56.247 DEBUG:boto:Path: /hbmuog-61/?prefix=obj95 2023-10-17T19:09:56.248 DEBUG:boto:Data: 2023-10-17T19:09:56.248 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.249 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.249 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.250 DEBUG:boto:Params: {} 2023-10-17T19:09:56.250 DEBUG:boto:Token: None 2023-10-17T19:09:56.251 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-61/ 2023-10-17T19:09:56.251 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po= 2023-10-17T19:09:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ssbKLQt7t4u+E2FfD1SOHNUX+Po=', 'Content-Length': '0'} 2023-10-17T19:09:56.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ecdbd4c8f69a52-00652edc04-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.299 DEBUG:boto:b'hbmuog-61obj951000falseobj952023-10-17T19:06:20.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:09:56.300 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-61 zones={a2, a1} 2023-10-17T19:09:56.300 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T19:09:56.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.302 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.417 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:09:56.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.417 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028336", 2023-10-17T19:09:56.418 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:31.280312Z" 2023-10-17T19:09:56.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.418 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027532", 2023-10-17T19:09:56.419 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:31.281941Z" 2023-10-17T19:09:56.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.419 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:09:56.419 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:09:56.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.420 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032573", 2023-10-17T19:09:56.420 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:21.866729Z" 2023-10-17T19:09:56.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.421 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:09:56.422 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000028336', 1: '00000000000000000000:00000000000000027532', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032573'} 2023-10-17T19:09:56.422 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:09:56.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.424 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.477 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.477 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.523 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:09:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:09:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:09:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:09:56.524 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:09:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13410015314449062430 2023-10-17T19:09:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:09:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.525 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:09:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.527 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028336", 2023-10-17T19:09:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.528 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.280312Z" 2023-10-17T19:09:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.531 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:09:56.531 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027532", 2023-10-17T19:09:56.532 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.533 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.533 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.281941Z" 2023-10-17T19:09:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:09:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.535 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:09:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:09:56.536 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:09:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:09:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.538 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032573", 2023-10-17T19:09:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 52, 2023-10-17T19:09:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:21.866729Z" 2023-10-17T19:09:56.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:09:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:09:56.540 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 61, 2023-10-17T19:09:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 61 2023-10-17T19:09:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.541 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:09:56Z" 2023-10-17T19:09:56.541 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:09:56.542 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13410015314449062430 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028336", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:06:31.280312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027532", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:06:31.281941Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:06:31.283420Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032573", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-10-17T19:06:21.866729Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-10-17T19:09:56Z" } 2023-10-17T19:09:56.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028336', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.280312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027532', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.281941Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.283420Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032573', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-10-17T19:06:21.866729Z'}}] 2023-10-17T19:09:56.544 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028336', 1: '00000000000000000000:00000000000000027532', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032573'} 2023-10-17T19:09:56.544 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028336', 1: '00000000000000000000:00000000000000027532', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032573'} 2023-10-17T19:09:56.545 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:09:56.545 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T19:09:56.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.546 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.648 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:09:56.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.649 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000009955", 2023-10-17T19:09:56.649 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:08:12.093848Z" 2023-10-17T19:09:56.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.650 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008564", 2023-10-17T19:09:56.650 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:08:12.095568Z" 2023-10-17T19:09:56.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.651 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004736", 2023-10-17T19:09:56.651 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:08:12.096952Z" 2023-10-17T19:09:56.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.651 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012768", 2023-10-17T19:09:56.652 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:08:12.098244Z" 2023-10-17T19:09:56.652 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.652 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:09:56.653 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000009955', 1: '00000000000000000000:00000000000000008564', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012768'} 2023-10-17T19:09:56.654 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:09:56.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.655 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.708 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.708 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.753 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:09:56.754 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:09:56.754 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:09:56.754 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:09:56.754 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:09:56.754 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T19:09:56.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.755 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:09:56.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.755 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:09:56.755 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.756 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.756 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000009955", 2023-10-17T19:09:56.756 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.756 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.757 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:08:12.093848Z" 2023-10-17T19:09:56.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.758 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:09:56.758 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.758 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008564", 2023-10-17T19:09:56.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:08:12.095568Z" 2023-10-17T19:09:56.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:09:56.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.760 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.760 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004736", 2023-10-17T19:09:56.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.761 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:08:12.096952Z" 2023-10-17T19:09:56.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.762 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:09:56.762 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.763 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:09:56.763 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012768", 2023-10-17T19:09:56.763 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:08:12.098244Z" 2023-10-17T19:09:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.764 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:09:56.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.765 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:09:56.765 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T19:09:56.765 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T19:09:56.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.766 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:09:56Z" 2023-10-17T19:09:56.766 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:09:56.767 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:08:12.093848Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008564", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:08:12.095568Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:08:12.096952Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:08:12.098244Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T19:09:56Z" } 2023-10-17T19:09:56.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:08:12.093848Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008564', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:08:12.095568Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:08:12.096952Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:08:12.098244Z'}}] 2023-10-17T19:09:56.768 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009955', 1: '00000000000000000000:00000000000000008564', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012768'} 2023-10-17T19:09:56.768 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009955', 1: '00000000000000000000:00000000000000008564', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012768'} 2023-10-17T19:09:56.769 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:09:56.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-10-17T19:09:56.771 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:09:56.771 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:09:56.772 DEBUG:boto:Method: PUT 2023-10-17T19:09:56.772 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:09:56.772 DEBUG:boto:Data: 2023-10-17T19:09:56.773 DEBUG:boto:Headers: {} 2023-10-17T19:09:56.773 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:56.774 DEBUG:boto:Port: 8000 2023-10-17T19:09:56.774 DEBUG:boto:Params: {} 2023-10-17T19:09:56.774 DEBUG:boto:Token: None 2023-10-17T19:09:56.775 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:09:56 GMT /hbmuog-62/ 2023-10-17T19:09:56.775 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:gO+hhixUUMj/jDS9qbYM49ByW1U= 2023-10-17T19:09:56.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gO+hhixUUMj/jDS9qbYM49ByW1U=', 'Content-Length': '0'} 2023-10-17T19:09:56.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3829fc3bb863ffa-00652edc04-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:09:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:56.792 DEBUG:rgw_multi.tests:created bucket=hbmuog-62 2023-10-17T19:09:56.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.793 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.844 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.844 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.875 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:09:56.890 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:09:56.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.890 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569796.790160_821.1", 2023-10-17T19:09:56.891 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:09:56.790160Z" 2023-10-17T19:09:56.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.891 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:09:56.891 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:09:56.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:09:56.892 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:09:56.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:09:56.893 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:09:56.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.894 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:09:56.894 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697569796.790160_821.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:09:56.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:09:56.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:09:56.896 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:09:56.947 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:56.948 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:56.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:56.978+0000 7f3855a94780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:09:56.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:56.980+0000 7f3855a94780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:09:56.995 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:09:56.995 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:09:56.996 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:09:56.996 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:09:56.996 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:09:56.996 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:09:56.996 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:09:56.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:56.997 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:09:56.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:56.997 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:09:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:09:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569796.790160_821.1", 2023-10-17T19:09:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:56.998 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:09:56.999 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:56.999 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:09:56.790160Z", 2023-10-17T19:09:56.999 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:09:56.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:56.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:09:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:09:57.000 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:09:57.001 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:57.001 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:57.001 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:57.001 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:09:57.002 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:09:57.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:57.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:57.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:57.002 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:09:57.003 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:57.003 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:09:57.003 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:09:57.003 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:57.003 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:57.004 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:57.004 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:09:57.004 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:09:57.004 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:57.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:57.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:09:57.005 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:09:57.005 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:09:57.005 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:09:57.006 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:09:57.006 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:09:57.006 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:09:57.006 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:09:57.007 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:09:57.007 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:09:57.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:57.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:09:57.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:09:57.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:57.008 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:09:57.009 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:09:57.009 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:09:57.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:09:57.009 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:09:56Z" 2023-10-17T19:09:57.009 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:09:57.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569796.790160_821.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:09:56.790160Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:09:56Z" } 2023-10-17T19:09:57.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569796.790160_821.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:09:56.790160Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:09:57.012 DEBUG:rgw_multi.tests:log_status={0: '1_1697569796.790160_821.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:09:57.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569796.790160_821.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:09:57.012 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:09:57.013 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:09:57.013 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:09:57.014 DEBUG:boto:Method: HEAD 2023-10-17T19:09:57.014 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:09:57.014 DEBUG:boto:Data: 2023-10-17T19:09:57.015 DEBUG:boto:Headers: {} 2023-10-17T19:09:57.015 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:57.016 DEBUG:boto:Port: 8000 2023-10-17T19:09:57.016 DEBUG:boto:Params: {} 2023-10-17T19:09:57.017 DEBUG:boto:Token: None 2023-10-17T19:09:57.017 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:09:57 GMT /hbmuog-62/ 2023-10-17T19:09:57.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Or3BissP56vmnCWG3lWpQOQ9qaw= 2023-10-17T19:09:57.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:09:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Or3BissP56vmnCWG3lWpQOQ9qaw=', 'Content-Length': '0'} 2023-10-17T19:09:57.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a40951bb2f870680-00652edc05-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:57.021 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:09:57.021 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:09:57.022 DEBUG:boto:Method: PUT 2023-10-17T19:09:57.022 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:09:57.022 DEBUG:boto:Data: 2023-10-17T19:09:57.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:09:57.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:09:57.024 DEBUG:boto:Port: 8000 2023-10-17T19:09:57.024 DEBUG:boto:Params: {} 2023-10-17T19:09:57.024 DEBUG:boto:Token: None 2023-10-17T19:09:57.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:09:57 GMT /hbmuog-62/dummy 2023-10-17T19:09:57.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:knYRC5DqD7zUdt/q5+5j/M3oEn8= 2023-10-17T19:09:57.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:09:57 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:knYRC5DqD7zUdt/q5+5j/M3oEn8='} 2023-10-17T19:09:57.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007864c336dd0756a2-00652edc05-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:09:57.033 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-62 2023-10-17T19:09:57.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:09:57.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:09:57.034 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:09:57.092 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:09:57.093 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:09:57.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:09:57.126+0000 7f643a2a6780 1 waiting to reach incremental sync.. 2023-10-17T19:10:02.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:02.127+0000 7f643a2a6780 1 waiting to reach incremental sync.. 2023-10-17T19:10:07.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:07.128+0000 7f643a2a6780 1 waiting to reach incremental sync.. 2023-10-17T19:10:12.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:12.129+0000 7f643a2a6780 1 waiting to reach incremental sync.. 2023-10-17T19:10:17.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:17.131+0000 7f643a2a6780 1 bucket sync caught up with source: 2023-10-17T19:10:17.132 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:17.132 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:17.132 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:17.131+0000 7f643a2a6780 0 bucket checkpoint complete 2023-10-17T19:10:17.149 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-62 2023-10-17T19:10:17.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:10:17.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:10:17.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:10:17.217 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:10:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:10:17.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:17.252+0000 7f76fcab8780 1 waiting to reach incremental sync.. 2023-10-17T19:10:22.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:22.252+0000 7f76fcab8780 1 waiting to reach incremental sync.. 2023-10-17T19:10:27.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:27.253+0000 7f76fcab8780 1 waiting to reach incremental sync.. 2023-10-17T19:10:32.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:32.254+0000 7f76fcab8780 1 bucket sync caught up with source: 2023-10-17T19:10:32.256 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , , , ] 2023-10-17T19:10:32.256 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , , , ] 2023-10-17T19:10:32.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:32.254+0000 7f76fcab8780 0 bucket checkpoint complete 2023-10-17T19:10:32.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-62 zones={a2, a1} 2023-10-17T19:10:32.275 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.275 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.276 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.276 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:32.277 DEBUG:boto:Data: 2023-10-17T19:10:32.277 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.278 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.278 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.278 DEBUG:boto:Params: {} 2023-10-17T19:10:32.279 DEBUG:boto:Token: None 2023-10-17T19:10:32.280 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4Yl8nAPIiLN7M/XccPfdEef9NxM= 2023-10-17T19:10:32.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4Yl8nAPIiLN7M/XccPfdEef9NxM=', 'Content-Length': '0'} 2023-10-17T19:10:32.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3af111c265f9f4d-00652edc28-1cee-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.284 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.285 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.285 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.285 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:32.286 DEBUG:boto:Data: 2023-10-17T19:10:32.286 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.287 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.288 DEBUG:boto:Params: {} 2023-10-17T19:10:32.288 DEBUG:boto:Token: None 2023-10-17T19:10:32.289 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.289 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4Yl8nAPIiLN7M/XccPfdEef9NxM= 2023-10-17T19:10:32.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4Yl8nAPIiLN7M/XccPfdEef9NxM=', 'Content-Length': '0'} 2023-10-17T19:10:32.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fee85aee95b66db1-00652edc28-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T19:10:32.294 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.294 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.295 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.295 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.295 DEBUG:boto:Method: GET 2023-10-17T19:10:32.296 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.296 DEBUG:boto:Data: 2023-10-17T19:10:32.297 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.297 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.298 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.298 DEBUG:boto:Params: {} 2023-10-17T19:10:32.299 DEBUG:boto:Token: None 2023-10-17T19:10:32.299 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/?versions 2023-10-17T19:10:32.300 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4= 2023-10-17T19:10:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4=', 'Content-Length': '0'} 2023-10-17T19:10:32.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0be2e8c68ff177-00652edc28-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.346 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.348 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T19:10:32.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T19:10:32.349 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.349 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.350 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.350 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.351 DEBUG:boto:Method: GET 2023-10-17T19:10:32.351 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.351 DEBUG:boto:Data: 2023-10-17T19:10:32.352 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.352 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.353 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.353 DEBUG:boto:Params: {} 2023-10-17T19:10:32.354 DEBUG:boto:Token: None 2023-10-17T19:10:32.354 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/?versions 2023-10-17T19:10:32.355 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4= 2023-10-17T19:10:32.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4=', 'Content-Length': '0'} 2023-10-17T19:10:32.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250cd94248e43c60-00652edc28-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.359 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T19:10:32.360 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.360 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.361 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.361 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.362 DEBUG:boto:Method: GET 2023-10-17T19:10:32.362 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.362 DEBUG:boto:Data: 2023-10-17T19:10:32.363 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.364 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.364 DEBUG:boto:Params: {} 2023-10-17T19:10:32.365 DEBUG:boto:Token: None 2023-10-17T19:10:32.365 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/?versions 2023-10-17T19:10:32.366 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4= 2023-10-17T19:10:32.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4=', 'Content-Length': '0'} 2023-10-17T19:10:32.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d48f3d28be3a5f-00652edc28-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.370 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.370 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.371 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.371 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.371 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.372 DEBUG:boto:Method: GET 2023-10-17T19:10:32.372 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:10:32.373 DEBUG:boto:Data: 2023-10-17T19:10:32.373 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.374 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.374 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.375 DEBUG:boto:Params: {} 2023-10-17T19:10:32.375 DEBUG:boto:Token: None 2023-10-17T19:10:32.375 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/?versions 2023-10-17T19:10:32.376 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4= 2023-10-17T19:10:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJalgM+Op1VertI8MSQSmTKdCQ4=', 'Content-Length': '0'} 2023-10-17T19:10:32.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c313f44399d6a67-00652edc28-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.380 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:10:32.381 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.382 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.382 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.383 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.383 DEBUG:boto:Method: GET 2023-10-17T19:10:32.383 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.384 DEBUG:boto:Data: 2023-10-17T19:10:32.384 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.385 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.385 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.386 DEBUG:boto:Params: {} 2023-10-17T19:10:32.386 DEBUG:boto:Token: None 2023-10-17T19:10:32.387 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8= 2023-10-17T19:10:32.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8=', 'Content-Length': '0'} 2023-10-17T19:10:32.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701bfafeb988844d-00652edc28-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.391 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.391 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.391 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.392 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.392 DEBUG:boto:Method: GET 2023-10-17T19:10:32.393 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.393 DEBUG:boto:Data: 2023-10-17T19:10:32.393 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.394 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.394 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.395 DEBUG:boto:Params: {} 2023-10-17T19:10:32.395 DEBUG:boto:Token: None 2023-10-17T19:10:32.396 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.396 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8= 2023-10-17T19:10:32.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8=', 'Content-Length': '0'} 2023-10-17T19:10:32.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a968158f3b2ae1e-00652edc28-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.399 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.400 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.400 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.400 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.401 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.401 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.402 DEBUG:boto:Data: 2023-10-17T19:10:32.402 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.402 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.403 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.403 DEBUG:boto:Params: {} 2023-10-17T19:10:32.404 DEBUG:boto:Token: None 2023-10-17T19:10:32.404 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.404 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sw2/37ceQc9RHfDE7g+bH0Nu2Qg= 2023-10-17T19:10:32.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sw2/37ceQc9RHfDE7g+bH0Nu2Qg=', 'Content-Length': '0'} 2023-10-17T19:10:32.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa2150c439f8605a-00652edc28-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.407 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.408 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.408 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.409 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.409 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.410 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.410 DEBUG:boto:Data: 2023-10-17T19:10:32.410 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.411 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.411 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.412 DEBUG:boto:Params: {} 2023-10-17T19:10:32.412 DEBUG:boto:Token: None 2023-10-17T19:10:32.413 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.413 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sw2/37ceQc9RHfDE7g+bH0Nu2Qg= 2023-10-17T19:10:32.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sw2/37ceQc9RHfDE7g+bH0Nu2Qg=', 'Content-Length': '0'} 2023-10-17T19:10:32.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3d6f8377d3eecc-00652edc28-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:10:32.416 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.417 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.417 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.418 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.418 DEBUG:boto:Method: GET 2023-10-17T19:10:32.418 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.419 DEBUG:boto:Data: 2023-10-17T19:10:32.419 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.420 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.420 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.421 DEBUG:boto:Params: {} 2023-10-17T19:10:32.421 DEBUG:boto:Token: None 2023-10-17T19:10:32.422 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.422 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8= 2023-10-17T19:10:32.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8=', 'Content-Length': '0'} 2023-10-17T19:10:32.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6261adc06c615cd-00652edc28-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.425 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.426 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.426 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.427 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.427 DEBUG:boto:Method: GET 2023-10-17T19:10:32.427 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.428 DEBUG:boto:Data: 2023-10-17T19:10:32.429 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.429 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.430 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.430 DEBUG:boto:Params: {} 2023-10-17T19:10:32.430 DEBUG:boto:Token: None 2023-10-17T19:10:32.431 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:10:32.432 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8= 2023-10-17T19:10:32.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8F7loqnJtAGpR8wVlWjkU368Wk8=', 'Content-Length': '0'} 2023-10-17T19:10:32.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892c0628ade2d973-00652edc28-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.435 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.435 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.436 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.436 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.436 DEBUG:boto:Method: GET 2023-10-17T19:10:32.437 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.437 DEBUG:boto:Data: 2023-10-17T19:10:32.438 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.439 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.439 DEBUG:boto:Params: {} 2023-10-17T19:10:32.440 DEBUG:boto:Token: None 2023-10-17T19:10:32.440 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.440 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes= 2023-10-17T19:10:32.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes=', 'Content-Length': '0'} 2023-10-17T19:10:32.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff1bb334eaedd2c-00652edc28-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.444 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.445 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.445 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.446 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.446 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.447 DEBUG:boto:Method: GET 2023-10-17T19:10:32.447 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.447 DEBUG:boto:Data: 2023-10-17T19:10:32.448 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.448 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.449 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.449 DEBUG:boto:Params: {} 2023-10-17T19:10:32.450 DEBUG:boto:Token: None 2023-10-17T19:10:32.450 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.451 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes= 2023-10-17T19:10:32.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes=', 'Content-Length': '0'} 2023-10-17T19:10:32.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aaee36914eed125-00652edc28-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.454 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.455 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.455 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.456 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.456 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:10:32.456 DEBUG:boto:Data: 2023-10-17T19:10:32.457 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.458 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.458 DEBUG:boto:Params: {} 2023-10-17T19:10:32.458 DEBUG:boto:Token: None 2023-10-17T19:10:32.459 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy 2023-10-17T19:10:32.459 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IHsfswa7RC0yb3I1wafV3wceqB0= 2023-10-17T19:10:32.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IHsfswa7RC0yb3I1wafV3wceqB0=', 'Content-Length': '0'} 2023-10-17T19:10:32.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001353f3498bc6590-00652edc28-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.463 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.463 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.463 DEBUG:boto:Method: HEAD 2023-10-17T19:10:32.464 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:10:32.464 DEBUG:boto:Data: 2023-10-17T19:10:32.465 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.465 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.466 DEBUG:boto:Params: {} 2023-10-17T19:10:32.466 DEBUG:boto:Token: None 2023-10-17T19:10:32.467 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy 2023-10-17T19:10:32.467 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IHsfswa7RC0yb3I1wafV3wceqB0= 2023-10-17T19:10:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IHsfswa7RC0yb3I1wafV3wceqB0=', 'Content-Length': '0'} 2023-10-17T19:10:32.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b5bce458318e6f4-00652edc28-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:10:32.471 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.471 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.472 DEBUG:boto:Method: GET 2023-10-17T19:10:32.472 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:10:32.472 DEBUG:boto:Data: 2023-10-17T19:10:32.473 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.473 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.474 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.474 DEBUG:boto:Params: {} 2023-10-17T19:10:32.475 DEBUG:boto:Token: None 2023-10-17T19:10:32.475 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy 2023-10-17T19:10:32.476 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v18cgvuqyhTd3DL3uMfRJX1W5QU= 2023-10-17T19:10:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v18cgvuqyhTd3DL3uMfRJX1W5QU=', 'Content-Length': '0'} 2023-10-17T19:10:32.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee5b8f97c0f85f6-00652edc28-1cee-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.518 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:10:32.519 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:10:32.519 DEBUG:boto:Method: GET 2023-10-17T19:10:32.520 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:10:32.520 DEBUG:boto:Data: 2023-10-17T19:10:32.521 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.521 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.522 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.522 DEBUG:boto:Params: {} 2023-10-17T19:10:32.523 DEBUG:boto:Token: None 2023-10-17T19:10:32.523 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/dummy 2023-10-17T19:10:32.524 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:v18cgvuqyhTd3DL3uMfRJX1W5QU= 2023-10-17T19:10:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v18cgvuqyhTd3DL3uMfRJX1W5QU=', 'Content-Length': '0'} 2023-10-17T19:10:32.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054fd73d4438a96b3-00652edc28-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.527 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.527 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.528 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.528 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.529 DEBUG:boto:Method: GET 2023-10-17T19:10:32.529 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.529 DEBUG:boto:Data: 2023-10-17T19:10:32.530 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.530 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:32.531 DEBUG:boto:Port: 8001 2023-10-17T19:10:32.531 DEBUG:boto:Params: {} 2023-10-17T19:10:32.532 DEBUG:boto:Token: None 2023-10-17T19:10:32.532 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.533 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes= 2023-10-17T19:10:32.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes=', 'Content-Length': '0'} 2023-10-17T19:10:32.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e7ff5496b8c78a-00652edc28-1cee-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.536 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.537 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:32.538 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:32.538 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.538 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.539 DEBUG:boto:Method: GET 2023-10-17T19:10:32.539 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:10:32.540 DEBUG:boto:Data: 2023-10-17T19:10:32.540 DEBUG:boto:Headers: {} 2023-10-17T19:10:32.541 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:32.541 DEBUG:boto:Port: 8000 2023-10-17T19:10:32.541 DEBUG:boto:Params: {} 2023-10-17T19:10:32.542 DEBUG:boto:Token: None 2023-10-17T19:10:32.542 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:10:32 GMT /hbmuog-62/ 2023-10-17T19:10:32.543 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes= 2023-10-17T19:10:32.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:32 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FGLCUEe5OvTw7mnT2hqvDNMLzes=', 'Content-Length': '0'} 2023-10-17T19:10:32.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ef94051800bb4a-00652edc28-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:32.590 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:10:32.591 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-62 zones={a2, a1} 2023-10-17T19:10:32.592 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T19:10:32.592 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:10:32.593 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:10:32.591+0000 7f98e720f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 167407) UID: 0 2023-10-17T19:10:32.593 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:10:32.591+0000 7f98eb4787c0 -1 shutting down 2023-10-17T19:10:35.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:10:38.691 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T19:10:38.692 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:38.693 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:38.693 DEBUG:boto:Method: HEAD 2023-10-17T19:10:38.693 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:38.694 DEBUG:boto:Data: 2023-10-17T19:10:38.694 DEBUG:boto:Headers: {} 2023-10-17T19:10:38.695 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.695 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.695 DEBUG:boto:Params: {} 2023-10-17T19:10:38.696 DEBUG:boto:Token: None 2023-10-17T19:10:38.696 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/ 2023-10-17T19:10:38.697 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk= 2023-10-17T19:10:38.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk=', 'Content-Length': '0'} 2023-10-17T19:10:38.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000257dfdc04f454340-00652edc2e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.700 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:10:38.701 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:10:38.701 DEBUG:boto:Method: PUT 2023-10-17T19:10:38.702 DEBUG:boto:Path: /hbmuog-62/obj11 2023-10-17T19:10:38.702 DEBUG:boto:Data: 2023-10-17T19:10:38.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:10:38.703 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.703 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.704 DEBUG:boto:Params: {} 2023-10-17T19:10:38.704 DEBUG:boto:Token: None 2023-10-17T19:10:38.705 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/obj11 2023-10-17T19:10:38.705 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LuDfJN8z7eIHLyxqcAs2Csm3H5E= 2023-10-17T19:10:38.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LuDfJN8z7eIHLyxqcAs2Csm3H5E='} 2023-10-17T19:10:38.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6ac8ce899ff83dd-00652edc2e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.713 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:38.713 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:38.713 DEBUG:boto:Method: HEAD 2023-10-17T19:10:38.714 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:38.714 DEBUG:boto:Data: 2023-10-17T19:10:38.715 DEBUG:boto:Headers: {} 2023-10-17T19:10:38.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.716 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.716 DEBUG:boto:Params: {} 2023-10-17T19:10:38.717 DEBUG:boto:Token: None 2023-10-17T19:10:38.717 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/ 2023-10-17T19:10:38.718 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk= 2023-10-17T19:10:38.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk=', 'Content-Length': '0'} 2023-10-17T19:10:38.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b70289d3335685b-00652edc2e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.721 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:10:38.722 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:10:38.722 DEBUG:boto:Method: PUT 2023-10-17T19:10:38.723 DEBUG:boto:Path: /hbmuog-62/obj22 2023-10-17T19:10:38.723 DEBUG:boto:Data: 2023-10-17T19:10:38.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:10:38.724 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.724 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.725 DEBUG:boto:Params: {} 2023-10-17T19:10:38.725 DEBUG:boto:Token: None 2023-10-17T19:10:38.726 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/obj22 2023-10-17T19:10:38.726 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k5abv8/bCFbpq7udG2A5KXEtdck= 2023-10-17T19:10:38.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k5abv8/bCFbpq7udG2A5KXEtdck='} 2023-10-17T19:10:38.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e15d43195e4ee986-00652edc2e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.732 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:38.732 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:38.732 DEBUG:boto:Method: HEAD 2023-10-17T19:10:38.733 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:38.733 DEBUG:boto:Data: 2023-10-17T19:10:38.734 DEBUG:boto:Headers: {} 2023-10-17T19:10:38.734 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.734 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.735 DEBUG:boto:Params: {} 2023-10-17T19:10:38.735 DEBUG:boto:Token: None 2023-10-17T19:10:38.736 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/ 2023-10-17T19:10:38.736 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk= 2023-10-17T19:10:38.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk=', 'Content-Length': '0'} 2023-10-17T19:10:38.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da86be791e6eac43-00652edc2e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.740 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:10:38.741 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:10:38.741 DEBUG:boto:Method: PUT 2023-10-17T19:10:38.742 DEBUG:boto:Path: /hbmuog-62/obj33 2023-10-17T19:10:38.742 DEBUG:boto:Data: 2023-10-17T19:10:38.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:10:38.743 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.743 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.744 DEBUG:boto:Params: {} 2023-10-17T19:10:38.744 DEBUG:boto:Token: None 2023-10-17T19:10:38.745 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/obj33 2023-10-17T19:10:38.745 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JTu5xb3cnVxz7C/WTY+058exttQ= 2023-10-17T19:10:38.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JTu5xb3cnVxz7C/WTY+058exttQ='} 2023-10-17T19:10:38.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa60675c94c4f230-00652edc2e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.795 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:38.796 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:38.796 DEBUG:boto:Method: HEAD 2023-10-17T19:10:38.797 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:38.797 DEBUG:boto:Data: 2023-10-17T19:10:38.798 DEBUG:boto:Headers: {} 2023-10-17T19:10:38.798 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.799 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.799 DEBUG:boto:Params: {} 2023-10-17T19:10:38.799 DEBUG:boto:Token: None 2023-10-17T19:10:38.800 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/ 2023-10-17T19:10:38.800 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk= 2023-10-17T19:10:38.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk=', 'Content-Length': '0'} 2023-10-17T19:10:38.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000959468762e5d3aba-00652edc2e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.804 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:10:38.805 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:10:38.805 DEBUG:boto:Method: PUT 2023-10-17T19:10:38.806 DEBUG:boto:Path: /hbmuog-62/obj44 2023-10-17T19:10:38.806 DEBUG:boto:Data: 2023-10-17T19:10:38.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:10:38.807 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.807 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.808 DEBUG:boto:Params: {} 2023-10-17T19:10:38.808 DEBUG:boto:Token: None 2023-10-17T19:10:38.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/obj44 2023-10-17T19:10:38.809 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:u+5EcweSoiRCYxsRv31LSdNNcW8= 2023-10-17T19:10:38.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:u+5EcweSoiRCYxsRv31LSdNNcW8='} 2023-10-17T19:10:38.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0619a1dee16a8c8-00652edc2e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.817 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:10:38.818 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:10:38.818 DEBUG:boto:Method: HEAD 2023-10-17T19:10:38.818 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:10:38.819 DEBUG:boto:Data: 2023-10-17T19:10:38.819 DEBUG:boto:Headers: {} 2023-10-17T19:10:38.820 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.820 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.820 DEBUG:boto:Params: {} 2023-10-17T19:10:38.821 DEBUG:boto:Token: None 2023-10-17T19:10:38.821 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/ 2023-10-17T19:10:38.822 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk= 2023-10-17T19:10:38.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bVn4bg3szhWCIqGOLxIIn9lDlAk=', 'Content-Length': '0'} 2023-10-17T19:10:38.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5579fa10206f2dc-00652edc2e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.825 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:10:38.826 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:10:38.826 DEBUG:boto:Method: PUT 2023-10-17T19:10:38.827 DEBUG:boto:Path: /hbmuog-62/obj55 2023-10-17T19:10:38.827 DEBUG:boto:Data: 2023-10-17T19:10:38.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:10:38.828 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:10:38.828 DEBUG:boto:Port: 8000 2023-10-17T19:10:38.829 DEBUG:boto:Params: {} 2023-10-17T19:10:38.829 DEBUG:boto:Token: None 2023-10-17T19:10:38.830 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:10:38 GMT /hbmuog-62/obj55 2023-10-17T19:10:38.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hxfxz2q1qVAkrL/lhM3PeEWzo5M= 2023-10-17T19:10:38.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:10:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hxfxz2q1qVAkrL/lhM3PeEWzo5M='} 2023-10-17T19:10:38.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000751080c9c8ae58b6-00652edc2e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:10:38.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:10:38.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:10:38.876 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:10:38.937 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:10:38.937 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:10:38.973 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:38.971+0000 7f1c7e6f5780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-17T19:10:38.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:38.972+0000 7f1c7e6f5780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-17T19:10:38.997 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T19:10:38.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T19:10:39.000 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T19:10:39.001 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T19:10:39.025 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:10:39.025 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:10:39.025 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T19:10:40.027 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:10:40.049 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:10:40.050 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:10:40.051 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T19:10:40.052 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T19:10:40.052 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-62 2023-10-17T19:10:40.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:10:40.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:10:40.053 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:10:40.149 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:10:40.150 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:10:40.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:40.188+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:10:40.190 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:40.191 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:10:41.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T19:10:45.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:45.190+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:10:45.191 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:45.192 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:10:50.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:50.192+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:10:50.193 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:50.193 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:10:55.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:10:55.192+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:10:55.194 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:10:55.195 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:00.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:00.194+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:00.196 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:00.196 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:05.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:05.195+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:05.197 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:05.197 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:10.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:10.196+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:10.198 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:10.199 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:15.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:15.198+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:15.200 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:15.200 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:20.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:20.200+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:20.201 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:20.201 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:25.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:25.201+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:25.203 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:25.203 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:30.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:30.203+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:30.204 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:30.204 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:35.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:35.203+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:35.205 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:35.205 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:40.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:40.204+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:40.206 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:40.206 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:45.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:45.205+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:45.207 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:45.207 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:50.206+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:50.208 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:50.209 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:11:55.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:11:55.209+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:11:55.210 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:11:55.210 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:00.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:00.210+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:00.211 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:12:00.211 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:05.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:05.211+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:05.212 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , , ] 2023-10-17T19:12:05.213 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:10.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:10.212+0000 7f2289e97780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:10.214 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , , 00000000001.81.6, ] 2023-10-17T19:12:10.214 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:15.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:15.213+0000 7f2289e97780 1 bucket sync caught up with source: 2023-10-17T19:12:15.215 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:15.215 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000001.70.6, 00000000002.83.6, 00000000002.68.6] 2023-10-17T19:12:15.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:15.213+0000 7f2289e97780 0 bucket checkpoint complete 2023-10-17T19:12:15.233 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-62 2023-10-17T19:12:15.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:12:15.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:12:15.235 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:12:15.301 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:15.302 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:15.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:15.341+0000 7f24fe490780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:15.343 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , , 00000000001.282.6, ] 2023-10-17T19:12:15.343 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:20.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:20.342+0000 7f24fe490780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:20.344 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , , 00000000001.282.6, ] 2023-10-17T19:12:20.345 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:25.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:25.344+0000 7f24fe490780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:25.346 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , , 00000000001.282.6, ] 2023-10-17T19:12:25.346 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:30.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:30.346+0000 7f24fe490780 1 waiting for incremental sync to catch up: 2023-10-17T19:12:30.350 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , , 00000000001.282.6, ] 2023-10-17T19:12:30.351 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:35.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:35.347+0000 7f24fe490780 1 bucket sync caught up with source: 2023-10-17T19:12:35.348 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:35.349 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , 00000000001.225.6, , , 00000000001.68.6, 00000000002.284.6, 00000000002.66.6] 2023-10-17T19:12:35.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:35.347+0000 7f24fe490780 0 bucket checkpoint complete 2023-10-17T19:12:35.366 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-62 zones={a2, a1} 2023-10-17T19:12:35.367 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.367 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.368 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.368 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:12:35.369 DEBUG:boto:Data: 2023-10-17T19:12:35.369 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.370 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.370 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.371 DEBUG:boto:Params: {} 2023-10-17T19:12:35.372 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:12:35.372 DEBUG:boto:Token: None 2023-10-17T19:12:35.373 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.373 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:obXSu68yLkvhXwzgWIUpbnhQOOk= 2023-10-17T19:12:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:obXSu68yLkvhXwzgWIUpbnhQOOk=', 'Content-Length': '0'} 2023-10-17T19:12:35.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa9c6df15d7d1880-00652edca3-1d4d-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.381 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.381 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.382 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.382 DEBUG:boto:Path: /hbmuog-62/ 2023-10-17T19:12:35.383 DEBUG:boto:Data: 2023-10-17T19:12:35.383 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.384 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.384 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.385 DEBUG:boto:Params: {} 2023-10-17T19:12:35.385 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:12:35.386 DEBUG:boto:Token: None 2023-10-17T19:12:35.386 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.387 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:obXSu68yLkvhXwzgWIUpbnhQOOk= 2023-10-17T19:12:35.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:obXSu68yLkvhXwzgWIUpbnhQOOk=', 'Content-Length': '0'} 2023-10-17T19:12:35.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd3bd44c10f1f708-00652edca3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T19:12:35.392 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.393 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.393 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.394 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.394 DEBUG:boto:Method: GET 2023-10-17T19:12:35.395 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.395 DEBUG:boto:Data: 2023-10-17T19:12:35.396 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.396 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.397 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.397 DEBUG:boto:Params: {} 2023-10-17T19:12:35.397 DEBUG:boto:Token: None 2023-10-17T19:12:35.398 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/?versions 2023-10-17T19:12:35.398 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c= 2023-10-17T19:12:35.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c=', 'Content-Length': '0'} 2023-10-17T19:12:35.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008002f2d7032085ff-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.402 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.404 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T19:12:35.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:12:35.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:12:35.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:12:35.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:12:35.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:12:35.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T19:12:35.407 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.408 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.408 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.409 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.410 DEBUG:boto:Method: GET 2023-10-17T19:12:35.410 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.411 DEBUG:boto:Data: 2023-10-17T19:12:35.411 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.412 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.412 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.412 DEBUG:boto:Params: {} 2023-10-17T19:12:35.413 DEBUG:boto:Token: None 2023-10-17T19:12:35.413 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/?versions 2023-10-17T19:12:35.414 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c= 2023-10-17T19:12:35.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c=', 'Content-Length': '0'} 2023-10-17T19:12:35.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000671c3f4e4290e88d-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.418 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.419 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-17T19:12:35.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:12:35.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:12:35.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:12:35.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:12:35.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:12:35.422 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.422 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.423 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.423 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.424 DEBUG:boto:Method: GET 2023-10-17T19:12:35.424 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.425 DEBUG:boto:Data: 2023-10-17T19:12:35.425 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.426 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.427 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.428 DEBUG:boto:Params: {} 2023-10-17T19:12:35.428 DEBUG:boto:Token: None 2023-10-17T19:12:35.429 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/?versions 2023-10-17T19:12:35.430 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c= 2023-10-17T19:12:35.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c=', 'Content-Length': '0'} 2023-10-17T19:12:35.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010fa3fddf705633c-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.435 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.436 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.436 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.437 DEBUG:boto:path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.437 DEBUG:boto:auth_path=/hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.438 DEBUG:boto:Method: GET 2023-10-17T19:12:35.438 DEBUG:boto:Path: /hbmuog-62/?versions&max-keys=999 2023-10-17T19:12:35.439 DEBUG:boto:Data: 2023-10-17T19:12:35.439 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.440 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.440 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.441 DEBUG:boto:Params: {} 2023-10-17T19:12:35.441 DEBUG:boto:Token: None 2023-10-17T19:12:35.442 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/?versions 2023-10-17T19:12:35.442 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c= 2023-10-17T19:12:35.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AYW2Hl+ylyRXjPLcdImH7A+Wm9c=', 'Content-Length': '0'} 2023-10-17T19:12:35.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb371215cc63db8-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.446 DEBUG:boto:b'hbmuog-62999falsedummynulltrue2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:12:35.447 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.448 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.448 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.449 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.449 DEBUG:boto:Method: GET 2023-10-17T19:12:35.450 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.450 DEBUG:boto:Data: 2023-10-17T19:12:35.451 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.451 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.452 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.452 DEBUG:boto:Params: {} 2023-10-17T19:12:35.452 DEBUG:boto:Token: None 2023-10-17T19:12:35.453 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.453 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E= 2023-10-17T19:12:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E=', 'Content-Length': '0'} 2023-10-17T19:12:35.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d3da3245e74a32-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.458 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.458 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.459 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.459 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.460 DEBUG:boto:Method: GET 2023-10-17T19:12:35.460 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.461 DEBUG:boto:Data: 2023-10-17T19:12:35.461 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.462 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.463 DEBUG:boto:Params: {} 2023-10-17T19:12:35.463 DEBUG:boto:Token: None 2023-10-17T19:12:35.464 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E= 2023-10-17T19:12:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E=', 'Content-Length': '0'} 2023-10-17T19:12:35.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ecd708e87fe24c5-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.467 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.468 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.468 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.469 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.469 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.470 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.470 DEBUG:boto:Data: 2023-10-17T19:12:35.471 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.471 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.471 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.472 DEBUG:boto:Params: {} 2023-10-17T19:12:35.473 DEBUG:boto:Token: None 2023-10-17T19:12:35.473 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.473 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kil9aNyt/dhfz8IXS9kqNlIxgRQ= 2023-10-17T19:12:35.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kil9aNyt/dhfz8IXS9kqNlIxgRQ=', 'Content-Length': '0'} 2023-10-17T19:12:35.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0d71389cd35450-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.476 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.477 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.477 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.478 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.478 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.479 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.479 DEBUG:boto:Data: 2023-10-17T19:12:35.480 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.480 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.481 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.481 DEBUG:boto:Params: {} 2023-10-17T19:12:35.481 DEBUG:boto:Token: None 2023-10-17T19:12:35.482 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.482 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Kil9aNyt/dhfz8IXS9kqNlIxgRQ= 2023-10-17T19:12:35.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Kil9aNyt/dhfz8IXS9kqNlIxgRQ=', 'Content-Length': '0'} 2023-10-17T19:12:35.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f22d327d21ee326-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:12:35.486 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.487 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.487 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.489 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.490 DEBUG:boto:Method: GET 2023-10-17T19:12:35.490 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.491 DEBUG:boto:Data: 2023-10-17T19:12:35.491 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.492 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.492 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.492 DEBUG:boto:Params: {} 2023-10-17T19:12:35.493 DEBUG:boto:Token: None 2023-10-17T19:12:35.494 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.494 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E= 2023-10-17T19:12:35.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E=', 'Content-Length': '0'} 2023-10-17T19:12:35.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b585c174eaab9234-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.499 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.500 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.501 DEBUG:boto:path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.501 DEBUG:boto:auth_path=/hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.502 DEBUG:boto:Method: GET 2023-10-17T19:12:35.503 DEBUG:boto:Path: /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.503 DEBUG:boto:Data: 2023-10-17T19:12:35.504 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.504 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.505 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.506 DEBUG:boto:Params: {} 2023-10-17T19:12:35.506 DEBUG:boto:Token: None 2023-10-17T19:12:35.507 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy?versionId=null 2023-10-17T19:12:35.507 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E= 2023-10-17T19:12:35.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vnp1C8hUAPYa4Nqr/HIrulcjz7E=', 'Content-Length': '0'} 2023-10-17T19:12:35.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66626bd1f5d45a0-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.511 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.511 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.512 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.512 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.513 DEBUG:boto:Method: GET 2023-10-17T19:12:35.513 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.514 DEBUG:boto:Data: 2023-10-17T19:12:35.515 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.515 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.516 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.517 DEBUG:boto:Params: {} 2023-10-17T19:12:35.518 DEBUG:boto:Token: None 2023-10-17T19:12:35.519 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.519 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dabae5395d561ff8-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.524 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.525 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.525 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.526 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.526 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.527 DEBUG:boto:Method: GET 2023-10-17T19:12:35.527 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.528 DEBUG:boto:Data: 2023-10-17T19:12:35.528 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.529 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.529 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.530 DEBUG:boto:Params: {} 2023-10-17T19:12:35.530 DEBUG:boto:Token: None 2023-10-17T19:12:35.531 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.531 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f8a259d4f5c3dc-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.535 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.536 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.537 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.537 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.538 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:12:35.539 DEBUG:boto:Data: 2023-10-17T19:12:35.539 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.540 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.541 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.541 DEBUG:boto:Params: {} 2023-10-17T19:12:35.542 DEBUG:boto:Token: None 2023-10-17T19:12:35.542 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy 2023-10-17T19:12:35.543 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wEuIFwsyUSz7pmrWIVULEgBjeBU= 2023-10-17T19:12:35.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wEuIFwsyUSz7pmrWIVULEgBjeBU=', 'Content-Length': '0'} 2023-10-17T19:12:35.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a3570c36c39b7e-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.546 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.547 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.547 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.548 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:12:35.548 DEBUG:boto:Data: 2023-10-17T19:12:35.549 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.549 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.549 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.550 DEBUG:boto:Params: {} 2023-10-17T19:12:35.550 DEBUG:boto:Token: None 2023-10-17T19:12:35.551 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy 2023-10-17T19:12:35.551 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wEuIFwsyUSz7pmrWIVULEgBjeBU= 2023-10-17T19:12:35.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wEuIFwsyUSz7pmrWIVULEgBjeBU=', 'Content-Length': '0'} 2023-10-17T19:12:35.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64d06d5fcc633a1-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-17T19:12:35.555 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.555 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.556 DEBUG:boto:Method: GET 2023-10-17T19:12:35.556 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:12:35.557 DEBUG:boto:Data: 2023-10-17T19:12:35.557 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.558 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.558 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.559 DEBUG:boto:Params: {} 2023-10-17T19:12:35.559 DEBUG:boto:Token: None 2023-10-17T19:12:35.560 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy 2023-10-17T19:12:35.560 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GV8nfVYO+CuXWmMRaS+5v95mXaY= 2023-10-17T19:12:35.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GV8nfVYO+CuXWmMRaS+5v95mXaY=', 'Content-Length': '0'} 2023-10-17T19:12:35.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4068cf6240a0b14-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.564 DEBUG:boto:path=/hbmuog-62/dummy 2023-10-17T19:12:35.564 DEBUG:boto:auth_path=/hbmuog-62/dummy 2023-10-17T19:12:35.565 DEBUG:boto:Method: GET 2023-10-17T19:12:35.565 DEBUG:boto:Path: /hbmuog-62/dummy 2023-10-17T19:12:35.566 DEBUG:boto:Data: 2023-10-17T19:12:35.566 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.567 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.567 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.568 DEBUG:boto:Params: {} 2023-10-17T19:12:35.568 DEBUG:boto:Token: None 2023-10-17T19:12:35.569 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/dummy 2023-10-17T19:12:35.570 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GV8nfVYO+CuXWmMRaS+5v95mXaY= 2023-10-17T19:12:35.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GV8nfVYO+CuXWmMRaS+5v95mXaY=', 'Content-Length': '0'} 2023-10-17T19:12:35.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:09:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110dc18b7804a7f7-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.618 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.619 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.620 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.621 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.621 DEBUG:boto:Method: GET 2023-10-17T19:12:35.622 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.622 DEBUG:boto:Data: 2023-10-17T19:12:35.623 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.623 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.624 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.624 DEBUG:boto:Params: {} 2023-10-17T19:12:35.625 DEBUG:boto:Token: None 2023-10-17T19:12:35.625 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.626 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a59be6a0a7b4a2-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.629 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.630 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.631 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.631 DEBUG:boto:path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.632 DEBUG:boto:auth_path=/hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.632 DEBUG:boto:Method: GET 2023-10-17T19:12:35.632 DEBUG:boto:Path: /hbmuog-62/?prefix=dummy 2023-10-17T19:12:35.633 DEBUG:boto:Data: 2023-10-17T19:12:35.634 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.635 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.636 DEBUG:boto:Params: {} 2023-10-17T19:12:35.637 DEBUG:boto:Token: None 2023-10-17T19:12:35.638 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.639 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8196b6639524fc0-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.643 DEBUG:boto:b'hbmuog-62dummy1000falsedummy2023-10-17T19:09:57.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:12:35.644 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.645 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.645 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.646 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.646 DEBUG:boto:Method: GET 2023-10-17T19:12:35.647 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.647 DEBUG:boto:Data: 2023-10-17T19:12:35.647 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.648 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.649 DEBUG:boto:Params: {} 2023-10-17T19:12:35.649 DEBUG:boto:Token: None 2023-10-17T19:12:35.650 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM= 2023-10-17T19:12:35.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM=', 'Content-Length': '0'} 2023-10-17T19:12:35.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8053f1c40d24ba0-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.654 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.654 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.655 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.655 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.656 DEBUG:boto:Method: GET 2023-10-17T19:12:35.656 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.657 DEBUG:boto:Data: 2023-10-17T19:12:35.657 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.658 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.658 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.658 DEBUG:boto:Params: {} 2023-10-17T19:12:35.659 DEBUG:boto:Token: None 2023-10-17T19:12:35.660 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.661 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM= 2023-10-17T19:12:35.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM=', 'Content-Length': '0'} 2023-10-17T19:12:35.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6606a04ae52c65-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.665 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.665 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.666 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.666 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.667 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.667 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.667 DEBUG:boto:Data: 2023-10-17T19:12:35.668 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.669 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.669 DEBUG:boto:Params: {} 2023-10-17T19:12:35.670 DEBUG:boto:Token: None 2023-10-17T19:12:35.670 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.670 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0at8kBKLborHPsGjmJNF3JguKm0= 2023-10-17T19:12:35.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0at8kBKLborHPsGjmJNF3JguKm0=', 'Content-Length': '0'} 2023-10-17T19:12:35.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a3f3e045311dca-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.673 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.674 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.674 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.675 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.675 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.676 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.676 DEBUG:boto:Data: 2023-10-17T19:12:35.676 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.677 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.677 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.678 DEBUG:boto:Params: {} 2023-10-17T19:12:35.678 DEBUG:boto:Token: None 2023-10-17T19:12:35.679 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.679 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0at8kBKLborHPsGjmJNF3JguKm0= 2023-10-17T19:12:35.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0at8kBKLborHPsGjmJNF3JguKm0=', 'Content-Length': '0'} 2023-10-17T19:12:35.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6f94604f2ffff1-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:12:35.683 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.683 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.684 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.684 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.684 DEBUG:boto:Method: GET 2023-10-17T19:12:35.685 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.685 DEBUG:boto:Data: 2023-10-17T19:12:35.686 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.687 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.687 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.688 DEBUG:boto:Params: {} 2023-10-17T19:12:35.689 DEBUG:boto:Token: None 2023-10-17T19:12:35.689 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.690 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM= 2023-10-17T19:12:35.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM=', 'Content-Length': '0'} 2023-10-17T19:12:35.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc72f4234c2c91d8-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.697 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.698 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.698 DEBUG:boto:path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.699 DEBUG:boto:auth_path=/hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.699 DEBUG:boto:Method: GET 2023-10-17T19:12:35.700 DEBUG:boto:Path: /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.700 DEBUG:boto:Data: 2023-10-17T19:12:35.701 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.701 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.702 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.702 DEBUG:boto:Params: {} 2023-10-17T19:12:35.703 DEBUG:boto:Token: None 2023-10-17T19:12:35.703 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11?versionId=null 2023-10-17T19:12:35.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM= 2023-10-17T19:12:35.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9A0tdhOu72qPcnbAfiiqX/pMrUM=', 'Content-Length': '0'} 2023-10-17T19:12:35.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d101a97cc4395c02-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.708 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.709 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.709 DEBUG:boto:path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.710 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.710 DEBUG:boto:Method: GET 2023-10-17T19:12:35.711 DEBUG:boto:Path: /hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.711 DEBUG:boto:Data: 2023-10-17T19:12:35.711 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.712 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.712 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.713 DEBUG:boto:Params: {} 2023-10-17T19:12:35.714 DEBUG:boto:Token: None 2023-10-17T19:12:35.714 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.715 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4e1ca11b3af9dc-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.720 DEBUG:boto:b'hbmuog-62obj111000falseobj112023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.721 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.722 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.722 DEBUG:boto:path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.723 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.724 DEBUG:boto:Method: GET 2023-10-17T19:12:35.724 DEBUG:boto:Path: /hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.725 DEBUG:boto:Data: 2023-10-17T19:12:35.725 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.726 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.726 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.727 DEBUG:boto:Params: {} 2023-10-17T19:12:35.727 DEBUG:boto:Token: None 2023-10-17T19:12:35.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.728 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d52fc05f0f44b76-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.778 DEBUG:boto:b'hbmuog-62obj111000falseobj112023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.779 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.780 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.780 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.781 DEBUG:boto:Path: /hbmuog-62/obj11 2023-10-17T19:12:35.781 DEBUG:boto:Data: 2023-10-17T19:12:35.782 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.782 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.783 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.783 DEBUG:boto:Params: {} 2023-10-17T19:12:35.784 DEBUG:boto:Token: None 2023-10-17T19:12:35.784 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11 2023-10-17T19:12:35.784 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h5NwbkKezkEnLPfkR10S/1ph5pA= 2023-10-17T19:12:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h5NwbkKezkEnLPfkR10S/1ph5pA=', 'Content-Length': '0'} 2023-10-17T19:12:35.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e732b621c97e74-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.788 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.788 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.788 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.789 DEBUG:boto:Path: /hbmuog-62/obj11 2023-10-17T19:12:35.789 DEBUG:boto:Data: 2023-10-17T19:12:35.790 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.790 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.791 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.791 DEBUG:boto:Params: {} 2023-10-17T19:12:35.792 DEBUG:boto:Token: None 2023-10-17T19:12:35.792 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11 2023-10-17T19:12:35.792 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h5NwbkKezkEnLPfkR10S/1ph5pA= 2023-10-17T19:12:35.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h5NwbkKezkEnLPfkR10S/1ph5pA=', 'Content-Length': '0'} 2023-10-17T19:12:35.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c753a4a96cfe5b-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:12:35.797 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.798 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.799 DEBUG:boto:Method: GET 2023-10-17T19:12:35.800 DEBUG:boto:Path: /hbmuog-62/obj11 2023-10-17T19:12:35.800 DEBUG:boto:Data: 2023-10-17T19:12:35.801 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.802 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.802 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.803 DEBUG:boto:Params: {} 2023-10-17T19:12:35.803 DEBUG:boto:Token: None 2023-10-17T19:12:35.804 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11 2023-10-17T19:12:35.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WPPOdzgBHllYVSKhci/rsaJBEyM= 2023-10-17T19:12:35.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WPPOdzgBHllYVSKhci/rsaJBEyM=', 'Content-Length': '0'} 2023-10-17T19:12:35.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f8944b05bde4d2-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.850 DEBUG:boto:path=/hbmuog-62/obj11 2023-10-17T19:12:35.851 DEBUG:boto:auth_path=/hbmuog-62/obj11 2023-10-17T19:12:35.852 DEBUG:boto:Method: GET 2023-10-17T19:12:35.852 DEBUG:boto:Path: /hbmuog-62/obj11 2023-10-17T19:12:35.852 DEBUG:boto:Data: 2023-10-17T19:12:35.853 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.853 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.854 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.854 DEBUG:boto:Params: {} 2023-10-17T19:12:35.855 DEBUG:boto:Token: None 2023-10-17T19:12:35.855 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj11 2023-10-17T19:12:35.856 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:WPPOdzgBHllYVSKhci/rsaJBEyM= 2023-10-17T19:12:35.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WPPOdzgBHllYVSKhci/rsaJBEyM=', 'Content-Length': '0'} 2023-10-17T19:12:35.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae50c9103a400e9d-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.859 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.860 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.860 DEBUG:boto:path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.860 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.861 DEBUG:boto:Method: GET 2023-10-17T19:12:35.861 DEBUG:boto:Path: /hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.862 DEBUG:boto:Data: 2023-10-17T19:12:35.862 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.863 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.863 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.863 DEBUG:boto:Params: {} 2023-10-17T19:12:35.864 DEBUG:boto:Token: None 2023-10-17T19:12:35.864 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.865 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc34b7cce14edd17-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.868 DEBUG:boto:b'hbmuog-62obj111000falseobj112023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.869 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.869 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.870 DEBUG:boto:path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.870 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.871 DEBUG:boto:Method: GET 2023-10-17T19:12:35.871 DEBUG:boto:Path: /hbmuog-62/?prefix=obj11 2023-10-17T19:12:35.872 DEBUG:boto:Data: 2023-10-17T19:12:35.872 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.872 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.873 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.873 DEBUG:boto:Params: {} 2023-10-17T19:12:35.874 DEBUG:boto:Token: None 2023-10-17T19:12:35.874 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.875 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f14e0e88b5669a-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.878 DEBUG:boto:b'hbmuog-62obj111000falseobj112023-10-17T19:10:38.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:12:35.879 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.880 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.880 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.881 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.881 DEBUG:boto:Method: GET 2023-10-17T19:12:35.881 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.882 DEBUG:boto:Data: 2023-10-17T19:12:35.882 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.883 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.883 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.883 DEBUG:boto:Params: {} 2023-10-17T19:12:35.884 DEBUG:boto:Token: None 2023-10-17T19:12:35.884 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.885 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U= 2023-10-17T19:12:35.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U=', 'Content-Length': '0'} 2023-10-17T19:12:35.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ad1736b4ea35dc-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.888 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.888 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.889 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.889 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.890 DEBUG:boto:Method: GET 2023-10-17T19:12:35.890 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.890 DEBUG:boto:Data: 2023-10-17T19:12:35.891 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.891 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.892 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.892 DEBUG:boto:Params: {} 2023-10-17T19:12:35.892 DEBUG:boto:Token: None 2023-10-17T19:12:35.893 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.893 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U= 2023-10-17T19:12:35.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U=', 'Content-Length': '0'} 2023-10-17T19:12:35.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010dba18543255aa-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.898 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.898 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.899 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.899 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.899 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.900 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.900 DEBUG:boto:Data: 2023-10-17T19:12:35.901 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.901 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.902 DEBUG:boto:Params: {} 2023-10-17T19:12:35.902 DEBUG:boto:Token: None 2023-10-17T19:12:35.903 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.904 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D+1RtpXqMOTIMcdAihe+pbnsvXo= 2023-10-17T19:12:35.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D+1RtpXqMOTIMcdAihe+pbnsvXo=', 'Content-Length': '0'} 2023-10-17T19:12:35.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6fa9b088203189-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.907 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.907 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.907 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.908 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.908 DEBUG:boto:Method: HEAD 2023-10-17T19:12:35.909 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.909 DEBUG:boto:Data: 2023-10-17T19:12:35.910 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.910 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.911 DEBUG:boto:Params: {} 2023-10-17T19:12:35.911 DEBUG:boto:Token: None 2023-10-17T19:12:35.912 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:D+1RtpXqMOTIMcdAihe+pbnsvXo= 2023-10-17T19:12:35.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:D+1RtpXqMOTIMcdAihe+pbnsvXo=', 'Content-Length': '0'} 2023-10-17T19:12:35.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928eaf1be0dc53f7-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:12:35.916 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.916 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.916 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.917 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.917 DEBUG:boto:Method: GET 2023-10-17T19:12:35.918 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.918 DEBUG:boto:Data: 2023-10-17T19:12:35.919 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.919 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.920 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.920 DEBUG:boto:Params: {} 2023-10-17T19:12:35.920 DEBUG:boto:Token: None 2023-10-17T19:12:35.921 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.921 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U= 2023-10-17T19:12:35.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U=', 'Content-Length': '0'} 2023-10-17T19:12:35.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a3ce959213f87a-00652edca3-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.966 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:35.967 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:35.968 DEBUG:boto:path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.968 DEBUG:boto:auth_path=/hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.968 DEBUG:boto:Method: GET 2023-10-17T19:12:35.969 DEBUG:boto:Path: /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.969 DEBUG:boto:Data: 2023-10-17T19:12:35.970 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.970 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.971 DEBUG:boto:Params: {} 2023-10-17T19:12:35.971 DEBUG:boto:Token: None 2023-10-17T19:12:35.972 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/obj22?versionId=null 2023-10-17T19:12:35.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U= 2023-10-17T19:12:35.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X1w5RKfVipC00C8a9H7KgU4jj/U=', 'Content-Length': '0'} 2023-10-17T19:12:35.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000107a3e26d1243ed3-00652edca3-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.976 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.976 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.977 DEBUG:boto:path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.977 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.977 DEBUG:boto:Method: GET 2023-10-17T19:12:35.978 DEBUG:boto:Path: /hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.978 DEBUG:boto:Data: 2023-10-17T19:12:35.979 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.979 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:35.980 DEBUG:boto:Port: 8001 2023-10-17T19:12:35.980 DEBUG:boto:Params: {} 2023-10-17T19:12:35.981 DEBUG:boto:Token: None 2023-10-17T19:12:35.981 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.982 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63c988c4f97184b-00652edca3-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:35.985 DEBUG:boto:b'hbmuog-62obj221000falseobj222023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:35.986 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:35.986 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:35.987 DEBUG:boto:path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.987 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.988 DEBUG:boto:Method: GET 2023-10-17T19:12:35.988 DEBUG:boto:Path: /hbmuog-62/?prefix=obj22 2023-10-17T19:12:35.989 DEBUG:boto:Data: 2023-10-17T19:12:35.989 DEBUG:boto:Headers: {} 2023-10-17T19:12:35.989 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:35.990 DEBUG:boto:Port: 8000 2023-10-17T19:12:35.990 DEBUG:boto:Params: {} 2023-10-17T19:12:35.991 DEBUG:boto:Token: None 2023-10-17T19:12:35.991 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:35 GMT /hbmuog-62/ 2023-10-17T19:12:35.992 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg= 2023-10-17T19:12:35.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uu0sWK87bUrxgrN1Ol0S5ksElYg=', 'Content-Length': '0'} 2023-10-17T19:12:35.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019cc87f29eb1015b-00652edca3-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.042 DEBUG:boto:b'hbmuog-62obj221000falseobj222023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.043 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:36.044 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:36.044 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.044 DEBUG:boto:Path: /hbmuog-62/obj22 2023-10-17T19:12:36.045 DEBUG:boto:Data: 2023-10-17T19:12:36.045 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.046 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.047 DEBUG:boto:Params: {} 2023-10-17T19:12:36.047 DEBUG:boto:Token: None 2023-10-17T19:12:36.047 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj22 2023-10-17T19:12:36.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eMG7m6qKeGm5Xkt/qfqPjJZH9Yc= 2023-10-17T19:12:36.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eMG7m6qKeGm5Xkt/qfqPjJZH9Yc=', 'Content-Length': '0'} 2023-10-17T19:12:36.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfc37bf51717803-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.051 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:36.051 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:36.052 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.052 DEBUG:boto:Path: /hbmuog-62/obj22 2023-10-17T19:12:36.053 DEBUG:boto:Data: 2023-10-17T19:12:36.053 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.054 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.054 DEBUG:boto:Params: {} 2023-10-17T19:12:36.054 DEBUG:boto:Token: None 2023-10-17T19:12:36.055 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj22 2023-10-17T19:12:36.055 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eMG7m6qKeGm5Xkt/qfqPjJZH9Yc= 2023-10-17T19:12:36.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eMG7m6qKeGm5Xkt/qfqPjJZH9Yc=', 'Content-Length': '0'} 2023-10-17T19:12:36.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc7370b20c33b42-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:12:36.059 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:36.059 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:36.060 DEBUG:boto:Method: GET 2023-10-17T19:12:36.060 DEBUG:boto:Path: /hbmuog-62/obj22 2023-10-17T19:12:36.061 DEBUG:boto:Data: 2023-10-17T19:12:36.061 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.062 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.062 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.062 DEBUG:boto:Params: {} 2023-10-17T19:12:36.063 DEBUG:boto:Token: None 2023-10-17T19:12:36.063 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj22 2023-10-17T19:12:36.064 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ILiqxN3t2V6VlAloAhwF8S6LmhQ= 2023-10-17T19:12:36.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ILiqxN3t2V6VlAloAhwF8S6LmhQ=', 'Content-Length': '0'} 2023-10-17T19:12:36.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0b3dd71be7f4de-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.067 DEBUG:boto:path=/hbmuog-62/obj22 2023-10-17T19:12:36.068 DEBUG:boto:auth_path=/hbmuog-62/obj22 2023-10-17T19:12:36.068 DEBUG:boto:Method: GET 2023-10-17T19:12:36.069 DEBUG:boto:Path: /hbmuog-62/obj22 2023-10-17T19:12:36.069 DEBUG:boto:Data: 2023-10-17T19:12:36.070 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.070 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.071 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.071 DEBUG:boto:Params: {} 2023-10-17T19:12:36.071 DEBUG:boto:Token: None 2023-10-17T19:12:36.072 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj22 2023-10-17T19:12:36.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ILiqxN3t2V6VlAloAhwF8S6LmhQ= 2023-10-17T19:12:36.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ILiqxN3t2V6VlAloAhwF8S6LmhQ=', 'Content-Length': '0'} 2023-10-17T19:12:36.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3051504b961c78-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.076 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.076 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.077 DEBUG:boto:path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.077 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.078 DEBUG:boto:Method: GET 2023-10-17T19:12:36.078 DEBUG:boto:Path: /hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.079 DEBUG:boto:Data: 2023-10-17T19:12:36.079 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.080 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.080 DEBUG:boto:Params: {} 2023-10-17T19:12:36.081 DEBUG:boto:Token: None 2023-10-17T19:12:36.081 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.082 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007545944900fff8a2-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.130 DEBUG:boto:b'hbmuog-62obj221000falseobj222023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.131 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.132 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.132 DEBUG:boto:path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.133 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.133 DEBUG:boto:Method: GET 2023-10-17T19:12:36.133 DEBUG:boto:Path: /hbmuog-62/?prefix=obj22 2023-10-17T19:12:36.134 DEBUG:boto:Data: 2023-10-17T19:12:36.134 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.135 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.136 DEBUG:boto:Params: {} 2023-10-17T19:12:36.136 DEBUG:boto:Token: None 2023-10-17T19:12:36.137 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d29ed9ba2b58864b-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.141 DEBUG:boto:b'hbmuog-62obj221000falseobj222023-10-17T19:10:38.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:12:36.142 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.142 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.143 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.143 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.143 DEBUG:boto:Method: GET 2023-10-17T19:12:36.144 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.144 DEBUG:boto:Data: 2023-10-17T19:12:36.145 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.145 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.145 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.146 DEBUG:boto:Params: {} 2023-10-17T19:12:36.146 DEBUG:boto:Token: None 2023-10-17T19:12:36.147 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.147 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY= 2023-10-17T19:12:36.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY=', 'Content-Length': '0'} 2023-10-17T19:12:36.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183a28edc3c1f94c-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.151 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.151 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.152 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.152 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.153 DEBUG:boto:Method: GET 2023-10-17T19:12:36.153 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.153 DEBUG:boto:Data: 2023-10-17T19:12:36.154 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.154 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.155 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.155 DEBUG:boto:Params: {} 2023-10-17T19:12:36.156 DEBUG:boto:Token: None 2023-10-17T19:12:36.156 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY= 2023-10-17T19:12:36.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY=', 'Content-Length': '0'} 2023-10-17T19:12:36.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a0ed962a189e297-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.160 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.160 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.161 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.161 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.161 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.162 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.162 DEBUG:boto:Data: 2023-10-17T19:12:36.163 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.163 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.164 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.164 DEBUG:boto:Params: {} 2023-10-17T19:12:36.164 DEBUG:boto:Token: None 2023-10-17T19:12:36.165 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.165 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tl/wkt641RrUTlrdr3w5w/5WyWY= 2023-10-17T19:12:36.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tl/wkt641RrUTlrdr3w5w/5WyWY=', 'Content-Length': '0'} 2023-10-17T19:12:36.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000962e49c619240ad2-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.168 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.169 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.169 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.170 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.170 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.170 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.171 DEBUG:boto:Data: 2023-10-17T19:12:36.171 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.172 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.172 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.172 DEBUG:boto:Params: {} 2023-10-17T19:12:36.173 DEBUG:boto:Token: None 2023-10-17T19:12:36.173 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.174 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tl/wkt641RrUTlrdr3w5w/5WyWY= 2023-10-17T19:12:36.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tl/wkt641RrUTlrdr3w5w/5WyWY=', 'Content-Length': '0'} 2023-10-17T19:12:36.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3c6d90958c514f-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:12:36.177 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.178 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.178 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.179 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.179 DEBUG:boto:Method: GET 2023-10-17T19:12:36.180 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.180 DEBUG:boto:Data: 2023-10-17T19:12:36.180 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.181 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.181 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.182 DEBUG:boto:Params: {} 2023-10-17T19:12:36.182 DEBUG:boto:Token: None 2023-10-17T19:12:36.183 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.183 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY= 2023-10-17T19:12:36.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY=', 'Content-Length': '0'} 2023-10-17T19:12:36.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003558d8936282cf27-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.187 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.187 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.188 DEBUG:boto:path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.188 DEBUG:boto:auth_path=/hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.189 DEBUG:boto:Method: GET 2023-10-17T19:12:36.189 DEBUG:boto:Path: /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.189 DEBUG:boto:Data: 2023-10-17T19:12:36.190 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.190 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.191 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.191 DEBUG:boto:Params: {} 2023-10-17T19:12:36.192 DEBUG:boto:Token: None 2023-10-17T19:12:36.192 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33?versionId=null 2023-10-17T19:12:36.193 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY= 2023-10-17T19:12:36.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ow/jBGieaRWtON+t/SQJczj+taY=', 'Content-Length': '0'} 2023-10-17T19:12:36.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdce8a0a8e5b5ea8-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.196 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.196 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.197 DEBUG:boto:path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.197 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.197 DEBUG:boto:Method: GET 2023-10-17T19:12:36.198 DEBUG:boto:Path: /hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.198 DEBUG:boto:Data: 2023-10-17T19:12:36.199 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.200 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.200 DEBUG:boto:Params: {} 2023-10-17T19:12:36.200 DEBUG:boto:Token: None 2023-10-17T19:12:36.201 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287a83b9e4d2ee95-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.250 DEBUG:boto:b'hbmuog-62obj331000falseobj332023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.251 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.252 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.252 DEBUG:boto:path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.253 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.253 DEBUG:boto:Method: GET 2023-10-17T19:12:36.254 DEBUG:boto:Path: /hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.254 DEBUG:boto:Data: 2023-10-17T19:12:36.254 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.255 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.256 DEBUG:boto:Params: {} 2023-10-17T19:12:36.256 DEBUG:boto:Token: None 2023-10-17T19:12:36.257 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a0aa40b7934148-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.262 DEBUG:boto:b'hbmuog-62obj331000falseobj332023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.262 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.263 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.263 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.264 DEBUG:boto:Path: /hbmuog-62/obj33 2023-10-17T19:12:36.264 DEBUG:boto:Data: 2023-10-17T19:12:36.264 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.265 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.265 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.266 DEBUG:boto:Params: {} 2023-10-17T19:12:36.266 DEBUG:boto:Token: None 2023-10-17T19:12:36.267 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33 2023-10-17T19:12:36.267 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7l9cPKFbZTRPEe7m+wY44zz/F7o= 2023-10-17T19:12:36.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7l9cPKFbZTRPEe7m+wY44zz/F7o=', 'Content-Length': '0'} 2023-10-17T19:12:36.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009325a1eccabf77fc-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.270 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.271 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.271 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.271 DEBUG:boto:Path: /hbmuog-62/obj33 2023-10-17T19:12:36.272 DEBUG:boto:Data: 2023-10-17T19:12:36.272 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.273 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.274 DEBUG:boto:Params: {} 2023-10-17T19:12:36.274 DEBUG:boto:Token: None 2023-10-17T19:12:36.275 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33 2023-10-17T19:12:36.275 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7l9cPKFbZTRPEe7m+wY44zz/F7o= 2023-10-17T19:12:36.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7l9cPKFbZTRPEe7m+wY44zz/F7o=', 'Content-Length': '0'} 2023-10-17T19:12:36.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0e7f3ca54dc29c-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:12:36.278 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.279 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.279 DEBUG:boto:Method: GET 2023-10-17T19:12:36.280 DEBUG:boto:Path: /hbmuog-62/obj33 2023-10-17T19:12:36.280 DEBUG:boto:Data: 2023-10-17T19:12:36.281 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.281 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.281 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.282 DEBUG:boto:Params: {} 2023-10-17T19:12:36.282 DEBUG:boto:Token: None 2023-10-17T19:12:36.283 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33 2023-10-17T19:12:36.283 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4n3jbyUc1pQjTGD3nBzUCGRY3c= 2023-10-17T19:12:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4n3jbyUc1pQjTGD3nBzUCGRY3c=', 'Content-Length': '0'} 2023-10-17T19:12:36.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edab21aec5b7cddc-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.287 DEBUG:boto:path=/hbmuog-62/obj33 2023-10-17T19:12:36.287 DEBUG:boto:auth_path=/hbmuog-62/obj33 2023-10-17T19:12:36.288 DEBUG:boto:Method: GET 2023-10-17T19:12:36.288 DEBUG:boto:Path: /hbmuog-62/obj33 2023-10-17T19:12:36.289 DEBUG:boto:Data: 2023-10-17T19:12:36.289 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.289 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.290 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.290 DEBUG:boto:Params: {} 2023-10-17T19:12:36.291 DEBUG:boto:Token: None 2023-10-17T19:12:36.291 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj33 2023-10-17T19:12:36.292 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:j4n3jbyUc1pQjTGD3nBzUCGRY3c= 2023-10-17T19:12:36.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4n3jbyUc1pQjTGD3nBzUCGRY3c=', 'Content-Length': '0'} 2023-10-17T19:12:36.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702b0e0674a11cad-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.295 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.295 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.296 DEBUG:boto:path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.296 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.297 DEBUG:boto:Method: GET 2023-10-17T19:12:36.297 DEBUG:boto:Path: /hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.297 DEBUG:boto:Data: 2023-10-17T19:12:36.298 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.298 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.299 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.299 DEBUG:boto:Params: {} 2023-10-17T19:12:36.300 DEBUG:boto:Token: None 2023-10-17T19:12:36.300 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.301 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004839ef7d064d81d0-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.306 DEBUG:boto:b'hbmuog-62obj331000falseobj332023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.306 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.307 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.307 DEBUG:boto:path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.308 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.308 DEBUG:boto:Method: GET 2023-10-17T19:12:36.309 DEBUG:boto:Path: /hbmuog-62/?prefix=obj33 2023-10-17T19:12:36.309 DEBUG:boto:Data: 2023-10-17T19:12:36.309 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.310 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.311 DEBUG:boto:Params: {} 2023-10-17T19:12:36.311 DEBUG:boto:Token: None 2023-10-17T19:12:36.312 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.312 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63df6a8c3254482-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.358 DEBUG:boto:b'hbmuog-62obj331000falseobj332023-10-17T19:10:38.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:12:36.360 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.360 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.361 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.361 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.361 DEBUG:boto:Method: GET 2023-10-17T19:12:36.362 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.362 DEBUG:boto:Data: 2023-10-17T19:12:36.363 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.363 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.363 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.364 DEBUG:boto:Params: {} 2023-10-17T19:12:36.364 DEBUG:boto:Token: None 2023-10-17T19:12:36.365 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s= 2023-10-17T19:12:36.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s=', 'Content-Length': '0'} 2023-10-17T19:12:36.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829574ea3a09db20-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.368 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.369 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.369 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.370 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.370 DEBUG:boto:Method: GET 2023-10-17T19:12:36.371 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.371 DEBUG:boto:Data: 2023-10-17T19:12:36.372 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.372 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.373 DEBUG:boto:Params: {} 2023-10-17T19:12:36.373 DEBUG:boto:Token: None 2023-10-17T19:12:36.374 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s= 2023-10-17T19:12:36.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s=', 'Content-Length': '0'} 2023-10-17T19:12:36.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1cf2c57a9b062a-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.378 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.378 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.379 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.379 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.379 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.380 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.380 DEBUG:boto:Data: 2023-10-17T19:12:36.381 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.381 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.381 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.382 DEBUG:boto:Params: {} 2023-10-17T19:12:36.382 DEBUG:boto:Token: None 2023-10-17T19:12:36.383 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EMhu5SHF4wR+ygpavnoAHcD5+pY= 2023-10-17T19:12:36.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EMhu5SHF4wR+ygpavnoAHcD5+pY=', 'Content-Length': '0'} 2023-10-17T19:12:36.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37ef4a959e94165-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.386 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.387 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.387 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.388 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.388 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.389 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.389 DEBUG:boto:Data: 2023-10-17T19:12:36.389 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.390 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.390 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.391 DEBUG:boto:Params: {} 2023-10-17T19:12:36.391 DEBUG:boto:Token: None 2023-10-17T19:12:36.392 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.392 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EMhu5SHF4wR+ygpavnoAHcD5+pY= 2023-10-17T19:12:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EMhu5SHF4wR+ygpavnoAHcD5+pY=', 'Content-Length': '0'} 2023-10-17T19:12:36.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011674028ddadda5a-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:12:36.396 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.396 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.396 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.397 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.397 DEBUG:boto:Method: GET 2023-10-17T19:12:36.398 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.398 DEBUG:boto:Data: 2023-10-17T19:12:36.399 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.400 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.400 DEBUG:boto:Params: {} 2023-10-17T19:12:36.401 DEBUG:boto:Token: None 2023-10-17T19:12:36.401 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s= 2023-10-17T19:12:36.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s=', 'Content-Length': '0'} 2023-10-17T19:12:36.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa1c2a73961ce23-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.405 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.405 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.406 DEBUG:boto:path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.406 DEBUG:boto:auth_path=/hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.407 DEBUG:boto:Method: GET 2023-10-17T19:12:36.407 DEBUG:boto:Path: /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.407 DEBUG:boto:Data: 2023-10-17T19:12:36.408 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.408 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.409 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.409 DEBUG:boto:Params: {} 2023-10-17T19:12:36.410 DEBUG:boto:Token: None 2023-10-17T19:12:36.410 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44?versionId=null 2023-10-17T19:12:36.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s= 2023-10-17T19:12:36.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MhH/88bb+cIVlNfzG+mu385Dk6s=', 'Content-Length': '0'} 2023-10-17T19:12:36.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87b68b25a431a0d-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.414 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.414 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.415 DEBUG:boto:path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.415 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.415 DEBUG:boto:Method: GET 2023-10-17T19:12:36.416 DEBUG:boto:Path: /hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.416 DEBUG:boto:Data: 2023-10-17T19:12:36.417 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.418 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.418 DEBUG:boto:Params: {} 2023-10-17T19:12:36.419 DEBUG:boto:Token: None 2023-10-17T19:12:36.419 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.420 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4262bb61b9e1c8-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.423 DEBUG:boto:b'hbmuog-62obj441000falseobj442023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.424 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.424 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.425 DEBUG:boto:path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.425 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.426 DEBUG:boto:Method: GET 2023-10-17T19:12:36.426 DEBUG:boto:Path: /hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.426 DEBUG:boto:Data: 2023-10-17T19:12:36.427 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.428 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.428 DEBUG:boto:Params: {} 2023-10-17T19:12:36.429 DEBUG:boto:Token: None 2023-10-17T19:12:36.429 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.429 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd10394494065a2-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.433 DEBUG:boto:b'hbmuog-62obj441000falseobj442023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.434 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.434 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.435 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.435 DEBUG:boto:Path: /hbmuog-62/obj44 2023-10-17T19:12:36.435 DEBUG:boto:Data: 2023-10-17T19:12:36.436 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.437 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.437 DEBUG:boto:Params: {} 2023-10-17T19:12:36.438 DEBUG:boto:Token: None 2023-10-17T19:12:36.438 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44 2023-10-17T19:12:36.439 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HSGzNcZyMxqXH1+JIJ+tVu5w8ZM= 2023-10-17T19:12:36.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HSGzNcZyMxqXH1+JIJ+tVu5w8ZM=', 'Content-Length': '0'} 2023-10-17T19:12:36.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb89bb64d0b7f9d0-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.442 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.442 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.443 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.443 DEBUG:boto:Path: /hbmuog-62/obj44 2023-10-17T19:12:36.444 DEBUG:boto:Data: 2023-10-17T19:12:36.444 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.444 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.445 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.445 DEBUG:boto:Params: {} 2023-10-17T19:12:36.446 DEBUG:boto:Token: None 2023-10-17T19:12:36.446 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44 2023-10-17T19:12:36.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HSGzNcZyMxqXH1+JIJ+tVu5w8ZM= 2023-10-17T19:12:36.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HSGzNcZyMxqXH1+JIJ+tVu5w8ZM=', 'Content-Length': '0'} 2023-10-17T19:12:36.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233aaa27b4080d59-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:12:36.450 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.451 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.451 DEBUG:boto:Method: GET 2023-10-17T19:12:36.451 DEBUG:boto:Path: /hbmuog-62/obj44 2023-10-17T19:12:36.452 DEBUG:boto:Data: 2023-10-17T19:12:36.452 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.453 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.454 DEBUG:boto:Params: {} 2023-10-17T19:12:36.454 DEBUG:boto:Token: None 2023-10-17T19:12:36.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44 2023-10-17T19:12:36.455 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbAeOVJl6TwtUeRaPJATiMy+0qs= 2023-10-17T19:12:36.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbAeOVJl6TwtUeRaPJATiMy+0qs=', 'Content-Length': '0'} 2023-10-17T19:12:36.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c621390645dfea0-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.498 DEBUG:boto:path=/hbmuog-62/obj44 2023-10-17T19:12:36.499 DEBUG:boto:auth_path=/hbmuog-62/obj44 2023-10-17T19:12:36.499 DEBUG:boto:Method: GET 2023-10-17T19:12:36.500 DEBUG:boto:Path: /hbmuog-62/obj44 2023-10-17T19:12:36.500 DEBUG:boto:Data: 2023-10-17T19:12:36.501 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.501 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.502 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.502 DEBUG:boto:Params: {} 2023-10-17T19:12:36.503 DEBUG:boto:Token: None 2023-10-17T19:12:36.503 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj44 2023-10-17T19:12:36.504 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jbAeOVJl6TwtUeRaPJATiMy+0qs= 2023-10-17T19:12:36.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbAeOVJl6TwtUeRaPJATiMy+0qs=', 'Content-Length': '0'} 2023-10-17T19:12:36.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed99f3e0765c755-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.507 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.507 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.508 DEBUG:boto:path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.508 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.509 DEBUG:boto:Method: GET 2023-10-17T19:12:36.509 DEBUG:boto:Path: /hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.510 DEBUG:boto:Data: 2023-10-17T19:12:36.510 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.510 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.511 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.513 DEBUG:boto:Params: {} 2023-10-17T19:12:36.514 DEBUG:boto:Token: None 2023-10-17T19:12:36.514 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.515 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea43127009ee401-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.519 DEBUG:boto:b'hbmuog-62obj441000falseobj442023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.519 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.520 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.520 DEBUG:boto:path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.521 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.521 DEBUG:boto:Method: GET 2023-10-17T19:12:36.521 DEBUG:boto:Path: /hbmuog-62/?prefix=obj44 2023-10-17T19:12:36.522 DEBUG:boto:Data: 2023-10-17T19:12:36.522 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.523 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.523 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.523 DEBUG:boto:Params: {} 2023-10-17T19:12:36.524 DEBUG:boto:Token: None 2023-10-17T19:12:36.524 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.525 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f7aabf17f75ce7-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.574 DEBUG:boto:b'hbmuog-62obj441000falseobj442023-10-17T19:10:38.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:12:36.576 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.576 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.577 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.577 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.577 DEBUG:boto:Method: GET 2023-10-17T19:12:36.578 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.578 DEBUG:boto:Data: 2023-10-17T19:12:36.579 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.580 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.580 DEBUG:boto:Params: {} 2023-10-17T19:12:36.581 DEBUG:boto:Token: None 2023-10-17T19:12:36.581 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.582 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc= 2023-10-17T19:12:36.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc=', 'Content-Length': '0'} 2023-10-17T19:12:36.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093df1fe9ebd65adc-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.585 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.586 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.586 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.586 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.587 DEBUG:boto:Method: GET 2023-10-17T19:12:36.587 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.588 DEBUG:boto:Data: 2023-10-17T19:12:36.588 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.589 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.589 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.589 DEBUG:boto:Params: {} 2023-10-17T19:12:36.590 DEBUG:boto:Token: None 2023-10-17T19:12:36.590 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc= 2023-10-17T19:12:36.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc=', 'Content-Length': '0'} 2023-10-17T19:12:36.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca00023a654e29b-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.594 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.595 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.595 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.596 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.596 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.597 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.597 DEBUG:boto:Data: 2023-10-17T19:12:36.597 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.598 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.598 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.599 DEBUG:boto:Params: {} 2023-10-17T19:12:36.599 DEBUG:boto:Token: None 2023-10-17T19:12:36.600 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.600 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1+JfU0ITUwwzPtohBAkOoY660Ng= 2023-10-17T19:12:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1+JfU0ITUwwzPtohBAkOoY660Ng=', 'Content-Length': '0'} 2023-10-17T19:12:36.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa771fcd3bb7d44c-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.603 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.604 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.604 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.604 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.605 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.605 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.606 DEBUG:boto:Data: 2023-10-17T19:12:36.606 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.606 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.607 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.607 DEBUG:boto:Params: {} 2023-10-17T19:12:36.608 DEBUG:boto:Token: None 2023-10-17T19:12:36.608 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.609 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:1+JfU0ITUwwzPtohBAkOoY660Ng= 2023-10-17T19:12:36.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1+JfU0ITUwwzPtohBAkOoY660Ng=', 'Content-Length': '0'} 2023-10-17T19:12:36.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048682df7bda440e8-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:12:36.612 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.613 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.613 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.613 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.614 DEBUG:boto:Method: GET 2023-10-17T19:12:36.614 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.615 DEBUG:boto:Data: 2023-10-17T19:12:36.615 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.616 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.616 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.616 DEBUG:boto:Params: {} 2023-10-17T19:12:36.617 DEBUG:boto:Token: None 2023-10-17T19:12:36.617 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.618 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc= 2023-10-17T19:12:36.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc=', 'Content-Length': '0'} 2023-10-17T19:12:36.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c3a56426314c23-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.621 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.622 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.622 DEBUG:boto:path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.623 DEBUG:boto:auth_path=/hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.623 DEBUG:boto:Method: GET 2023-10-17T19:12:36.624 DEBUG:boto:Path: /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.624 DEBUG:boto:Data: 2023-10-17T19:12:36.624 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.625 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.626 DEBUG:boto:Params: {} 2023-10-17T19:12:36.626 DEBUG:boto:Token: None 2023-10-17T19:12:36.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55?versionId=null 2023-10-17T19:12:36.627 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc= 2023-10-17T19:12:36.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7elFpFgkpAp4kmGiGKTCz7LjNlc=', 'Content-Length': '0'} 2023-10-17T19:12:36.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83818dd94f3e663-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.631 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.631 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.632 DEBUG:boto:path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.632 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.633 DEBUG:boto:Method: GET 2023-10-17T19:12:36.633 DEBUG:boto:Path: /hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.633 DEBUG:boto:Data: 2023-10-17T19:12:36.634 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.635 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.635 DEBUG:boto:Params: {} 2023-10-17T19:12:36.636 DEBUG:boto:Token: None 2023-10-17T19:12:36.636 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a05391331d618f-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.640 DEBUG:boto:b'hbmuog-62obj551000falseobj552023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.641 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.642 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.642 DEBUG:boto:path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.642 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.643 DEBUG:boto:Method: GET 2023-10-17T19:12:36.643 DEBUG:boto:Path: /hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.644 DEBUG:boto:Data: 2023-10-17T19:12:36.644 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.644 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.645 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.645 DEBUG:boto:Params: {} 2023-10-17T19:12:36.646 DEBUG:boto:Token: None 2023-10-17T19:12:36.646 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d04be79d0d8e346-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.650 DEBUG:boto:b'hbmuog-62obj551000falseobj552023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.651 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.651 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.652 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.652 DEBUG:boto:Path: /hbmuog-62/obj55 2023-10-17T19:12:36.653 DEBUG:boto:Data: 2023-10-17T19:12:36.653 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.653 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.654 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.654 DEBUG:boto:Params: {} 2023-10-17T19:12:36.655 DEBUG:boto:Token: None 2023-10-17T19:12:36.655 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55 2023-10-17T19:12:36.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:de3TZOnZj108Vmtm/xUlkbhVvH4= 2023-10-17T19:12:36.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:de3TZOnZj108Vmtm/xUlkbhVvH4=', 'Content-Length': '0'} 2023-10-17T19:12:36.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010acfaff56719f1c-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.658 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.659 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.659 DEBUG:boto:Method: HEAD 2023-10-17T19:12:36.660 DEBUG:boto:Path: /hbmuog-62/obj55 2023-10-17T19:12:36.660 DEBUG:boto:Data: 2023-10-17T19:12:36.660 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.661 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.662 DEBUG:boto:Params: {} 2023-10-17T19:12:36.662 DEBUG:boto:Token: None 2023-10-17T19:12:36.662 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55 2023-10-17T19:12:36.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:de3TZOnZj108Vmtm/xUlkbhVvH4= 2023-10-17T19:12:36.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:de3TZOnZj108Vmtm/xUlkbhVvH4=', 'Content-Length': '0'} 2023-10-17T19:12:36.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef79a5b07929416-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:12:36.666 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.667 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.667 DEBUG:boto:Method: GET 2023-10-17T19:12:36.668 DEBUG:boto:Path: /hbmuog-62/obj55 2023-10-17T19:12:36.668 DEBUG:boto:Data: 2023-10-17T19:12:36.669 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.669 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.669 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.670 DEBUG:boto:Params: {} 2023-10-17T19:12:36.670 DEBUG:boto:Token: None 2023-10-17T19:12:36.671 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55 2023-10-17T19:12:36.671 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C37/UVLGeBn6QC3GAqZqKErGp6w= 2023-10-17T19:12:36.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C37/UVLGeBn6QC3GAqZqKErGp6w=', 'Content-Length': '0'} 2023-10-17T19:12:36.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-62:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000603a3bce1d715e7e-00652edca4-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.714 DEBUG:boto:path=/hbmuog-62/obj55 2023-10-17T19:12:36.715 DEBUG:boto:auth_path=/hbmuog-62/obj55 2023-10-17T19:12:36.715 DEBUG:boto:Method: GET 2023-10-17T19:12:36.716 DEBUG:boto:Path: /hbmuog-62/obj55 2023-10-17T19:12:36.716 DEBUG:boto:Data: 2023-10-17T19:12:36.717 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.717 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.718 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.718 DEBUG:boto:Params: {} 2023-10-17T19:12:36.719 DEBUG:boto:Token: None 2023-10-17T19:12:36.719 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/obj55 2023-10-17T19:12:36.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:C37/UVLGeBn6QC3GAqZqKErGp6w= 2023-10-17T19:12:36.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C37/UVLGeBn6QC3GAqZqKErGp6w=', 'Content-Length': '0'} 2023-10-17T19:12:36.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:10:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b4d5d0bdb0c8bc-00652edca4-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.723 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.723 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.724 DEBUG:boto:path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.724 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.725 DEBUG:boto:Method: GET 2023-10-17T19:12:36.725 DEBUG:boto:Path: /hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.726 DEBUG:boto:Data: 2023-10-17T19:12:36.726 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.727 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:12:36.727 DEBUG:boto:Port: 8001 2023-10-17T19:12:36.727 DEBUG:boto:Params: {} 2023-10-17T19:12:36.728 DEBUG:boto:Token: None 2023-10-17T19:12:36.728 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.729 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2e673c623f8746-00652edca4-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.732 DEBUG:boto:b'hbmuog-62obj551000falseobj552023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.733 DEBUG:boto:path=/hbmuog-62/ 2023-10-17T19:12:36.734 DEBUG:boto:auth_path=/hbmuog-62/ 2023-10-17T19:12:36.734 DEBUG:boto:path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.734 DEBUG:boto:auth_path=/hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.735 DEBUG:boto:Method: GET 2023-10-17T19:12:36.735 DEBUG:boto:Path: /hbmuog-62/?prefix=obj55 2023-10-17T19:12:36.736 DEBUG:boto:Data: 2023-10-17T19:12:36.736 DEBUG:boto:Headers: {} 2023-10-17T19:12:36.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:36.737 DEBUG:boto:Port: 8000 2023-10-17T19:12:36.738 DEBUG:boto:Params: {} 2023-10-17T19:12:36.738 DEBUG:boto:Token: None 2023-10-17T19:12:36.739 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:12:36 GMT /hbmuog-62/ 2023-10-17T19:12:36.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak= 2023-10-17T19:12:36.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oOQmt6FxxJJGUoWBbqT49ECf/ak=', 'Content-Length': '0'} 2023-10-17T19:12:36.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c90d84380b7e7b-00652edca4-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:36.782 DEBUG:boto:b'hbmuog-62obj551000falseobj552023-10-17T19:10:38.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:12:36.783 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-62 zones={a2, a1} 2023-10-17T19:12:36.784 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-17T19:12:36.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:36.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:36.785 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:36.844 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:36.844 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:36.898 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:12:36.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:36.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028759", 2023-10-17T19:12:36.899 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:10:55.286123Z" 2023-10-17T19:12:36.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:36.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027816", 2023-10-17T19:12:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:10:55.287830Z" 2023-10-17T19:12:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:36.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:12:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:12:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:36.901 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032715", 2023-10-17T19:12:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:10:38.815749Z" 2023-10-17T19:12:36.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:36.902 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:12:36.903 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:36.903 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:12:36.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:36.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:36.905 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:36.959 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:36.959 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:37.004 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:37.004 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:37.005 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:37.005 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:37.005 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 14823676944330065128 2023-10-17T19:12:37.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.006 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:37.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:37.006 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:37.006 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:37.006 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:37.007 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028759", 2023-10-17T19:12:37.007 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:37.007 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:37.007 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:37.008 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.286123Z" 2023-10-17T19:12:37.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:37.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:37.009 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:37.009 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:37.009 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:37.009 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027816", 2023-10-17T19:12:37.009 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:37.010 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:37.010 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:37.010 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.287830Z" 2023-10-17T19:12:37.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "full-sync", 2023-10-17T19:12:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:12:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:12:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:12:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:37.012 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:12:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:37.013 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032715", 2023-10-17T19:12:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:37.014 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 53, 2023-10-17T19:12:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:38.815749Z" 2023-10-17T19:12:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:37.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 62, 2023-10-17T19:12:37.016 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 53 2023-10-17T19:12:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:37.017 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:36Z" 2023-10-17T19:12:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:37.018 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14823676944330065128 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028759", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.286123Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027816", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.287830Z" } }, { "key": 2, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000019739", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:06:31.283420Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032715", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-17T19:10:38.815749Z" } } ] }, "full_sync": { "total": 62, "complete": 53 }, "current_time": "2023-10-17T19:12:36Z" } 2023-10-17T19:12:37.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028759', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.286123Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027816', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.287830Z'}}, {'key': 2, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000019739', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.283420Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032715', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-17T19:10:38.815749Z'}}] 2023-10-17T19:12:37.019 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:37.020 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:37.020 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000019739 target= 2023-10-17T19:12:42.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:42.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:42.022 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:42.133 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:42.134 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:42.134 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:42.134 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:42.135 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:42.135 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 14823676944330065128 2023-10-17T19:12:42.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.135 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:42.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:42.136 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:42.136 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:42.136 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:42.136 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028759", 2023-10-17T19:12:42.136 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:42.137 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:42.137 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:42.137 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.286123Z" 2023-10-17T19:12:42.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:42.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:42.138 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:42.138 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:42.138 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:42.139 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027816", 2023-10-17T19:12:42.139 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:42.139 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:42.139 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.287830Z" 2023-10-17T19:12:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:42.140 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "full-sync", 2023-10-17T19:12:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:12:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:12:42.141 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:12:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:12:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:42.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:42.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032715", 2023-10-17T19:12:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 53, 2023-10-17T19:12:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:38.815749Z" 2023-10-17T19:12:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 62, 2023-10-17T19:12:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 53 2023-10-17T19:12:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:42Z" 2023-10-17T19:12:42.146 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:42.147 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14823676944330065128 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028759", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.286123Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027816", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.287830Z" } }, { "key": 2, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000019739", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:06:31.283420Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032715", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-17T19:10:38.815749Z" } } ] }, "full_sync": { "total": 62, "complete": 53 }, "current_time": "2023-10-17T19:12:42Z" } 2023-10-17T19:12:42.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028759', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.286123Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027816', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.287830Z'}}, {'key': 2, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000019739', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.283420Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032715', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-17T19:10:38.815749Z'}}] 2023-10-17T19:12:42.148 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:42.149 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:42.149 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000019739 target= 2023-10-17T19:12:47.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:47.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:47.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:47.262 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:47.263 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:47.263 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:47.263 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:47.264 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:47.264 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 14823676944330065128 2023-10-17T19:12:47.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.264 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:47.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.265 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:47.265 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.265 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.265 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028759", 2023-10-17T19:12:47.266 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.266 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.266 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.266 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.286123Z" 2023-10-17T19:12:47.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.267 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:47.268 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.268 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.269 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000027816", 2023-10-17T19:12:47.269 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.270 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:55.287830Z" 2023-10-17T19:12:47.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.271 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:47.271 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.271 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.271 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000019739", 2023-10-17T19:12:47.272 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.272 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:12:47.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.272 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:06:31.283420Z" 2023-10-17T19:12:47.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.273 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:47.274 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.274 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.274 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000032715", 2023-10-17T19:12:47.274 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.275 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 53, 2023-10-17T19:12:47.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.275 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:10:38.815749Z" 2023-10-17T19:12:47.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.276 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:47.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.276 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:47.276 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 62, 2023-10-17T19:12:47.277 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 62 2023-10-17T19:12:47.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.277 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:47Z" 2023-10-17T19:12:47.277 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:47.279 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14823676944330065128 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028759", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.286123Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027816", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:10:55.287830Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:06:31.283420Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032715", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-17T19:10:38.815749Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-17T19:12:47Z" } 2023-10-17T19:12:47.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028759', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.286123Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027816', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:10:55.287830Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:06:31.283420Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032715', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-17T19:10:38.815749Z'}}] 2023-10-17T19:12:47.280 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:47.280 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028759', 1: '00000000000000000000:00000000000000027816', 2: '00000000000000000000:00000000000000019739', 3: '00000000000000000000:00000000000000032715'} 2023-10-17T19:12:47.280 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:12:47.281 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-17T19:12:47.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.282 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:47.384 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:12:47.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.385 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000010378", 2023-10-17T19:12:47.385 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:12:29.112956Z" 2023-10-17T19:12:47.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.386 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008847", 2023-10-17T19:12:47.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:12:29.115047Z" 2023-10-17T19:12:47.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.387 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004736", 2023-10-17T19:12:47.387 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:08:12.096952Z" 2023-10-17T19:12:47.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.387 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012909", 2023-10-17T19:12:47.388 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:12:13.370266Z" 2023-10-17T19:12:47.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.388 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:12:47.389 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000010378', 1: '00000000000000000000:00000000000000008847', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012909'} 2023-10-17T19:12:47.390 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:12:47.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.391 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:47.446 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:47.446 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:47.492 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:47.493 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:47.493 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:47.493 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:47.493 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:47.494 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 13326966719507974529 2023-10-17T19:12:47.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.494 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:47.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:47.495 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.495 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.495 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000010378", 2023-10-17T19:12:47.496 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.496 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.496 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.496 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:12:29.112956Z" 2023-10-17T19:12:47.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.497 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:47.497 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.497 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.498 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000008847", 2023-10-17T19:12:47.498 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.498 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.498 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.499 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:12:29.115047Z" 2023-10-17T19:12:47.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.499 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:47.500 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.500 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.500 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000004736", 2023-10-17T19:12:47.500 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.501 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.501 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.501 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:08:12.096952Z" 2023-10-17T19:12:47.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.502 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:47.502 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.503 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:12:47.503 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000012909", 2023-10-17T19:12:47.503 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.503 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.504 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:12:13.370266Z" 2023-10-17T19:12:47.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:47.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.505 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:47.505 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 0, 2023-10-17T19:12:47.505 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 0 2023-10-17T19:12:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:47Z" 2023-10-17T19:12:47.506 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:47.507 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13326966719507974529 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010378", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:12:29.112956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008847", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:12:29.115047Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004736", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:08:12.096952Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:12:13.370266Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-17T19:12:47Z" } 2023-10-17T19:12:47.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010378', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:12:29.112956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008847', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:12:29.115047Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004736', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:08:12.096952Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:12:13.370266Z'}}] 2023-10-17T19:12:47.508 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010378', 1: '00000000000000000000:00000000000000008847', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012909'} 2023-10-17T19:12:47.509 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010378', 1: '00000000000000000000:00000000000000008847', 2: '00000000000000000000:00000000000000004736', 3: '00000000000000000000:00000000000000012909'} 2023-10-17T19:12:47.509 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-17T19:12:47.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-10-17T19:12:47.511 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:47.512 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:47.512 DEBUG:boto:Method: PUT 2023-10-17T19:12:47.513 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:47.513 DEBUG:boto:Data: 2023-10-17T19:12:47.513 DEBUG:boto:Headers: {} 2023-10-17T19:12:47.514 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:47.514 DEBUG:boto:Port: 8000 2023-10-17T19:12:47.515 DEBUG:boto:Params: {} 2023-10-17T19:12:47.515 DEBUG:boto:Token: None 2023-10-17T19:12:47.516 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:12:47 GMT /hbmuog-63/ 2023-10-17T19:12:47.516 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EmOj19ly0lWQlGc//nunKKejrgk= 2023-10-17T19:12:47.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EmOj19ly0lWQlGc//nunKKejrgk=', 'Content-Length': '0'} 2023-10-17T19:12:47.531 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1413bf0fe8a5942-00652edcaf-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:47.532 DEBUG:rgw_multi.tests:created bucket=hbmuog-63 2023-10-17T19:12:47.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.534 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:47.586 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:47.586 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:12:47.629 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:12:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569967.530068_834.1", 2023-10-17T19:12:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:12:47.530068Z" 2023-10-17T19:12:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.630 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:12:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:12:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:12:47.631 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:12:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:12:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:12:47.632 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.633 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:12:47.634 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:12:47.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:12:47.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:47.635 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:47.687 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:47.688 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:47.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:47.716+0000 7feef5c09780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:12:47.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:47.717+0000 7feef5c09780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:12:47.731 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:47.732 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:47.732 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:47.732 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:47.732 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:47.733 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:12:47.733 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:47.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.733 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:47.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.734 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:47.734 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.734 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:47.734 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569967.523870_831.1", 2023-10-17T19:12:47.735 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.735 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:12:47.735 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.735 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:12:47.523870Z", 2023-10-17T19:12:47.736 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:47.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.737 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:47.737 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.737 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:47.737 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:12:47.737 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.738 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.738 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.738 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:12:47.739 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:47.739 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.740 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:47.740 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.740 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:47.741 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:12:47.741 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.741 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.741 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.742 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:12:47.742 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:47.742 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:47.743 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:47.743 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:47.743 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:47.744 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:12:47.744 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:47.745 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:47.745 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:47.745 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:12:47.746 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:47.746 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.746 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:47.746 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:47.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.747 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:47.747 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:12:47.747 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:12:47.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:47.748 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:47Z" 2023-10-17T19:12:47.748 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:47.749 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569967.523870_831.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:12:47.523870Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:12:47Z" } 2023-10-17T19:12:47.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569967.523870_831.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:12:47.523870Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:12:47.750 DEBUG:rgw_multi.tests:log_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:12:47.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569967.523870_831.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:12:47.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697569967.530068_834.1 target=1_1697569967.523870_831.1 2023-10-17T19:12:52.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:52.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:12:52.754 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:12:52.813 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:52.813 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:52.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:52.845+0000 7f3c9bf1d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:12:52.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:52.846+0000 7f3c9bf1d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:12:52.866 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:12:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:12:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:12:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:12:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:12:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:12:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:12:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:12:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569967.530068_834.1", 2023-10-17T19:12:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:12:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:12:47.530068Z", 2023-10-17T19:12:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:12:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:12:52.872 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:52.872 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:52.872 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:52.872 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:12:52.873 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:52.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:52.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:52.873 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:12:52.874 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:52.874 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:52.874 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:12:52.874 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:52.874 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:52.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:52.875 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:12:52.875 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:52.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:52.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:12:52.876 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:12:52.876 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:12:52.876 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:12:52.876 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:12:52.877 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:12:52.877 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:12:52.877 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:12:52.877 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:12:52.878 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:12:52.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:52.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:12:52.878 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:12:52.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.879 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:12:52.879 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:12:52.879 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:12:52.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:12:52.879 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:12:52Z" 2023-10-17T19:12:52.880 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:12:52.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697569967.530068_834.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:12:47.530068Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:12:52Z" } 2023-10-17T19:12:52.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569967.530068_834.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:12:47.530068Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:12:52.882 DEBUG:rgw_multi.tests:log_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:12:52.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:12:52.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:12:52.883 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:52.884 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:52.884 DEBUG:boto:Method: HEAD 2023-10-17T19:12:52.884 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:52.885 DEBUG:boto:Data: 2023-10-17T19:12:52.885 DEBUG:boto:Headers: {} 2023-10-17T19:12:52.886 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.886 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.886 DEBUG:boto:Params: {} 2023-10-17T19:12:52.887 DEBUG:boto:Token: None 2023-10-17T19:12:52.887 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/ 2023-10-17T19:12:52.888 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:12:52.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:12:52.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000022118850d1d6947-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.891 DEBUG:boto:path=/hbmuog-63/obj11 2023-10-17T19:12:52.892 DEBUG:boto:auth_path=/hbmuog-63/obj11 2023-10-17T19:12:52.892 DEBUG:boto:Method: PUT 2023-10-17T19:12:52.893 DEBUG:boto:Path: /hbmuog-63/obj11 2023-10-17T19:12:52.893 DEBUG:boto:Data: 2023-10-17T19:12:52.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:52.894 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.895 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.895 DEBUG:boto:Params: {} 2023-10-17T19:12:52.895 DEBUG:boto:Token: None 2023-10-17T19:12:52.896 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/obj11 2023-10-17T19:12:52.896 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:X4VOAnJ431mp0T1Fkq/newNtn5s= 2023-10-17T19:12:52.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X4VOAnJ431mp0T1Fkq/newNtn5s='} 2023-10-17T19:12:52.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcbdd1e9314cff61-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.904 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:52.904 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:52.904 DEBUG:boto:Method: HEAD 2023-10-17T19:12:52.905 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:52.905 DEBUG:boto:Data: 2023-10-17T19:12:52.906 DEBUG:boto:Headers: {} 2023-10-17T19:12:52.906 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.906 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.907 DEBUG:boto:Params: {} 2023-10-17T19:12:52.907 DEBUG:boto:Token: None 2023-10-17T19:12:52.908 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/ 2023-10-17T19:12:52.908 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:12:52.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:12:52.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba5348b87a8049c-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.911 DEBUG:boto:path=/hbmuog-63/obj22 2023-10-17T19:12:52.912 DEBUG:boto:auth_path=/hbmuog-63/obj22 2023-10-17T19:12:52.912 DEBUG:boto:Method: PUT 2023-10-17T19:12:52.913 DEBUG:boto:Path: /hbmuog-63/obj22 2023-10-17T19:12:52.913 DEBUG:boto:Data: 2023-10-17T19:12:52.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:52.914 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.915 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.915 DEBUG:boto:Params: {} 2023-10-17T19:12:52.916 DEBUG:boto:Token: None 2023-10-17T19:12:52.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/obj22 2023-10-17T19:12:52.917 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ePPKi79EB2AwDUpn+YY11XzJwGI= 2023-10-17T19:12:52.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ePPKi79EB2AwDUpn+YY11XzJwGI='} 2023-10-17T19:12:52.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7e7fafcbe807be7-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.967 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:52.968 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:52.968 DEBUG:boto:Method: HEAD 2023-10-17T19:12:52.969 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:52.969 DEBUG:boto:Data: 2023-10-17T19:12:52.970 DEBUG:boto:Headers: {} 2023-10-17T19:12:52.970 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.970 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.971 DEBUG:boto:Params: {} 2023-10-17T19:12:52.971 DEBUG:boto:Token: None 2023-10-17T19:12:52.972 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/ 2023-10-17T19:12:52.972 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:12:52.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:12:52.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007147f49790f7f6bb-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.976 DEBUG:boto:path=/hbmuog-63/obj33 2023-10-17T19:12:52.976 DEBUG:boto:auth_path=/hbmuog-63/obj33 2023-10-17T19:12:52.976 DEBUG:boto:Method: PUT 2023-10-17T19:12:52.977 DEBUG:boto:Path: /hbmuog-63/obj33 2023-10-17T19:12:52.977 DEBUG:boto:Data: 2023-10-17T19:12:52.978 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:52.978 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.978 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.979 DEBUG:boto:Params: {} 2023-10-17T19:12:52.979 DEBUG:boto:Token: None 2023-10-17T19:12:52.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/obj33 2023-10-17T19:12:52.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:h6TrHmSBPPtbpJ4Eq7gPAidS8K4= 2023-10-17T19:12:52.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h6TrHmSBPPtbpJ4Eq7gPAidS8K4='} 2023-10-17T19:12:52.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000addc6be10cee1190-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.986 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:52.987 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:52.987 DEBUG:boto:Method: HEAD 2023-10-17T19:12:52.988 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:52.988 DEBUG:boto:Data: 2023-10-17T19:12:52.989 DEBUG:boto:Headers: {} 2023-10-17T19:12:52.989 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.989 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.990 DEBUG:boto:Params: {} 2023-10-17T19:12:52.990 DEBUG:boto:Token: None 2023-10-17T19:12:52.991 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/ 2023-10-17T19:12:52.991 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:12:52.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:12:52.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdd20b207a6c3c80-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:52.995 DEBUG:boto:path=/hbmuog-63/obj44 2023-10-17T19:12:52.995 DEBUG:boto:auth_path=/hbmuog-63/obj44 2023-10-17T19:12:52.996 DEBUG:boto:Method: PUT 2023-10-17T19:12:52.996 DEBUG:boto:Path: /hbmuog-63/obj44 2023-10-17T19:12:52.996 DEBUG:boto:Data: 2023-10-17T19:12:52.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:52.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:52.998 DEBUG:boto:Port: 8000 2023-10-17T19:12:52.998 DEBUG:boto:Params: {} 2023-10-17T19:12:52.998 DEBUG:boto:Token: None 2023-10-17T19:12:52.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:52 GMT /hbmuog-63/obj44 2023-10-17T19:12:52.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wLwOSAa/zK1HZNabQ/G+9VNr5+A= 2023-10-17T19:12:53.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wLwOSAa/zK1HZNabQ/G+9VNr5+A='} 2023-10-17T19:12:53.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1c375656795a35e-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.043 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.044 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.044 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.044 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.045 DEBUG:boto:Data: 2023-10-17T19:12:53.045 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.046 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.046 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.047 DEBUG:boto:Params: {} 2023-10-17T19:12:53.047 DEBUG:boto:Token: None 2023-10-17T19:12:53.048 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.048 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c395d2d4343f8b9-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.054 DEBUG:boto:path=/hbmuog-63/obj55 2023-10-17T19:12:53.054 DEBUG:boto:auth_path=/hbmuog-63/obj55 2023-10-17T19:12:53.055 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.055 DEBUG:boto:Path: /hbmuog-63/obj55 2023-10-17T19:12:53.055 DEBUG:boto:Data: 2023-10-17T19:12:53.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.056 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.057 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.057 DEBUG:boto:Params: {} 2023-10-17T19:12:53.057 DEBUG:boto:Token: None 2023-10-17T19:12:53.058 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj55 2023-10-17T19:12:53.058 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jMa8as22mMr4/cW3a32h+gLUCYc= 2023-10-17T19:12:53.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jMa8as22mMr4/cW3a32h+gLUCYc='} 2023-10-17T19:12:53.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aed2f822fdf863a1-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.064 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:53.120 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:53.121 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:53.165 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:53.165 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:53.169 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 5 2023-10-17T19:12:53.205 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.206 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.206 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.207 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.207 DEBUG:boto:Data: 2023-10-17T19:12:53.208 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.208 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.209 DEBUG:boto:Params: {} 2023-10-17T19:12:53.209 DEBUG:boto:Token: None 2023-10-17T19:12:53.210 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d0c0f11f133e6d4-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.214 DEBUG:boto:path=/hbmuog-63/obj17 2023-10-17T19:12:53.215 DEBUG:boto:auth_path=/hbmuog-63/obj17 2023-10-17T19:12:53.215 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.216 DEBUG:boto:Path: /hbmuog-63/obj17 2023-10-17T19:12:53.216 DEBUG:boto:Data: 2023-10-17T19:12:53.217 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.217 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.217 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.218 DEBUG:boto:Params: {} 2023-10-17T19:12:53.218 DEBUG:boto:Token: None 2023-10-17T19:12:53.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj17 2023-10-17T19:12:53.219 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:xK3iM9cTLv2tLaL91hm2TNQFbJ8= 2023-10-17T19:12:53.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xK3iM9cTLv2tLaL91hm2TNQFbJ8='} 2023-10-17T19:12:53.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002470bed8ba409cc1-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.225 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.226 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.226 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.227 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.227 DEBUG:boto:Data: 2023-10-17T19:12:53.228 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.228 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.229 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.229 DEBUG:boto:Params: {} 2023-10-17T19:12:53.229 DEBUG:boto:Token: None 2023-10-17T19:12:53.230 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.230 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000466fbf998b5a3db7-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.234 DEBUG:boto:path=/hbmuog-63/obj34 2023-10-17T19:12:53.235 DEBUG:boto:auth_path=/hbmuog-63/obj34 2023-10-17T19:12:53.235 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.235 DEBUG:boto:Path: /hbmuog-63/obj34 2023-10-17T19:12:53.236 DEBUG:boto:Data: 2023-10-17T19:12:53.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.236 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.237 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.237 DEBUG:boto:Params: {} 2023-10-17T19:12:53.238 DEBUG:boto:Token: None 2023-10-17T19:12:53.238 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj34 2023-10-17T19:12:53.239 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rr2eEDfl7QI1BoFCgDYoOfV6Pms= 2023-10-17T19:12:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rr2eEDfl7QI1BoFCgDYoOfV6Pms='} 2023-10-17T19:12:53.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb2be91392f92750-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.245 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.245 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.246 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.246 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.246 DEBUG:boto:Data: 2023-10-17T19:12:53.247 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.247 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.248 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.248 DEBUG:boto:Params: {} 2023-10-17T19:12:53.248 DEBUG:boto:Token: None 2023-10-17T19:12:53.249 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.249 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c9677f742e2b304-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.253 DEBUG:boto:path=/hbmuog-63/obj51 2023-10-17T19:12:53.253 DEBUG:boto:auth_path=/hbmuog-63/obj51 2023-10-17T19:12:53.254 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.254 DEBUG:boto:Path: /hbmuog-63/obj51 2023-10-17T19:12:53.255 DEBUG:boto:Data: 2023-10-17T19:12:53.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.256 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.256 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.256 DEBUG:boto:Params: {} 2023-10-17T19:12:53.257 DEBUG:boto:Token: None 2023-10-17T19:12:53.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj51 2023-10-17T19:12:53.258 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Aw+cfBI7g+xEDBrD4OJhZLmLsc8= 2023-10-17T19:12:53.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Aw+cfBI7g+xEDBrD4OJhZLmLsc8='} 2023-10-17T19:12:53.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000237d5467646bc278-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.303 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.304 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.304 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.305 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.305 DEBUG:boto:Data: 2023-10-17T19:12:53.305 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.306 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.306 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.307 DEBUG:boto:Params: {} 2023-10-17T19:12:53.307 DEBUG:boto:Token: None 2023-10-17T19:12:53.308 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.308 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ded2a2b032291c6-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.312 DEBUG:boto:path=/hbmuog-63/obj68 2023-10-17T19:12:53.313 DEBUG:boto:auth_path=/hbmuog-63/obj68 2023-10-17T19:12:53.313 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.314 DEBUG:boto:Path: /hbmuog-63/obj68 2023-10-17T19:12:53.314 DEBUG:boto:Data: 2023-10-17T19:12:53.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.315 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.315 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.316 DEBUG:boto:Params: {} 2023-10-17T19:12:53.316 DEBUG:boto:Token: None 2023-10-17T19:12:53.317 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj68 2023-10-17T19:12:53.317 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Cn9JIMoF7cyoWDwgeUF0Vtd8oC0= 2023-10-17T19:12:53.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Cn9JIMoF7cyoWDwgeUF0Vtd8oC0='} 2023-10-17T19:12:53.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021297b4e501733db-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.324 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.324 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.324 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.325 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.325 DEBUG:boto:Data: 2023-10-17T19:12:53.326 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.329 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.330 DEBUG:boto:Params: {} 2023-10-17T19:12:53.330 DEBUG:boto:Token: None 2023-10-17T19:12:53.331 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.331 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c2a059245010964-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.335 DEBUG:boto:path=/hbmuog-63/obj85 2023-10-17T19:12:53.335 DEBUG:boto:auth_path=/hbmuog-63/obj85 2023-10-17T19:12:53.336 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.336 DEBUG:boto:Path: /hbmuog-63/obj85 2023-10-17T19:12:53.336 DEBUG:boto:Data: 2023-10-17T19:12:53.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.338 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.338 DEBUG:boto:Params: {} 2023-10-17T19:12:53.339 DEBUG:boto:Token: None 2023-10-17T19:12:53.339 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj85 2023-10-17T19:12:53.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cKsiXoZU3IosbyNhZtksGH+kO9w= 2023-10-17T19:12:53.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKsiXoZU3IosbyNhZtksGH+kO9w='} 2023-10-17T19:12:53.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a98be37e17dbf785-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.388 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:53.443 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:53.444 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:53.490 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:53.491 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:53.500 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 10 2023-10-17T19:12:53.549 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.549 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.550 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.550 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.551 DEBUG:boto:Data: 2023-10-17T19:12:53.551 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.552 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.552 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.552 DEBUG:boto:Params: {} 2023-10-17T19:12:53.553 DEBUG:boto:Token: None 2023-10-17T19:12:53.553 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.554 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc15875be281094d-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.558 DEBUG:boto:path=/hbmuog-63/obj19 2023-10-17T19:12:53.558 DEBUG:boto:auth_path=/hbmuog-63/obj19 2023-10-17T19:12:53.559 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.559 DEBUG:boto:Path: /hbmuog-63/obj19 2023-10-17T19:12:53.560 DEBUG:boto:Data: 2023-10-17T19:12:53.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.561 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.561 DEBUG:boto:Params: {} 2023-10-17T19:12:53.562 DEBUG:boto:Token: None 2023-10-17T19:12:53.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj19 2023-10-17T19:12:53.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:I4QBaBnCwazh5QxOO/42CMxNsT8= 2023-10-17T19:12:53.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I4QBaBnCwazh5QxOO/42CMxNsT8='} 2023-10-17T19:12:53.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a5471a0329804c-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.568 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.569 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.569 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.570 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.570 DEBUG:boto:Data: 2023-10-17T19:12:53.571 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.571 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.571 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.572 DEBUG:boto:Params: {} 2023-10-17T19:12:53.572 DEBUG:boto:Token: None 2023-10-17T19:12:53.573 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.573 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006e28914e9a1a7c8-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.577 DEBUG:boto:path=/hbmuog-63/obj38 2023-10-17T19:12:53.577 DEBUG:boto:auth_path=/hbmuog-63/obj38 2023-10-17T19:12:53.578 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.578 DEBUG:boto:Path: /hbmuog-63/obj38 2023-10-17T19:12:53.579 DEBUG:boto:Data: 2023-10-17T19:12:53.579 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.579 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.580 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.580 DEBUG:boto:Params: {} 2023-10-17T19:12:53.581 DEBUG:boto:Token: None 2023-10-17T19:12:53.581 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj38 2023-10-17T19:12:53.581 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2YHUNhqi4s4kysb3yUnWRn6Ie3Y= 2023-10-17T19:12:53.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2YHUNhqi4s4kysb3yUnWRn6Ie3Y='} 2023-10-17T19:12:53.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e99a823e97536351-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.586 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.586 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.587 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.587 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.588 DEBUG:boto:Data: 2023-10-17T19:12:53.588 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.589 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.589 DEBUG:boto:Params: {} 2023-10-17T19:12:53.590 DEBUG:boto:Token: None 2023-10-17T19:12:53.590 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3caf8e0d60ae03e-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.597 DEBUG:boto:path=/hbmuog-63/obj57 2023-10-17T19:12:53.597 DEBUG:boto:auth_path=/hbmuog-63/obj57 2023-10-17T19:12:53.598 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.598 DEBUG:boto:Path: /hbmuog-63/obj57 2023-10-17T19:12:53.599 DEBUG:boto:Data: 2023-10-17T19:12:53.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.599 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.600 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.600 DEBUG:boto:Params: {} 2023-10-17T19:12:53.601 DEBUG:boto:Token: None 2023-10-17T19:12:53.601 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj57 2023-10-17T19:12:53.602 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vtaNPWGFMFxh0nLfbLL0DEsc/a8= 2023-10-17T19:12:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vtaNPWGFMFxh0nLfbLL0DEsc/a8='} 2023-10-17T19:12:53.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5cf50f1045b6fee-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.609 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.610 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.610 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.611 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.611 DEBUG:boto:Data: 2023-10-17T19:12:53.612 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.612 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.613 DEBUG:boto:Params: {} 2023-10-17T19:12:53.613 DEBUG:boto:Token: None 2023-10-17T19:12:53.614 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67e15752570d161-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.618 DEBUG:boto:path=/hbmuog-63/obj76 2023-10-17T19:12:53.618 DEBUG:boto:auth_path=/hbmuog-63/obj76 2023-10-17T19:12:53.619 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.619 DEBUG:boto:Path: /hbmuog-63/obj76 2023-10-17T19:12:53.620 DEBUG:boto:Data: 2023-10-17T19:12:53.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.620 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.621 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.621 DEBUG:boto:Params: {} 2023-10-17T19:12:53.622 DEBUG:boto:Token: None 2023-10-17T19:12:53.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj76 2023-10-17T19:12:53.623 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GrD79vHtf01f/jSxSFjv8bqJIuQ= 2023-10-17T19:12:53.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GrD79vHtf01f/jSxSFjv8bqJIuQ='} 2023-10-17T19:12:53.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9edd056705e4f00-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.671 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.672 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.672 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.672 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.673 DEBUG:boto:Data: 2023-10-17T19:12:53.673 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.674 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.674 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.674 DEBUG:boto:Params: {} 2023-10-17T19:12:53.675 DEBUG:boto:Token: None 2023-10-17T19:12:53.675 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.676 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073d76fc93a7c2bd0-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.680 DEBUG:boto:path=/hbmuog-63/obj95 2023-10-17T19:12:53.680 DEBUG:boto:auth_path=/hbmuog-63/obj95 2023-10-17T19:12:53.681 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.681 DEBUG:boto:Path: /hbmuog-63/obj95 2023-10-17T19:12:53.681 DEBUG:boto:Data: 2023-10-17T19:12:53.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.682 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.683 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.683 DEBUG:boto:Params: {} 2023-10-17T19:12:53.684 DEBUG:boto:Token: None 2023-10-17T19:12:53.684 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj95 2023-10-17T19:12:53.684 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4YSq8s8VmtvPo2NRfiF4m7IlzV8= 2023-10-17T19:12:53.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4YSq8s8VmtvPo2NRfiF4m7IlzV8='} 2023-10-17T19:12:53.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000088e6bb75a4a1252-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:53.691 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:53.745 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:53.745 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:53.791 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:53.792 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:53.798 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 15 2023-10-17T19:12:53.844 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.845 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.845 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.846 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.846 DEBUG:boto:Data: 2023-10-17T19:12:53.847 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.847 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.847 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.848 DEBUG:boto:Params: {} 2023-10-17T19:12:53.848 DEBUG:boto:Token: None 2023-10-17T19:12:53.849 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.849 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001b5f11f18fad35d-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.853 DEBUG:boto:path=/hbmuog-63/obj23 2023-10-17T19:12:53.854 DEBUG:boto:auth_path=/hbmuog-63/obj23 2023-10-17T19:12:53.854 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.854 DEBUG:boto:Path: /hbmuog-63/obj23 2023-10-17T19:12:53.855 DEBUG:boto:Data: 2023-10-17T19:12:53.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.856 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.856 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.856 DEBUG:boto:Params: {} 2023-10-17T19:12:53.857 DEBUG:boto:Token: None 2023-10-17T19:12:53.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj23 2023-10-17T19:12:53.858 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:a/6F7Dv+F0A0+/fBPT0XjLa77GE= 2023-10-17T19:12:53.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a/6F7Dv+F0A0+/fBPT0XjLa77GE='} 2023-10-17T19:12:53.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050b551531d08545f-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.864 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.865 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.865 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.865 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.866 DEBUG:boto:Data: 2023-10-17T19:12:53.866 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.867 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.867 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.868 DEBUG:boto:Params: {} 2023-10-17T19:12:53.868 DEBUG:boto:Token: None 2023-10-17T19:12:53.869 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.869 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083473a796585242c-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.873 DEBUG:boto:path=/hbmuog-63/obj46 2023-10-17T19:12:53.873 DEBUG:boto:auth_path=/hbmuog-63/obj46 2023-10-17T19:12:53.874 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.874 DEBUG:boto:Path: /hbmuog-63/obj46 2023-10-17T19:12:53.875 DEBUG:boto:Data: 2023-10-17T19:12:53.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.875 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.876 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.876 DEBUG:boto:Params: {} 2023-10-17T19:12:53.877 DEBUG:boto:Token: None 2023-10-17T19:12:53.877 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj46 2023-10-17T19:12:53.878 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PbsMFiOK4YwhdNWBDxqLiZ1HmLU= 2023-10-17T19:12:53.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PbsMFiOK4YwhdNWBDxqLiZ1HmLU='} 2023-10-17T19:12:53.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aed36297b490c6ce-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.927 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.928 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.928 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.928 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.929 DEBUG:boto:Data: 2023-10-17T19:12:53.929 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.930 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.930 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.930 DEBUG:boto:Params: {} 2023-10-17T19:12:53.931 DEBUG:boto:Token: None 2023-10-17T19:12:53.931 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.932 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225314936f8bd1fe-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.935 DEBUG:boto:path=/hbmuog-63/obj69 2023-10-17T19:12:53.936 DEBUG:boto:auth_path=/hbmuog-63/obj69 2023-10-17T19:12:53.936 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.936 DEBUG:boto:Path: /hbmuog-63/obj69 2023-10-17T19:12:53.937 DEBUG:boto:Data: 2023-10-17T19:12:53.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.938 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.938 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.940 DEBUG:boto:Params: {} 2023-10-17T19:12:53.941 DEBUG:boto:Token: None 2023-10-17T19:12:53.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj69 2023-10-17T19:12:53.942 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zA/LEZAdscZNkGBGaUz31zk9kRs= 2023-10-17T19:12:53.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zA/LEZAdscZNkGBGaUz31zk9kRs='} 2023-10-17T19:12:53.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fd924f792b16312-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.949 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:53.949 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:53.950 DEBUG:boto:Method: HEAD 2023-10-17T19:12:53.950 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:53.950 DEBUG:boto:Data: 2023-10-17T19:12:53.951 DEBUG:boto:Headers: {} 2023-10-17T19:12:53.951 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.952 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.952 DEBUG:boto:Params: {} 2023-10-17T19:12:53.952 DEBUG:boto:Token: None 2023-10-17T19:12:53.953 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/ 2023-10-17T19:12:53.953 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:12:53.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:12:53.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9c1ef388f7c6f45-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:53.958 DEBUG:boto:path=/hbmuog-63/obj92 2023-10-17T19:12:53.958 DEBUG:boto:auth_path=/hbmuog-63/obj92 2023-10-17T19:12:53.958 DEBUG:boto:Method: PUT 2023-10-17T19:12:53.959 DEBUG:boto:Path: /hbmuog-63/obj92 2023-10-17T19:12:53.959 DEBUG:boto:Data: 2023-10-17T19:12:53.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:53.960 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:53.960 DEBUG:boto:Port: 8000 2023-10-17T19:12:53.961 DEBUG:boto:Params: {} 2023-10-17T19:12:53.961 DEBUG:boto:Token: None 2023-10-17T19:12:53.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:53 GMT /hbmuog-63/obj92 2023-10-17T19:12:53.962 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:LS1IVM8up4gHXNTXy0GZjHy/vYk= 2023-10-17T19:12:53.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LS1IVM8up4gHXNTXy0GZjHy/vYk='} 2023-10-17T19:12:54.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f3cb137b63c5ae2-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.011 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.012 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.012 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.012 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.013 DEBUG:boto:Data: 2023-10-17T19:12:54.013 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.014 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.014 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.015 DEBUG:boto:Params: {} 2023-10-17T19:12:54.015 DEBUG:boto:Token: None 2023-10-17T19:12:54.016 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000234ed1958a08bd3a-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.020 DEBUG:boto:path=/hbmuog-63/obj115 2023-10-17T19:12:54.021 DEBUG:boto:auth_path=/hbmuog-63/obj115 2023-10-17T19:12:54.021 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.022 DEBUG:boto:Path: /hbmuog-63/obj115 2023-10-17T19:12:54.022 DEBUG:boto:Data: 2023-10-17T19:12:54.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.023 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.023 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.024 DEBUG:boto:Params: {} 2023-10-17T19:12:54.024 DEBUG:boto:Token: None 2023-10-17T19:12:54.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj115 2023-10-17T19:12:54.025 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:udJ3x2v9Cz6gS2qsp29vWgIMS88= 2023-10-17T19:12:54.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:udJ3x2v9Cz6gS2qsp29vWgIMS88='} 2023-10-17T19:12:54.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb22d061c8b5e401-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.032 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:54.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:54.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:54.136 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:54.136 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:54.145 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 20 2023-10-17T19:12:54.196 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.197 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.197 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.198 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.198 DEBUG:boto:Data: 2023-10-17T19:12:54.198 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.199 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.199 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.200 DEBUG:boto:Params: {} 2023-10-17T19:12:54.200 DEBUG:boto:Token: None 2023-10-17T19:12:54.201 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.201 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f081a8addecf9521-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.206 DEBUG:boto:path=/hbmuog-63/obj29 2023-10-17T19:12:54.206 DEBUG:boto:auth_path=/hbmuog-63/obj29 2023-10-17T19:12:54.207 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.207 DEBUG:boto:Path: /hbmuog-63/obj29 2023-10-17T19:12:54.207 DEBUG:boto:Data: 2023-10-17T19:12:54.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.209 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.209 DEBUG:boto:Params: {} 2023-10-17T19:12:54.210 DEBUG:boto:Token: None 2023-10-17T19:12:54.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj29 2023-10-17T19:12:54.211 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:iMExVY69L5fjlJJIdhWD3ycLlxg= 2023-10-17T19:12:54.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iMExVY69L5fjlJJIdhWD3ycLlxg='} 2023-10-17T19:12:54.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd3bd5e2d1c2d281-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.217 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.217 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.218 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.218 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.219 DEBUG:boto:Data: 2023-10-17T19:12:54.219 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.220 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.220 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.220 DEBUG:boto:Params: {} 2023-10-17T19:12:54.221 DEBUG:boto:Token: None 2023-10-17T19:12:54.221 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.222 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007df01919c44158f0-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.227 DEBUG:boto:path=/hbmuog-63/obj58 2023-10-17T19:12:54.227 DEBUG:boto:auth_path=/hbmuog-63/obj58 2023-10-17T19:12:54.228 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.228 DEBUG:boto:Path: /hbmuog-63/obj58 2023-10-17T19:12:54.229 DEBUG:boto:Data: 2023-10-17T19:12:54.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.229 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.230 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.230 DEBUG:boto:Params: {} 2023-10-17T19:12:54.231 DEBUG:boto:Token: None 2023-10-17T19:12:54.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj58 2023-10-17T19:12:54.232 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8c/dThgnUS9NCY0/rSiVwMI3lsU= 2023-10-17T19:12:54.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8c/dThgnUS9NCY0/rSiVwMI3lsU='} 2023-10-17T19:12:54.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f973fed276f1325b-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.279 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.280 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.280 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.281 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.281 DEBUG:boto:Data: 2023-10-17T19:12:54.281 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.282 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.282 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.283 DEBUG:boto:Params: {} 2023-10-17T19:12:54.283 DEBUG:boto:Token: None 2023-10-17T19:12:54.284 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.284 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e912522fbf5f5b5-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.289 DEBUG:boto:path=/hbmuog-63/obj87 2023-10-17T19:12:54.289 DEBUG:boto:auth_path=/hbmuog-63/obj87 2023-10-17T19:12:54.289 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.290 DEBUG:boto:Path: /hbmuog-63/obj87 2023-10-17T19:12:54.290 DEBUG:boto:Data: 2023-10-17T19:12:54.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.291 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.291 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.292 DEBUG:boto:Params: {} 2023-10-17T19:12:54.292 DEBUG:boto:Token: None 2023-10-17T19:12:54.293 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj87 2023-10-17T19:12:54.293 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hSMQ7CbgzTZHD5HyolJWjejc+hE= 2023-10-17T19:12:54.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hSMQ7CbgzTZHD5HyolJWjejc+hE='} 2023-10-17T19:12:54.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd1cf8c0fde5a2f-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.299 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.299 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.300 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.300 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.301 DEBUG:boto:Data: 2023-10-17T19:12:54.301 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.301 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.302 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.302 DEBUG:boto:Params: {} 2023-10-17T19:12:54.303 DEBUG:boto:Token: None 2023-10-17T19:12:54.303 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.304 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b847658bce39b9-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.308 DEBUG:boto:path=/hbmuog-63/obj116 2023-10-17T19:12:54.308 DEBUG:boto:auth_path=/hbmuog-63/obj116 2023-10-17T19:12:54.309 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.309 DEBUG:boto:Path: /hbmuog-63/obj116 2023-10-17T19:12:54.310 DEBUG:boto:Data: 2023-10-17T19:12:54.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.310 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.311 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.311 DEBUG:boto:Params: {} 2023-10-17T19:12:54.312 DEBUG:boto:Token: None 2023-10-17T19:12:54.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj116 2023-10-17T19:12:54.313 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:N18AT6MhnhWeHmmZ0J2K2w9JzgE= 2023-10-17T19:12:54.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N18AT6MhnhWeHmmZ0J2K2w9JzgE='} 2023-10-17T19:12:54.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ddd764b158d4505-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.360 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.360 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.361 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.361 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.362 DEBUG:boto:Data: 2023-10-17T19:12:54.362 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.362 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.363 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.363 DEBUG:boto:Params: {} 2023-10-17T19:12:54.364 DEBUG:boto:Token: None 2023-10-17T19:12:54.364 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.365 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be2b32041317930e-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.369 DEBUG:boto:path=/hbmuog-63/obj145 2023-10-17T19:12:54.370 DEBUG:boto:auth_path=/hbmuog-63/obj145 2023-10-17T19:12:54.370 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.371 DEBUG:boto:Path: /hbmuog-63/obj145 2023-10-17T19:12:54.371 DEBUG:boto:Data: 2023-10-17T19:12:54.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.372 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.372 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.373 DEBUG:boto:Params: {} 2023-10-17T19:12:54.373 DEBUG:boto:Token: None 2023-10-17T19:12:54.374 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj145 2023-10-17T19:12:54.374 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hOs3Tq7amC3ifotsmgXcr6SKwjc= 2023-10-17T19:12:54.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hOs3Tq7amC3ifotsmgXcr6SKwjc='} 2023-10-17T19:12:54.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb7951bea5e9e03c-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.381 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:54.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:54.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:54.484 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:54.485 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:54.495 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 25 2023-10-17T19:12:54.555 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.556 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.557 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.557 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.558 DEBUG:boto:Data: 2023-10-17T19:12:54.558 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.559 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.559 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.559 DEBUG:boto:Params: {} 2023-10-17T19:12:54.560 DEBUG:boto:Token: None 2023-10-17T19:12:54.560 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.561 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001803f100f57add0-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.566 DEBUG:boto:path=/hbmuog-63/obj31 2023-10-17T19:12:54.566 DEBUG:boto:auth_path=/hbmuog-63/obj31 2023-10-17T19:12:54.567 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.567 DEBUG:boto:Path: /hbmuog-63/obj31 2023-10-17T19:12:54.567 DEBUG:boto:Data: 2023-10-17T19:12:54.568 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.568 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.569 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.569 DEBUG:boto:Params: {} 2023-10-17T19:12:54.569 DEBUG:boto:Token: None 2023-10-17T19:12:54.570 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj31 2023-10-17T19:12:54.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wrKfsWpPqurMtCWI4I9P82bwVzE= 2023-10-17T19:12:54.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wrKfsWpPqurMtCWI4I9P82bwVzE='} 2023-10-17T19:12:54.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000960866292ab729ef-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.619 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.620 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.620 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.620 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.621 DEBUG:boto:Data: 2023-10-17T19:12:54.621 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.622 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.622 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.623 DEBUG:boto:Params: {} 2023-10-17T19:12:54.623 DEBUG:boto:Token: None 2023-10-17T19:12:54.624 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.624 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c27f5760e288fb62-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.629 DEBUG:boto:path=/hbmuog-63/obj62 2023-10-17T19:12:54.629 DEBUG:boto:auth_path=/hbmuog-63/obj62 2023-10-17T19:12:54.630 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.630 DEBUG:boto:Path: /hbmuog-63/obj62 2023-10-17T19:12:54.630 DEBUG:boto:Data: 2023-10-17T19:12:54.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.631 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.632 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.632 DEBUG:boto:Params: {} 2023-10-17T19:12:54.633 DEBUG:boto:Token: None 2023-10-17T19:12:54.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj62 2023-10-17T19:12:54.634 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2BwDpJ/fdA3Q7+0pfDCMN8hHfMk= 2023-10-17T19:12:54.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BwDpJ/fdA3Q7+0pfDCMN8hHfMk='} 2023-10-17T19:12:54.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bbdfd04dd65629c-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.640 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.641 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.641 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.641 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.643 DEBUG:boto:Data: 2023-10-17T19:12:54.643 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.643 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.644 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.644 DEBUG:boto:Params: {} 2023-10-17T19:12:54.645 DEBUG:boto:Token: None 2023-10-17T19:12:54.645 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.646 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209966d0ebf50e80-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.650 DEBUG:boto:path=/hbmuog-63/obj93 2023-10-17T19:12:54.650 DEBUG:boto:auth_path=/hbmuog-63/obj93 2023-10-17T19:12:54.651 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.651 DEBUG:boto:Path: /hbmuog-63/obj93 2023-10-17T19:12:54.652 DEBUG:boto:Data: 2023-10-17T19:12:54.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.653 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.653 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.653 DEBUG:boto:Params: {} 2023-10-17T19:12:54.654 DEBUG:boto:Token: None 2023-10-17T19:12:54.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj93 2023-10-17T19:12:54.655 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YCiUIuGplwHMcj/U9SIUEnqtiDI= 2023-10-17T19:12:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YCiUIuGplwHMcj/U9SIUEnqtiDI='} 2023-10-17T19:12:54.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc45f8c07a87db93-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.699 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.700 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.700 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.700 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.701 DEBUG:boto:Data: 2023-10-17T19:12:54.701 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.702 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.702 DEBUG:boto:Params: {} 2023-10-17T19:12:54.703 DEBUG:boto:Token: None 2023-10-17T19:12:54.703 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d2aef81c7028db-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.708 DEBUG:boto:path=/hbmuog-63/obj124 2023-10-17T19:12:54.709 DEBUG:boto:auth_path=/hbmuog-63/obj124 2023-10-17T19:12:54.709 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.709 DEBUG:boto:Path: /hbmuog-63/obj124 2023-10-17T19:12:54.710 DEBUG:boto:Data: 2023-10-17T19:12:54.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.711 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.711 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.711 DEBUG:boto:Params: {} 2023-10-17T19:12:54.712 DEBUG:boto:Token: None 2023-10-17T19:12:54.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj124 2023-10-17T19:12:54.713 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:hKRDHXNo7c2d7It6wjjzSk1jA7g= 2023-10-17T19:12:54.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hKRDHXNo7c2d7It6wjjzSk1jA7g='} 2023-10-17T19:12:54.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c267673b1bdb32f-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.719 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.719 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.720 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.720 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.721 DEBUG:boto:Data: 2023-10-17T19:12:54.721 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.721 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.722 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.722 DEBUG:boto:Params: {} 2023-10-17T19:12:54.723 DEBUG:boto:Token: None 2023-10-17T19:12:54.723 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003197e7259131b781-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.728 DEBUG:boto:path=/hbmuog-63/obj155 2023-10-17T19:12:54.729 DEBUG:boto:auth_path=/hbmuog-63/obj155 2023-10-17T19:12:54.729 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.730 DEBUG:boto:Path: /hbmuog-63/obj155 2023-10-17T19:12:54.731 DEBUG:boto:Data: 2023-10-17T19:12:54.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.732 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.732 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.733 DEBUG:boto:Params: {} 2023-10-17T19:12:54.734 DEBUG:boto:Token: None 2023-10-17T19:12:54.735 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj155 2023-10-17T19:12:54.735 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Y8qELTEyvahGHt1rK4HO4nio6+c= 2023-10-17T19:12:54.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y8qELTEyvahGHt1rK4HO4nio6+c='} 2023-10-17T19:12:54.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079f4fca71e1c8442-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:12:54.784 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:12:54.840 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:54.840 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:54.891 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:12:54.892 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-63 2023-10-17T19:12:54.902 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 30 2023-10-17T19:12:54.963 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.964 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.965 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.965 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.965 DEBUG:boto:Data: 2023-10-17T19:12:54.966 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.966 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.967 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.967 DEBUG:boto:Params: {} 2023-10-17T19:12:54.967 DEBUG:boto:Token: None 2023-10-17T19:12:54.968 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.968 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfb6e48ec1d6b44c-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.973 DEBUG:boto:path=/hbmuog-63/obj37 2023-10-17T19:12:54.974 DEBUG:boto:auth_path=/hbmuog-63/obj37 2023-10-17T19:12:54.974 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.975 DEBUG:boto:Path: /hbmuog-63/obj37 2023-10-17T19:12:54.975 DEBUG:boto:Data: 2023-10-17T19:12:54.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.976 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.976 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.977 DEBUG:boto:Params: {} 2023-10-17T19:12:54.977 DEBUG:boto:Token: None 2023-10-17T19:12:54.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj37 2023-10-17T19:12:54.978 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OUxceuf2nsV4FQmCcF6Lv7iEr/M= 2023-10-17T19:12:54.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OUxceuf2nsV4FQmCcF6Lv7iEr/M='} 2023-10-17T19:12:54.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008046291ffb5550b8-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.984 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:54.985 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:54.985 DEBUG:boto:Method: HEAD 2023-10-17T19:12:54.986 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:54.986 DEBUG:boto:Data: 2023-10-17T19:12:54.987 DEBUG:boto:Headers: {} 2023-10-17T19:12:54.987 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.988 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.988 DEBUG:boto:Params: {} 2023-10-17T19:12:54.989 DEBUG:boto:Token: None 2023-10-17T19:12:54.989 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/ 2023-10-17T19:12:54.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:12:54.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:12:54.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a396fe478dc6cc28-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:54.994 DEBUG:boto:path=/hbmuog-63/obj74 2023-10-17T19:12:54.994 DEBUG:boto:auth_path=/hbmuog-63/obj74 2023-10-17T19:12:54.995 DEBUG:boto:Method: PUT 2023-10-17T19:12:54.995 DEBUG:boto:Path: /hbmuog-63/obj74 2023-10-17T19:12:54.995 DEBUG:boto:Data: 2023-10-17T19:12:54.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:54.996 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:54.997 DEBUG:boto:Port: 8000 2023-10-17T19:12:54.997 DEBUG:boto:Params: {} 2023-10-17T19:12:54.997 DEBUG:boto:Token: None 2023-10-17T19:12:54.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:54 GMT /hbmuog-63/obj74 2023-10-17T19:12:54.998 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vzJOH3FOpTotFoIboOoIlJpNFhI= 2023-10-17T19:12:54.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vzJOH3FOpTotFoIboOoIlJpNFhI='} 2023-10-17T19:12:55.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e474278bf9d3cef0-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.005 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:55.005 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:55.005 DEBUG:boto:Method: HEAD 2023-10-17T19:12:55.006 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:55.006 DEBUG:boto:Data: 2023-10-17T19:12:55.007 DEBUG:boto:Headers: {} 2023-10-17T19:12:55.007 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.007 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.008 DEBUG:boto:Params: {} 2023-10-17T19:12:55.008 DEBUG:boto:Token: None 2023-10-17T19:12:55.009 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/ 2023-10-17T19:12:55.009 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:12:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:12:55.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8de06b70da1674e-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.014 DEBUG:boto:path=/hbmuog-63/obj111 2023-10-17T19:12:55.014 DEBUG:boto:auth_path=/hbmuog-63/obj111 2023-10-17T19:12:55.015 DEBUG:boto:Method: PUT 2023-10-17T19:12:55.015 DEBUG:boto:Path: /hbmuog-63/obj111 2023-10-17T19:12:55.016 DEBUG:boto:Data: 2023-10-17T19:12:55.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:55.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.017 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.018 DEBUG:boto:Params: {} 2023-10-17T19:12:55.018 DEBUG:boto:Token: None 2023-10-17T19:12:55.019 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/obj111 2023-10-17T19:12:55.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zU0Kw+bLn8wkNDA94jPT1goGzeg= 2023-10-17T19:12:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zU0Kw+bLn8wkNDA94jPT1goGzeg='} 2023-10-17T19:12:55.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a61d7f7b1ccb266-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.063 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:55.064 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:55.064 DEBUG:boto:Method: HEAD 2023-10-17T19:12:55.065 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:55.065 DEBUG:boto:Data: 2023-10-17T19:12:55.065 DEBUG:boto:Headers: {} 2023-10-17T19:12:55.066 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.066 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.067 DEBUG:boto:Params: {} 2023-10-17T19:12:55.067 DEBUG:boto:Token: None 2023-10-17T19:12:55.068 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/ 2023-10-17T19:12:55.069 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:12:55.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:12:55.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042daa328b4b22e60-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.074 DEBUG:boto:path=/hbmuog-63/obj148 2023-10-17T19:12:55.074 DEBUG:boto:auth_path=/hbmuog-63/obj148 2023-10-17T19:12:55.074 DEBUG:boto:Method: PUT 2023-10-17T19:12:55.075 DEBUG:boto:Path: /hbmuog-63/obj148 2023-10-17T19:12:55.075 DEBUG:boto:Data: 2023-10-17T19:12:55.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:55.076 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.077 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.077 DEBUG:boto:Params: {} 2023-10-17T19:12:55.078 DEBUG:boto:Token: None 2023-10-17T19:12:55.078 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/obj148 2023-10-17T19:12:55.079 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ea6o1y89+c8mTjEyaTB2Lt9Zx38= 2023-10-17T19:12:55.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ea6o1y89+c8mTjEyaTB2Lt9Zx38='} 2023-10-17T19:12:55.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000601d7b8f5ec18f2a-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.086 DEBUG:boto:path=/hbmuog-63/ 2023-10-17T19:12:55.087 DEBUG:boto:auth_path=/hbmuog-63/ 2023-10-17T19:12:55.088 DEBUG:boto:Method: HEAD 2023-10-17T19:12:55.088 DEBUG:boto:Path: /hbmuog-63/ 2023-10-17T19:12:55.089 DEBUG:boto:Data: 2023-10-17T19:12:55.089 DEBUG:boto:Headers: {} 2023-10-17T19:12:55.090 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.090 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.091 DEBUG:boto:Params: {} 2023-10-17T19:12:55.091 DEBUG:boto:Token: None 2023-10-17T19:12:55.092 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/ 2023-10-17T19:12:55.092 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:12:55.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:12:55.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000214e5c6cb9f2e252-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.097 DEBUG:boto:path=/hbmuog-63/obj185 2023-10-17T19:12:55.098 DEBUG:boto:auth_path=/hbmuog-63/obj185 2023-10-17T19:12:55.098 DEBUG:boto:Method: PUT 2023-10-17T19:12:55.099 DEBUG:boto:Path: /hbmuog-63/obj185 2023-10-17T19:12:55.099 DEBUG:boto:Data: 2023-10-17T19:12:55.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:12:55.100 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:12:55.100 DEBUG:boto:Port: 8000 2023-10-17T19:12:55.101 DEBUG:boto:Params: {} 2023-10-17T19:12:55.101 DEBUG:boto:Token: None 2023-10-17T19:12:55.102 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:12:55 GMT /hbmuog-63/obj185 2023-10-17T19:12:55.102 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQbeXEzKAbJI3JFcfVzNtJJSQlI= 2023-10-17T19:12:55.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbeXEzKAbJI3JFcfVzNtJJSQlI='} 2023-10-17T19:12:55.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0e11cab72c120c2-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:12:55.151 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-63 2023-10-17T19:12:55.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:12:55.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:12:55.153 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:12:55.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:12:55.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:12:55.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:12:55.248+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:13:00.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:00.249+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:13:05.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:05.250+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:13:10.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:10.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:13:15.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:15.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:20.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:20.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:25.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:25.252+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:30.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:30.253+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:35.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:35.254+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:40.256 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:40.255+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:45.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:45.256+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:50.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:13:55.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:13:55.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:00.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:05.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:05.258+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:10.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:10.259+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:15.261 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:15.260+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:20.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:20.261+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:25.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:25.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:30.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:30.261+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:35.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:40.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:40.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:45.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:45.263+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:50.264+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:14:55.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:14:55.265+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:15:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:00.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:15:05.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:05.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:15:10.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:10.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:15:15.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:15.267+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:20.269 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:20.267+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:25.270 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:25.268+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:30.270 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:30.269+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:35.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:35.270+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:40.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:40.271+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:45.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:45.270+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:50.271+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:15:55.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:15:55.272+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:00.274 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:00.272+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:05.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:05.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:10.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:10.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:15.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:15.276+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:20.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:20.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:25.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:25.276+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:30.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:30.277+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:35.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:35.277+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:40.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:40.278+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:45.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:45.279+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:50.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:16:55.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:16:55.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:00.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:00.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:05.281+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:10.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:10.281+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:15.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:15.283+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:20.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:20.284+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:25.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:25.285+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:30.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:30.284+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:35.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:35.285+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:40.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:40.286+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:45.287+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:17:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.288+0000 7fadb5999780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-17T19:17:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.288+0000 7fadb5999780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:17:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.288+0000 7fadb5999780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:17:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-17T19:17:50.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-10-17T19:17:50.310 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.311 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.311 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.312 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.312 DEBUG:boto:Data: 2023-10-17T19:17:50.312 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.313 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.314 DEBUG:boto:Params: {} 2023-10-17T19:17:50.314 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:17:50.315 DEBUG:boto:Token: None 2023-10-17T19:17:50.315 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.316 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ohoEmQ8NrhPucVc4lWZW4b+mnEk= 2023-10-17T19:17:50.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ohoEmQ8NrhPucVc4lWZW4b+mnEk=', 'Content-Length': '0'} 2023-10-17T19:17:50.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000632af682a691b72f-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.338 DEBUG:rgw_multi.tests:created bucket=hbmuog-64 2023-10-17T19:17:50.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:17:50.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:17:50.340 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:17:50.398 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:17:50.399 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:17:50.436 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:17:50.450 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:17:50.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.450 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570270.335817_873.1", 2023-10-17T19:17:50.450 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:17:50.335817Z" 2023-10-17T19:17:50.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.451 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:17:50.451 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:17:50.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.452 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:17:50.452 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:17:50.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.453 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:17:50.453 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:17:50.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.453 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:17:50.454 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:17:50.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:17:50.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:17:50.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:17:50.456 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:17:50.507 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:17:50.507 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:17:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.536+0000 7f511e104780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:17:50.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.537+0000 7f511e104780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:17:50.552 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:17:50.552 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:17:50.552 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:17:50.552 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:17:50.552 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:17:50.553 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:17:50.553 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:17:50.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.553 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:17:50.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:17:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:17:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:17:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570270.335817_873.1", 2023-10-17T19:17:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:17:50.555 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:17:50.555 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:17:50.555 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:17:50.335817Z", 2023-10-17T19:17:50.555 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:17:50.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.556 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:17:50.556 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:17:50.556 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:17:50.557 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:17:50.557 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:17:50.557 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:17:50.557 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:17:50.558 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:17:50.558 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:17:50.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.559 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:17:50.559 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:17:50.559 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:17:50.559 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:17:50.559 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:17:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:17:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:17:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:17:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:17:50.562 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:17:50.562 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:17:50.562 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:17:50.562 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:17:50.563 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:17:50.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:17:50.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:17:50.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:17:50Z" 2023-10-17T19:17:50.564 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:17:50.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570270.335817_873.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:17:50.335817Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:17:50Z" } 2023-10-17T19:17:50.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570270.335817_873.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:17:50.335817Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:17:50.566 DEBUG:rgw_multi.tests:log_status={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:17:50.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:17:50.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:17:50.567 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.568 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.568 DEBUG:boto:Method: HEAD 2023-10-17T19:17:50.569 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.569 DEBUG:boto:Data: 2023-10-17T19:17:50.569 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.570 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.570 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.570 DEBUG:boto:Params: {} 2023-10-17T19:17:50.571 DEBUG:boto:Token: None 2023-10-17T19:17:50.571 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.572 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:17:50.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:17:50.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f584df89f6d04b8-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.575 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:17:50.575 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:17:50.576 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.576 DEBUG:boto:Path: /hbmuog-64/obj11 2023-10-17T19:17:50.576 DEBUG:boto:Data: 2023-10-17T19:17:50.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:17:50.577 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.578 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.578 DEBUG:boto:Params: {} 2023-10-17T19:17:50.578 DEBUG:boto:Token: None 2023-10-17T19:17:50.579 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/obj11 2023-10-17T19:17:50.579 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:q38I4sEkUlx3+iS384+ktxu34Ps= 2023-10-17T19:17:50.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q38I4sEkUlx3+iS384+ktxu34Ps='} 2023-10-17T19:17:50.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d9e11caf2ce72e5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.586 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.587 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.587 DEBUG:boto:Method: HEAD 2023-10-17T19:17:50.588 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.588 DEBUG:boto:Data: 2023-10-17T19:17:50.588 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.589 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.589 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.589 DEBUG:boto:Params: {} 2023-10-17T19:17:50.590 DEBUG:boto:Token: None 2023-10-17T19:17:50.590 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.591 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:17:50.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:17:50.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004464e8d368521ca-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.594 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:17:50.594 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:17:50.595 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.595 DEBUG:boto:Path: /hbmuog-64/obj22 2023-10-17T19:17:50.595 DEBUG:boto:Data: 2023-10-17T19:17:50.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:17:50.596 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.597 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.597 DEBUG:boto:Params: {} 2023-10-17T19:17:50.597 DEBUG:boto:Token: None 2023-10-17T19:17:50.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/obj22 2023-10-17T19:17:50.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:urgUbFqKRzW9AD1XmfO0l9HFfoA= 2023-10-17T19:17:50.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:urgUbFqKRzW9AD1XmfO0l9HFfoA='} 2023-10-17T19:17:50.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b50c11b982931b4-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.603 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.603 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.604 DEBUG:boto:Method: HEAD 2023-10-17T19:17:50.604 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.604 DEBUG:boto:Data: 2023-10-17T19:17:50.605 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.605 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.605 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.606 DEBUG:boto:Params: {} 2023-10-17T19:17:50.606 DEBUG:boto:Token: None 2023-10-17T19:17:50.607 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.607 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:17:50.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:17:50.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082623cd3f58820ea-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.610 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:17:50.610 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:17:50.611 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.611 DEBUG:boto:Path: /hbmuog-64/obj33 2023-10-17T19:17:50.611 DEBUG:boto:Data: 2023-10-17T19:17:50.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:17:50.612 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.613 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.613 DEBUG:boto:Params: {} 2023-10-17T19:17:50.613 DEBUG:boto:Token: None 2023-10-17T19:17:50.614 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/obj33 2023-10-17T19:17:50.614 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:cni1zWucfyHJxLAtWA1JgAq7kwo= 2023-10-17T19:17:50.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cni1zWucfyHJxLAtWA1JgAq7kwo='} 2023-10-17T19:17:50.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd8b8354ce8bd8e5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.659 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.659 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.659 DEBUG:boto:Method: HEAD 2023-10-17T19:17:50.660 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.660 DEBUG:boto:Data: 2023-10-17T19:17:50.660 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.661 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.661 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.662 DEBUG:boto:Params: {} 2023-10-17T19:17:50.662 DEBUG:boto:Token: None 2023-10-17T19:17:50.662 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.663 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:17:50.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:17:50.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f799dcf04e8cbf21-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.666 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:17:50.666 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:17:50.667 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.667 DEBUG:boto:Path: /hbmuog-64/obj44 2023-10-17T19:17:50.667 DEBUG:boto:Data: 2023-10-17T19:17:50.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:17:50.668 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.668 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.669 DEBUG:boto:Params: {} 2023-10-17T19:17:50.669 DEBUG:boto:Token: None 2023-10-17T19:17:50.669 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/obj44 2023-10-17T19:17:50.670 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oYQpYkY7Kqhsa9vJ0ouJHamj6bQ= 2023-10-17T19:17:50.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oYQpYkY7Kqhsa9vJ0ouJHamj6bQ='} 2023-10-17T19:17:50.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099d45756a0152d7b-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.676 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:17:50.676 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:17:50.676 DEBUG:boto:Method: HEAD 2023-10-17T19:17:50.677 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:17:50.677 DEBUG:boto:Data: 2023-10-17T19:17:50.678 DEBUG:boto:Headers: {} 2023-10-17T19:17:50.678 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.678 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.679 DEBUG:boto:Params: {} 2023-10-17T19:17:50.679 DEBUG:boto:Token: None 2023-10-17T19:17:50.679 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/ 2023-10-17T19:17:50.680 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:17:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:17:50.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015680c21ee493f6f-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.684 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:17:50.684 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:17:50.684 DEBUG:boto:Method: PUT 2023-10-17T19:17:50.685 DEBUG:boto:Path: /hbmuog-64/obj55 2023-10-17T19:17:50.685 DEBUG:boto:Data: 2023-10-17T19:17:50.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:17:50.686 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:17:50.686 DEBUG:boto:Port: 8000 2023-10-17T19:17:50.686 DEBUG:boto:Params: {} 2023-10-17T19:17:50.687 DEBUG:boto:Token: None 2023-10-17T19:17:50.687 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:17:50 GMT /hbmuog-64/obj55 2023-10-17T19:17:50.688 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vc41eCl6JLWmJaP83dHo2rWSav8= 2023-10-17T19:17:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vc41eCl6JLWmJaP83dHo2rWSav8='} 2023-10-17T19:17:50.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000418c68aecc28d7c5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:17:50.731 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-64 2023-10-17T19:17:50.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:17:50.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:17:50.732 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:17:50.790 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:17:50.791 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:17:50.827 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:50.825+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:17:55.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:17:55.827+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:18:00.828 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:00.828+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:18:05.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:05.828+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:18:10.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:10.829+0000 7fa1f118f780 1 bucket sync caught up with source: 2023-10-17T19:18:10.832 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.185.6, 00000000002.91.6, 00000000002.114.6] 2023-10-17T19:18:10.832 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.185.6, 00000000002.91.6, 00000000002.114.6] 2023-10-17T19:18:10.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:10.829+0000 7fa1f118f780 0 bucket checkpoint complete 2023-10-17T19:18:10.848 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hbmuog-64 2023-10-17T19:18:10.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:18:10.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:18:10.849 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:18:10.913 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:10.913 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:10.951 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:10.949+0000 7f0db6a7d780 1 waiting to reach incremental sync.. 2023-10-17T19:18:15.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:15.951+0000 7f0db6a7d780 1 bucket sync caught up with source: 2023-10-17T19:18:15.953 INFO:teuthology.orchestra.run.smithi049.stderr: local status: [, , , , , , , , 00000000001.484.6, 00000000002.158.6, 00000000002.398.6] 2023-10-17T19:18:15.954 INFO:teuthology.orchestra.run.smithi049.stderr: remote markers: [, , , , , , , , 00000000001.484.6, 00000000002.158.6, 00000000002.398.6] 2023-10-17T19:18:15.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:15.951+0000 7f0db6a7d780 0 bucket checkpoint complete 2023-10-17T19:18:15.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hbmuog-64 zones={a2, a1} 2023-10-17T19:18:15.971 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:15.971 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:15.972 DEBUG:boto:Method: HEAD 2023-10-17T19:18:15.972 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:15.972 DEBUG:boto:Data: 2023-10-17T19:18:15.973 DEBUG:boto:Headers: {} 2023-10-17T19:18:15.973 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:15.974 DEBUG:boto:Port: 8001 2023-10-17T19:18:15.974 DEBUG:boto:Params: {} 2023-10-17T19:18:15.975 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:18:15.975 DEBUG:boto:Token: None 2023-10-17T19:18:15.976 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:15 GMT /hbmuog-64/ 2023-10-17T19:18:15.976 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4= 2023-10-17T19:18:15.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4=', 'Content-Length': '0'} 2023-10-17T19:18:15.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb418fcc4d8c55f1-00652eddf7-1d4d-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:15.981 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:15.982 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:15.982 DEBUG:boto:Method: HEAD 2023-10-17T19:18:15.983 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:15.983 DEBUG:boto:Data: 2023-10-17T19:18:15.983 DEBUG:boto:Headers: {} 2023-10-17T19:18:15.984 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:15.984 DEBUG:boto:Port: 8000 2023-10-17T19:18:15.985 DEBUG:boto:Params: {} 2023-10-17T19:18:15.985 DEBUG:boto:Token: None 2023-10-17T19:18:15.985 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:15 GMT /hbmuog-64/ 2023-10-17T19:18:15.986 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4= 2023-10-17T19:18:15.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4=', 'Content-Length': '0'} 2023-10-17T19:18:15.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b9efd1337762e0f-00652eddf7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:15.990 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-17T19:18:15.990 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:15.990 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:15.991 DEBUG:boto:path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:15.991 DEBUG:boto:auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:15.992 DEBUG:boto:Method: GET 2023-10-17T19:18:15.992 DEBUG:boto:Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:15.992 DEBUG:boto:Data: 2023-10-17T19:18:15.993 DEBUG:boto:Headers: {} 2023-10-17T19:18:15.993 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:15.994 DEBUG:boto:Port: 8001 2023-10-17T19:18:15.994 DEBUG:boto:Params: {} 2023-10-17T19:18:15.995 DEBUG:boto:Token: None 2023-10-17T19:18:15.995 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:15 GMT /hbmuog-64/?versions 2023-10-17T19:18:15.995 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:CjyZJ+6T9mg+paddRGzexX+NJuk= 2023-10-17T19:18:15.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CjyZJ+6T9mg+paddRGzexX+NJuk=', 'Content-Length': '0'} 2023-10-17T19:18:15.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e643ee8952f624-00652eddf7-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:15.999 DEBUG:boto:b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:18:16.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:18:16.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:18:16.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:18:16.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:18:16.002 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-17T19:18:16.002 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.003 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.003 DEBUG:boto:path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.003 DEBUG:boto:auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.004 DEBUG:boto:Method: GET 2023-10-17T19:18:16.004 DEBUG:boto:Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.005 DEBUG:boto:Data: 2023-10-17T19:18:16.005 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.006 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.006 DEBUG:boto:Params: {} 2023-10-17T19:18:16.007 DEBUG:boto:Token: None 2023-10-17T19:18:16.007 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/?versions 2023-10-17T19:18:16.008 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:18:16.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:18:16.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa481eb620b7abd-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.011 DEBUG:boto:b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-17T19:18:16.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-17T19:18:16.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-17T19:18:16.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-17T19:18:16.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-17T19:18:16.014 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.014 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.015 DEBUG:boto:path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.015 DEBUG:boto:auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.015 DEBUG:boto:Method: GET 2023-10-17T19:18:16.016 DEBUG:boto:Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.016 DEBUG:boto:Data: 2023-10-17T19:18:16.016 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.017 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.017 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.018 DEBUG:boto:Params: {} 2023-10-17T19:18:16.018 DEBUG:boto:Token: None 2023-10-17T19:18:16.018 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/?versions 2023-10-17T19:18:16.019 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:18:16.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:18:16.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4cab6b3de5668e-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.023 DEBUG:boto:b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.023 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.024 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.024 DEBUG:boto:path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.025 DEBUG:boto:auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.025 DEBUG:boto:Method: GET 2023-10-17T19:18:16.025 DEBUG:boto:Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:18:16.026 DEBUG:boto:Data: 2023-10-17T19:18:16.026 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.027 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.027 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.027 DEBUG:boto:Params: {} 2023-10-17T19:18:16.028 DEBUG:boto:Token: None 2023-10-17T19:18:16.028 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/?versions 2023-10-17T19:18:16.029 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:18:16.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:18:16.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5756d3ecb16b47-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.032 DEBUG:boto:b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:18:16.033 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.034 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.034 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.035 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.035 DEBUG:boto:Method: GET 2023-10-17T19:18:16.035 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.036 DEBUG:boto:Data: 2023-10-17T19:18:16.036 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.037 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.037 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.037 DEBUG:boto:Params: {} 2023-10-17T19:18:16.038 DEBUG:boto:Token: None 2023-10-17T19:18:16.038 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.039 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:18:16.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:18:16.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5c9a7f53d438c3-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.042 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.042 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.043 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.043 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.043 DEBUG:boto:Method: GET 2023-10-17T19:18:16.044 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.044 DEBUG:boto:Data: 2023-10-17T19:18:16.045 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.045 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.045 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.046 DEBUG:boto:Params: {} 2023-10-17T19:18:16.046 DEBUG:boto:Token: None 2023-10-17T19:18:16.047 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.047 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:18:16.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:18:16.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afe955e940727cd-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.050 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.051 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.051 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.052 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.052 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.052 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.053 DEBUG:boto:Data: 2023-10-17T19:18:16.053 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.053 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.054 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.054 DEBUG:boto:Params: {} 2023-10-17T19:18:16.055 DEBUG:boto:Token: None 2023-10-17T19:18:16.055 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA= 2023-10-17T19:18:16.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA=', 'Content-Length': '0'} 2023-10-17T19:18:16.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe71834a2302ea2a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.058 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.059 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.059 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.059 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.060 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.060 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.061 DEBUG:boto:Data: 2023-10-17T19:18:16.061 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.061 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.062 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.062 DEBUG:boto:Params: {} 2023-10-17T19:18:16.063 DEBUG:boto:Token: None 2023-10-17T19:18:16.063 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.064 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA= 2023-10-17T19:18:16.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA=', 'Content-Length': '0'} 2023-10-17T19:18:16.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003133f98bd1b97c3c-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:18:16.067 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.068 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.068 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.068 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.069 DEBUG:boto:Method: GET 2023-10-17T19:18:16.069 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.070 DEBUG:boto:Data: 2023-10-17T19:18:16.070 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.070 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.071 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.071 DEBUG:boto:Params: {} 2023-10-17T19:18:16.072 DEBUG:boto:Token: None 2023-10-17T19:18:16.072 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:18:16.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:18:16.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b9bda415bf781a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.076 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.076 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.076 DEBUG:boto:path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.077 DEBUG:boto:auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.077 DEBUG:boto:Method: GET 2023-10-17T19:18:16.078 DEBUG:boto:Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.078 DEBUG:boto:Data: 2023-10-17T19:18:16.078 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.079 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.079 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.080 DEBUG:boto:Params: {} 2023-10-17T19:18:16.080 DEBUG:boto:Token: None 2023-10-17T19:18:16.081 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11?versionId=null 2023-10-17T19:18:16.081 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:18:16.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:18:16.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435ef6b811b4c4e3-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.084 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.085 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.085 DEBUG:boto:path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.086 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.086 DEBUG:boto:Method: GET 2023-10-17T19:18:16.086 DEBUG:boto:Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.087 DEBUG:boto:Data: 2023-10-17T19:18:16.087 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.088 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.088 DEBUG:boto:Params: {} 2023-10-17T19:18:16.089 DEBUG:boto:Token: None 2023-10-17T19:18:16.089 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb78ef927390fb84-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.134 DEBUG:boto:b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.135 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.135 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.136 DEBUG:boto:path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.136 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.136 DEBUG:boto:Method: GET 2023-10-17T19:18:16.137 DEBUG:boto:Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.137 DEBUG:boto:Data: 2023-10-17T19:18:16.138 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.138 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.139 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.139 DEBUG:boto:Params: {} 2023-10-17T19:18:16.139 DEBUG:boto:Token: None 2023-10-17T19:18:16.140 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.140 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8146735e6b9e5bd-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.144 DEBUG:boto:b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.144 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.145 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.145 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.145 DEBUG:boto:Path: /hbmuog-64/obj11 2023-10-17T19:18:16.146 DEBUG:boto:Data: 2023-10-17T19:18:16.146 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.147 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.147 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.149 DEBUG:boto:Params: {} 2023-10-17T19:18:16.150 DEBUG:boto:Token: None 2023-10-17T19:18:16.150 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11 2023-10-17T19:18:16.150 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8= 2023-10-17T19:18:16.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8=', 'Content-Length': '0'} 2023-10-17T19:18:16.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002575848d76cd631f-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.153 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.154 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.154 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.154 DEBUG:boto:Path: /hbmuog-64/obj11 2023-10-17T19:18:16.155 DEBUG:boto:Data: 2023-10-17T19:18:16.155 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.155 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.156 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.156 DEBUG:boto:Params: {} 2023-10-17T19:18:16.157 DEBUG:boto:Token: None 2023-10-17T19:18:16.157 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11 2023-10-17T19:18:16.157 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8= 2023-10-17T19:18:16.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8=', 'Content-Length': '0'} 2023-10-17T19:18:16.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b23f7bea02c553aa-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-17T19:18:16.161 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.161 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.162 DEBUG:boto:Method: GET 2023-10-17T19:18:16.162 DEBUG:boto:Path: /hbmuog-64/obj11 2023-10-17T19:18:16.162 DEBUG:boto:Data: 2023-10-17T19:18:16.163 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.163 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.164 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.164 DEBUG:boto:Params: {} 2023-10-17T19:18:16.164 DEBUG:boto:Token: None 2023-10-17T19:18:16.165 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11 2023-10-17T19:18:16.165 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI= 2023-10-17T19:18:16.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI=', 'Content-Length': '0'} 2023-10-17T19:18:16.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010246b53cd73547d-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.169 DEBUG:boto:path=/hbmuog-64/obj11 2023-10-17T19:18:16.169 DEBUG:boto:auth_path=/hbmuog-64/obj11 2023-10-17T19:18:16.169 DEBUG:boto:Method: GET 2023-10-17T19:18:16.170 DEBUG:boto:Path: /hbmuog-64/obj11 2023-10-17T19:18:16.170 DEBUG:boto:Data: 2023-10-17T19:18:16.171 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.171 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.171 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.172 DEBUG:boto:Params: {} 2023-10-17T19:18:16.172 DEBUG:boto:Token: None 2023-10-17T19:18:16.173 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj11 2023-10-17T19:18:16.173 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI= 2023-10-17T19:18:16.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI=', 'Content-Length': '0'} 2023-10-17T19:18:16.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000dfe4a4d2b88798-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.218 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.219 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.219 DEBUG:boto:path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.220 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.220 DEBUG:boto:Method: GET 2023-10-17T19:18:16.221 DEBUG:boto:Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.221 DEBUG:boto:Data: 2023-10-17T19:18:16.221 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.222 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.222 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.223 DEBUG:boto:Params: {} 2023-10-17T19:18:16.223 DEBUG:boto:Token: None 2023-10-17T19:18:16.223 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.224 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc7975c3426d7e1-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.227 DEBUG:boto:b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.228 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.228 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.229 DEBUG:boto:path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.229 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.230 DEBUG:boto:Method: GET 2023-10-17T19:18:16.230 DEBUG:boto:Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:18:16.230 DEBUG:boto:Data: 2023-10-17T19:18:16.231 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.231 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.232 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.232 DEBUG:boto:Params: {} 2023-10-17T19:18:16.233 DEBUG:boto:Token: None 2023-10-17T19:18:16.233 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.234 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbcb36e6c5d9e82f-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.237 DEBUG:boto:b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:18:16.238 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.238 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.239 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.239 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.240 DEBUG:boto:Method: GET 2023-10-17T19:18:16.240 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.241 DEBUG:boto:Data: 2023-10-17T19:18:16.241 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.241 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.242 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.242 DEBUG:boto:Params: {} 2023-10-17T19:18:16.243 DEBUG:boto:Token: None 2023-10-17T19:18:16.243 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.244 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:18:16.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:18:16.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3410c9ffa4eb14-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.247 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.247 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.248 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.248 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.249 DEBUG:boto:Method: GET 2023-10-17T19:18:16.249 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.249 DEBUG:boto:Data: 2023-10-17T19:18:16.250 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.250 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.251 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.251 DEBUG:boto:Params: {} 2023-10-17T19:18:16.252 DEBUG:boto:Token: None 2023-10-17T19:18:16.252 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.252 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:18:16.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:18:16.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aeea560c79e8971-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.256 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.256 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.256 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.257 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.257 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.258 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.258 DEBUG:boto:Data: 2023-10-17T19:18:16.258 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.259 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.259 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.259 DEBUG:boto:Params: {} 2023-10-17T19:18:16.260 DEBUG:boto:Token: None 2023-10-17T19:18:16.260 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.261 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ= 2023-10-17T19:18:16.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ=', 'Content-Length': '0'} 2023-10-17T19:18:16.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d16304f6337994-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.264 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.264 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.264 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.265 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.265 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.265 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.266 DEBUG:boto:Data: 2023-10-17T19:18:16.266 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.267 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.267 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.267 DEBUG:boto:Params: {} 2023-10-17T19:18:16.268 DEBUG:boto:Token: None 2023-10-17T19:18:16.268 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.269 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ= 2023-10-17T19:18:16.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ=', 'Content-Length': '0'} 2023-10-17T19:18:16.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bf1ab5f1ea3c3c-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:18:16.272 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.272 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.273 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.273 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.274 DEBUG:boto:Method: GET 2023-10-17T19:18:16.274 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.274 DEBUG:boto:Data: 2023-10-17T19:18:16.275 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.275 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.276 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.276 DEBUG:boto:Params: {} 2023-10-17T19:18:16.276 DEBUG:boto:Token: None 2023-10-17T19:18:16.277 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.277 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:18:16.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:18:16.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc15af1596dffe8a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.322 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.323 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.323 DEBUG:boto:path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.324 DEBUG:boto:auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.324 DEBUG:boto:Method: GET 2023-10-17T19:18:16.325 DEBUG:boto:Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.325 DEBUG:boto:Data: 2023-10-17T19:18:16.325 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.326 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.326 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.327 DEBUG:boto:Params: {} 2023-10-17T19:18:16.327 DEBUG:boto:Token: None 2023-10-17T19:18:16.328 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22?versionId=null 2023-10-17T19:18:16.328 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:18:16.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:18:16.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5a8e7f9d1c2591-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.331 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.332 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.332 DEBUG:boto:path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.332 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.333 DEBUG:boto:Method: GET 2023-10-17T19:18:16.333 DEBUG:boto:Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.334 DEBUG:boto:Data: 2023-10-17T19:18:16.334 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.334 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.335 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.335 DEBUG:boto:Params: {} 2023-10-17T19:18:16.336 DEBUG:boto:Token: None 2023-10-17T19:18:16.336 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.337 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003932d7868cfcde58-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.340 DEBUG:boto:b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.340 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.341 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.341 DEBUG:boto:path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.342 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.342 DEBUG:boto:Method: GET 2023-10-17T19:18:16.343 DEBUG:boto:Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.343 DEBUG:boto:Data: 2023-10-17T19:18:16.343 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.344 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.344 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.344 DEBUG:boto:Params: {} 2023-10-17T19:18:16.345 DEBUG:boto:Token: None 2023-10-17T19:18:16.345 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.346 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a57b651dfa6f5f6-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.390 DEBUG:boto:b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.391 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.391 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.392 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.392 DEBUG:boto:Path: /hbmuog-64/obj22 2023-10-17T19:18:16.393 DEBUG:boto:Data: 2023-10-17T19:18:16.393 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.393 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.394 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.394 DEBUG:boto:Params: {} 2023-10-17T19:18:16.395 DEBUG:boto:Token: None 2023-10-17T19:18:16.395 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22 2023-10-17T19:18:16.395 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U= 2023-10-17T19:18:16.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U=', 'Content-Length': '0'} 2023-10-17T19:18:16.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074189c29a7081fd2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.399 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.399 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.399 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.400 DEBUG:boto:Path: /hbmuog-64/obj22 2023-10-17T19:18:16.400 DEBUG:boto:Data: 2023-10-17T19:18:16.401 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.401 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.401 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.402 DEBUG:boto:Params: {} 2023-10-17T19:18:16.402 DEBUG:boto:Token: None 2023-10-17T19:18:16.403 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22 2023-10-17T19:18:16.403 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U= 2023-10-17T19:18:16.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U=', 'Content-Length': '0'} 2023-10-17T19:18:16.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50ccfb9e40300aa-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-17T19:18:16.407 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.407 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.407 DEBUG:boto:Method: GET 2023-10-17T19:18:16.408 DEBUG:boto:Path: /hbmuog-64/obj22 2023-10-17T19:18:16.408 DEBUG:boto:Data: 2023-10-17T19:18:16.408 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.409 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.409 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.410 DEBUG:boto:Params: {} 2023-10-17T19:18:16.410 DEBUG:boto:Token: None 2023-10-17T19:18:16.411 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22 2023-10-17T19:18:16.411 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4= 2023-10-17T19:18:16.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4=', 'Content-Length': '0'} 2023-10-17T19:18:16.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103034f2d907f97f-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.414 DEBUG:boto:path=/hbmuog-64/obj22 2023-10-17T19:18:16.415 DEBUG:boto:auth_path=/hbmuog-64/obj22 2023-10-17T19:18:16.415 DEBUG:boto:Method: GET 2023-10-17T19:18:16.415 DEBUG:boto:Path: /hbmuog-64/obj22 2023-10-17T19:18:16.416 DEBUG:boto:Data: 2023-10-17T19:18:16.416 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.417 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.417 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.417 DEBUG:boto:Params: {} 2023-10-17T19:18:16.418 DEBUG:boto:Token: None 2023-10-17T19:18:16.418 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj22 2023-10-17T19:18:16.419 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4= 2023-10-17T19:18:16.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4=', 'Content-Length': '0'} 2023-10-17T19:18:16.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa35a354a64e4b8-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.422 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.423 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.423 DEBUG:boto:path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.423 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.424 DEBUG:boto:Method: GET 2023-10-17T19:18:16.424 DEBUG:boto:Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.425 DEBUG:boto:Data: 2023-10-17T19:18:16.425 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.425 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.426 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.426 DEBUG:boto:Params: {} 2023-10-17T19:18:16.427 DEBUG:boto:Token: None 2023-10-17T19:18:16.427 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.428 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a45b89e25a18a73-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.431 DEBUG:boto:b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.432 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.433 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.433 DEBUG:boto:path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.433 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.434 DEBUG:boto:Method: GET 2023-10-17T19:18:16.434 DEBUG:boto:Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:18:16.435 DEBUG:boto:Data: 2023-10-17T19:18:16.435 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.436 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.436 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.436 DEBUG:boto:Params: {} 2023-10-17T19:18:16.437 DEBUG:boto:Token: None 2023-10-17T19:18:16.437 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.438 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf2de6d7c080b1f-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.441 DEBUG:boto:b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:18:16.442 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.442 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.443 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.443 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.444 DEBUG:boto:Method: GET 2023-10-17T19:18:16.444 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.444 DEBUG:boto:Data: 2023-10-17T19:18:16.445 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.445 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.445 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.446 DEBUG:boto:Params: {} 2023-10-17T19:18:16.446 DEBUG:boto:Token: None 2023-10-17T19:18:16.447 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.447 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:18:16.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:18:16.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86deefcf1164cae-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.450 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.451 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.451 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.452 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.452 DEBUG:boto:Method: GET 2023-10-17T19:18:16.452 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.453 DEBUG:boto:Data: 2023-10-17T19:18:16.453 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.453 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.454 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.454 DEBUG:boto:Params: {} 2023-10-17T19:18:16.455 DEBUG:boto:Token: None 2023-10-17T19:18:16.455 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.456 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:18:16.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:18:16.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ccd11cbc39c596-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.459 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.459 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.460 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.460 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.460 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.461 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.461 DEBUG:boto:Data: 2023-10-17T19:18:16.461 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.462 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.462 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.463 DEBUG:boto:Params: {} 2023-10-17T19:18:16.463 DEBUG:boto:Token: None 2023-10-17T19:18:16.463 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.464 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg= 2023-10-17T19:18:16.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg=', 'Content-Length': '0'} 2023-10-17T19:18:16.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad9816d501c3868-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.467 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.467 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.467 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.468 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.468 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.468 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.469 DEBUG:boto:Data: 2023-10-17T19:18:16.469 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.470 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.470 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.470 DEBUG:boto:Params: {} 2023-10-17T19:18:16.471 DEBUG:boto:Token: None 2023-10-17T19:18:16.471 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.472 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg= 2023-10-17T19:18:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg=', 'Content-Length': '0'} 2023-10-17T19:18:16.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab8d859902515e6-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:18:16.475 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.475 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.476 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.476 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.477 DEBUG:boto:Method: GET 2023-10-17T19:18:16.477 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.477 DEBUG:boto:Data: 2023-10-17T19:18:16.478 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.478 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.478 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.479 DEBUG:boto:Params: {} 2023-10-17T19:18:16.479 DEBUG:boto:Token: None 2023-10-17T19:18:16.480 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.480 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:18:16.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:18:16.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a546f42116aa83-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.483 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.484 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.484 DEBUG:boto:path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.485 DEBUG:boto:auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.485 DEBUG:boto:Method: GET 2023-10-17T19:18:16.485 DEBUG:boto:Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.486 DEBUG:boto:Data: 2023-10-17T19:18:16.486 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.487 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.487 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.487 DEBUG:boto:Params: {} 2023-10-17T19:18:16.488 DEBUG:boto:Token: None 2023-10-17T19:18:16.488 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33?versionId=null 2023-10-17T19:18:16.489 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:18:16.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:18:16.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db5fb901655b756-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.492 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.492 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.492 DEBUG:boto:path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.493 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.493 DEBUG:boto:Method: GET 2023-10-17T19:18:16.494 DEBUG:boto:Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.494 DEBUG:boto:Data: 2023-10-17T19:18:16.494 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.495 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.495 DEBUG:boto:Params: {} 2023-10-17T19:18:16.496 DEBUG:boto:Token: None 2023-10-17T19:18:16.496 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfbe72aa21bddac7-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.542 DEBUG:boto:b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.543 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.543 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.544 DEBUG:boto:path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.544 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.544 DEBUG:boto:Method: GET 2023-10-17T19:18:16.545 DEBUG:boto:Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.545 DEBUG:boto:Data: 2023-10-17T19:18:16.545 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.546 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.546 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.547 DEBUG:boto:Params: {} 2023-10-17T19:18:16.547 DEBUG:boto:Token: None 2023-10-17T19:18:16.547 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.548 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad111c8dc8bbb81-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.552 DEBUG:boto:b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.552 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.553 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.553 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.553 DEBUG:boto:Path: /hbmuog-64/obj33 2023-10-17T19:18:16.554 DEBUG:boto:Data: 2023-10-17T19:18:16.554 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.554 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.555 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.555 DEBUG:boto:Params: {} 2023-10-17T19:18:16.556 DEBUG:boto:Token: None 2023-10-17T19:18:16.556 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33 2023-10-17T19:18:16.556 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A= 2023-10-17T19:18:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A=', 'Content-Length': '0'} 2023-10-17T19:18:16.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ce630e1e5c8bf4-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.559 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.560 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.560 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.561 DEBUG:boto:Path: /hbmuog-64/obj33 2023-10-17T19:18:16.561 DEBUG:boto:Data: 2023-10-17T19:18:16.561 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.562 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.562 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.562 DEBUG:boto:Params: {} 2023-10-17T19:18:16.563 DEBUG:boto:Token: None 2023-10-17T19:18:16.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33 2023-10-17T19:18:16.564 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A= 2023-10-17T19:18:16.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A=', 'Content-Length': '0'} 2023-10-17T19:18:16.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e73c0b1e110f4ffe-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-17T19:18:16.567 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.567 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.568 DEBUG:boto:Method: GET 2023-10-17T19:18:16.568 DEBUG:boto:Path: /hbmuog-64/obj33 2023-10-17T19:18:16.568 DEBUG:boto:Data: 2023-10-17T19:18:16.569 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.569 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.570 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.570 DEBUG:boto:Params: {} 2023-10-17T19:18:16.570 DEBUG:boto:Token: None 2023-10-17T19:18:16.571 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33 2023-10-17T19:18:16.571 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o= 2023-10-17T19:18:16.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o=', 'Content-Length': '0'} 2023-10-17T19:18:16.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca68f9b03a44007-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.574 DEBUG:boto:path=/hbmuog-64/obj33 2023-10-17T19:18:16.575 DEBUG:boto:auth_path=/hbmuog-64/obj33 2023-10-17T19:18:16.575 DEBUG:boto:Method: GET 2023-10-17T19:18:16.575 DEBUG:boto:Path: /hbmuog-64/obj33 2023-10-17T19:18:16.576 DEBUG:boto:Data: 2023-10-17T19:18:16.576 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.576 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.577 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.577 DEBUG:boto:Params: {} 2023-10-17T19:18:16.578 DEBUG:boto:Token: None 2023-10-17T19:18:16.578 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj33 2023-10-17T19:18:16.578 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o= 2023-10-17T19:18:16.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o=', 'Content-Length': '0'} 2023-10-17T19:18:16.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006465aae08bd5f59e-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.622 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.623 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.623 DEBUG:boto:path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.623 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.624 DEBUG:boto:Method: GET 2023-10-17T19:18:16.624 DEBUG:boto:Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.625 DEBUG:boto:Data: 2023-10-17T19:18:16.625 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.625 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.626 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.626 DEBUG:boto:Params: {} 2023-10-17T19:18:16.627 DEBUG:boto:Token: None 2023-10-17T19:18:16.627 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.628 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c6e150b6c984cd1-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.631 DEBUG:boto:b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.632 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.632 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.633 DEBUG:boto:path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.633 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.633 DEBUG:boto:Method: GET 2023-10-17T19:18:16.634 DEBUG:boto:Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:18:16.634 DEBUG:boto:Data: 2023-10-17T19:18:16.635 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.635 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.636 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.636 DEBUG:boto:Params: {} 2023-10-17T19:18:16.636 DEBUG:boto:Token: None 2023-10-17T19:18:16.637 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.637 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55ee0617a4187e9-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.641 DEBUG:boto:b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:18:16.642 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.642 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.643 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.643 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.643 DEBUG:boto:Method: GET 2023-10-17T19:18:16.644 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.644 DEBUG:boto:Data: 2023-10-17T19:18:16.645 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.645 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.645 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.646 DEBUG:boto:Params: {} 2023-10-17T19:18:16.646 DEBUG:boto:Token: None 2023-10-17T19:18:16.647 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.647 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:18:16.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:18:16.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624b8a0d917c7d78-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.690 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.691 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.691 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.692 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.692 DEBUG:boto:Method: GET 2023-10-17T19:18:16.692 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.693 DEBUG:boto:Data: 2023-10-17T19:18:16.693 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.694 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.694 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.694 DEBUG:boto:Params: {} 2023-10-17T19:18:16.695 DEBUG:boto:Token: None 2023-10-17T19:18:16.695 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.696 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:18:16.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:18:16.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e0fb8d59c2204e1-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.699 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.699 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.700 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.700 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.700 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.701 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.701 DEBUG:boto:Data: 2023-10-17T19:18:16.702 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.702 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.702 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.703 DEBUG:boto:Params: {} 2023-10-17T19:18:16.703 DEBUG:boto:Token: None 2023-10-17T19:18:16.704 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.704 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w= 2023-10-17T19:18:16.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w=', 'Content-Length': '0'} 2023-10-17T19:18:16.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0295afd75abe64-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.707 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.707 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.708 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.708 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.709 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.709 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.709 DEBUG:boto:Data: 2023-10-17T19:18:16.710 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.711 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.711 DEBUG:boto:Params: {} 2023-10-17T19:18:16.711 DEBUG:boto:Token: None 2023-10-17T19:18:16.712 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w= 2023-10-17T19:18:16.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w=', 'Content-Length': '0'} 2023-10-17T19:18:16.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89391769e1b600a-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:18:16.716 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.716 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.716 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.717 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.717 DEBUG:boto:Method: GET 2023-10-17T19:18:16.718 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.718 DEBUG:boto:Data: 2023-10-17T19:18:16.718 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.719 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.719 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.720 DEBUG:boto:Params: {} 2023-10-17T19:18:16.720 DEBUG:boto:Token: None 2023-10-17T19:18:16.721 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.721 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:18:16.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:18:16.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddbf17dee2a185b-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.724 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.725 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.725 DEBUG:boto:path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.726 DEBUG:boto:auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.726 DEBUG:boto:Method: GET 2023-10-17T19:18:16.726 DEBUG:boto:Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.727 DEBUG:boto:Data: 2023-10-17T19:18:16.727 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.728 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.728 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.728 DEBUG:boto:Params: {} 2023-10-17T19:18:16.729 DEBUG:boto:Token: None 2023-10-17T19:18:16.729 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44?versionId=null 2023-10-17T19:18:16.730 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:18:16.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:18:16.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324e0dbffb99f065-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.774 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.775 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.775 DEBUG:boto:path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.776 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.776 DEBUG:boto:Method: GET 2023-10-17T19:18:16.776 DEBUG:boto:Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.777 DEBUG:boto:Data: 2023-10-17T19:18:16.777 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.778 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.778 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.779 DEBUG:boto:Params: {} 2023-10-17T19:18:16.779 DEBUG:boto:Token: None 2023-10-17T19:18:16.780 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.780 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c296033b03284b2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.784 DEBUG:boto:b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.784 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.785 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.785 DEBUG:boto:path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.786 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.786 DEBUG:boto:Method: GET 2023-10-17T19:18:16.787 DEBUG:boto:Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.787 DEBUG:boto:Data: 2023-10-17T19:18:16.787 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.788 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.789 DEBUG:boto:Params: {} 2023-10-17T19:18:16.789 DEBUG:boto:Token: None 2023-10-17T19:18:16.790 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318887b12ee78244-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.794 DEBUG:boto:b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.794 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.795 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.795 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.795 DEBUG:boto:Path: /hbmuog-64/obj44 2023-10-17T19:18:16.796 DEBUG:boto:Data: 2023-10-17T19:18:16.796 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.797 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.797 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.797 DEBUG:boto:Params: {} 2023-10-17T19:18:16.798 DEBUG:boto:Token: None 2023-10-17T19:18:16.798 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44 2023-10-17T19:18:16.799 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg= 2023-10-17T19:18:16.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg=', 'Content-Length': '0'} 2023-10-17T19:18:16.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004654b8392ec3ba36-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.802 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.802 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.803 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.803 DEBUG:boto:Path: /hbmuog-64/obj44 2023-10-17T19:18:16.803 DEBUG:boto:Data: 2023-10-17T19:18:16.804 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.804 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.804 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.805 DEBUG:boto:Params: {} 2023-10-17T19:18:16.805 DEBUG:boto:Token: None 2023-10-17T19:18:16.806 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44 2023-10-17T19:18:16.806 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg= 2023-10-17T19:18:16.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg=', 'Content-Length': '0'} 2023-10-17T19:18:16.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da175fb100bcac5b-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-17T19:18:16.810 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.810 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.811 DEBUG:boto:Method: GET 2023-10-17T19:18:16.811 DEBUG:boto:Path: /hbmuog-64/obj44 2023-10-17T19:18:16.811 DEBUG:boto:Data: 2023-10-17T19:18:16.812 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.813 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.813 DEBUG:boto:Params: {} 2023-10-17T19:18:16.813 DEBUG:boto:Token: None 2023-10-17T19:18:16.814 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44 2023-10-17T19:18:16.814 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk= 2023-10-17T19:18:16.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk=', 'Content-Length': '0'} 2023-10-17T19:18:16.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb4b73143fd03a2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.858 DEBUG:boto:path=/hbmuog-64/obj44 2023-10-17T19:18:16.859 DEBUG:boto:auth_path=/hbmuog-64/obj44 2023-10-17T19:18:16.859 DEBUG:boto:Method: GET 2023-10-17T19:18:16.860 DEBUG:boto:Path: /hbmuog-64/obj44 2023-10-17T19:18:16.860 DEBUG:boto:Data: 2023-10-17T19:18:16.861 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.861 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.862 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.862 DEBUG:boto:Params: {} 2023-10-17T19:18:16.862 DEBUG:boto:Token: None 2023-10-17T19:18:16.863 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj44 2023-10-17T19:18:16.863 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk= 2023-10-17T19:18:16.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk=', 'Content-Length': '0'} 2023-10-17T19:18:16.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22e7b99cab35311-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.867 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.867 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.867 DEBUG:boto:path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.868 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.868 DEBUG:boto:Method: GET 2023-10-17T19:18:16.869 DEBUG:boto:Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.869 DEBUG:boto:Data: 2023-10-17T19:18:16.869 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.870 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.870 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.871 DEBUG:boto:Params: {} 2023-10-17T19:18:16.871 DEBUG:boto:Token: None 2023-10-17T19:18:16.871 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.872 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6e42e0f81a41f7-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.876 DEBUG:boto:b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.876 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.877 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.877 DEBUG:boto:path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.878 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.878 DEBUG:boto:Method: GET 2023-10-17T19:18:16.879 DEBUG:boto:Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:18:16.879 DEBUG:boto:Data: 2023-10-17T19:18:16.879 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.880 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.880 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.881 DEBUG:boto:Params: {} 2023-10-17T19:18:16.881 DEBUG:boto:Token: None 2023-10-17T19:18:16.882 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.882 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec0ca246b2e809d-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.886 DEBUG:boto:b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:18:16.887 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.887 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.888 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.888 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.889 DEBUG:boto:Method: GET 2023-10-17T19:18:16.889 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.889 DEBUG:boto:Data: 2023-10-17T19:18:16.890 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.890 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.891 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.891 DEBUG:boto:Params: {} 2023-10-17T19:18:16.893 DEBUG:boto:Token: None 2023-10-17T19:18:16.894 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.894 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:18:16.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:18:16.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001036bbd031ee0c67-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.897 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.898 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.898 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.899 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.899 DEBUG:boto:Method: GET 2023-10-17T19:18:16.899 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.900 DEBUG:boto:Data: 2023-10-17T19:18:16.900 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.901 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.901 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.901 DEBUG:boto:Params: {} 2023-10-17T19:18:16.902 DEBUG:boto:Token: None 2023-10-17T19:18:16.902 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.903 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:18:16.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:18:16.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9353e1c484b06e-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.906 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.907 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.907 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.907 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.908 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.908 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.909 DEBUG:boto:Data: 2023-10-17T19:18:16.909 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.909 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.910 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.910 DEBUG:boto:Params: {} 2023-10-17T19:18:16.911 DEBUG:boto:Token: None 2023-10-17T19:18:16.911 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU= 2023-10-17T19:18:16.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU=', 'Content-Length': '0'} 2023-10-17T19:18:16.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056f3badd5b14bc50-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.915 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.915 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.916 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.916 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.916 DEBUG:boto:Method: HEAD 2023-10-17T19:18:16.917 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.917 DEBUG:boto:Data: 2023-10-17T19:18:16.918 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.918 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.919 DEBUG:boto:Params: {} 2023-10-17T19:18:16.919 DEBUG:boto:Token: None 2023-10-17T19:18:16.920 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU= 2023-10-17T19:18:16.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU=', 'Content-Length': '0'} 2023-10-17T19:18:16.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4e555838520a5a-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:18:16.923 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.924 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.924 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.925 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.925 DEBUG:boto:Method: GET 2023-10-17T19:18:16.925 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.926 DEBUG:boto:Data: 2023-10-17T19:18:16.926 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.927 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.927 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.928 DEBUG:boto:Params: {} 2023-10-17T19:18:16.928 DEBUG:boto:Token: None 2023-10-17T19:18:16.928 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.929 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:18:16.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:18:16.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e18ff04ddb9968e-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.974 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:16.975 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:16.975 DEBUG:boto:path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.976 DEBUG:boto:auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.976 DEBUG:boto:Method: GET 2023-10-17T19:18:16.976 DEBUG:boto:Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.977 DEBUG:boto:Data: 2023-10-17T19:18:16.977 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.978 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.978 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.979 DEBUG:boto:Params: {} 2023-10-17T19:18:16.979 DEBUG:boto:Token: None 2023-10-17T19:18:16.979 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/obj55?versionId=null 2023-10-17T19:18:16.980 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:18:16.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:18:16.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b6418d943b16d1-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.983 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.984 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.984 DEBUG:boto:path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.985 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.985 DEBUG:boto:Method: GET 2023-10-17T19:18:16.986 DEBUG:boto:Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.986 DEBUG:boto:Data: 2023-10-17T19:18:16.986 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.987 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:16.987 DEBUG:boto:Port: 8001 2023-10-17T19:18:16.988 DEBUG:boto:Params: {} 2023-10-17T19:18:16.988 DEBUG:boto:Token: None 2023-10-17T19:18:16.989 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.989 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:16.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331bd9d58685efef-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:16.993 DEBUG:boto:b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:16.994 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:16.994 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:16.994 DEBUG:boto:path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.995 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.995 DEBUG:boto:Method: GET 2023-10-17T19:18:16.995 DEBUG:boto:Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:18:16.996 DEBUG:boto:Data: 2023-10-17T19:18:16.996 DEBUG:boto:Headers: {} 2023-10-17T19:18:16.997 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:16.997 DEBUG:boto:Port: 8000 2023-10-17T19:18:16.997 DEBUG:boto:Params: {} 2023-10-17T19:18:16.998 DEBUG:boto:Token: None 2023-10-17T19:18:16.998 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:16 GMT /hbmuog-64/ 2023-10-17T19:18:16.999 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:18:16.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:18:17.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4af6bf93f055ab-00652eddf9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.002 DEBUG:boto:b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:17.003 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:17.003 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:17.004 DEBUG:boto:Method: HEAD 2023-10-17T19:18:17.004 DEBUG:boto:Path: /hbmuog-64/obj55 2023-10-17T19:18:17.005 DEBUG:boto:Data: 2023-10-17T19:18:17.005 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.005 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:17.006 DEBUG:boto:Port: 8001 2023-10-17T19:18:17.006 DEBUG:boto:Params: {} 2023-10-17T19:18:17.007 DEBUG:boto:Token: None 2023-10-17T19:18:17.007 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/obj55 2023-10-17T19:18:17.007 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00= 2023-10-17T19:18:17.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00=', 'Content-Length': '0'} 2023-10-17T19:18:17.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d23111fbde76752-00652eddf9-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.010 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:17.011 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:17.011 DEBUG:boto:Method: HEAD 2023-10-17T19:18:17.011 DEBUG:boto:Path: /hbmuog-64/obj55 2023-10-17T19:18:17.012 DEBUG:boto:Data: 2023-10-17T19:18:17.012 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.013 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:17.013 DEBUG:boto:Port: 8000 2023-10-17T19:18:17.013 DEBUG:boto:Params: {} 2023-10-17T19:18:17.014 DEBUG:boto:Token: None 2023-10-17T19:18:17.014 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/obj55 2023-10-17T19:18:17.015 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00= 2023-10-17T19:18:17.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00=', 'Content-Length': '0'} 2023-10-17T19:18:17.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae09af494381c5ce-00652eddf9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-17T19:18:17.018 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:17.018 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:17.019 DEBUG:boto:Method: GET 2023-10-17T19:18:17.019 DEBUG:boto:Path: /hbmuog-64/obj55 2023-10-17T19:18:17.020 DEBUG:boto:Data: 2023-10-17T19:18:17.020 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.020 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:17.021 DEBUG:boto:Port: 8001 2023-10-17T19:18:17.021 DEBUG:boto:Params: {} 2023-10-17T19:18:17.022 DEBUG:boto:Token: None 2023-10-17T19:18:17.022 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/obj55 2023-10-17T19:18:17.023 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw= 2023-10-17T19:18:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw=', 'Content-Length': '0'} 2023-10-17T19:18:17.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c645ca3d06c7b3e-00652eddf9-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.026 DEBUG:boto:path=/hbmuog-64/obj55 2023-10-17T19:18:17.026 DEBUG:boto:auth_path=/hbmuog-64/obj55 2023-10-17T19:18:17.027 DEBUG:boto:Method: GET 2023-10-17T19:18:17.027 DEBUG:boto:Path: /hbmuog-64/obj55 2023-10-17T19:18:17.027 DEBUG:boto:Data: 2023-10-17T19:18:17.028 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.028 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:17.028 DEBUG:boto:Port: 8000 2023-10-17T19:18:17.029 DEBUG:boto:Params: {} 2023-10-17T19:18:17.029 DEBUG:boto:Token: None 2023-10-17T19:18:17.030 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/obj55 2023-10-17T19:18:17.030 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw= 2023-10-17T19:18:17.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw=', 'Content-Length': '0'} 2023-10-17T19:18:17.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4278dbfb0fbb404-00652eddf9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.074 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:17.075 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:17.075 DEBUG:boto:path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.076 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.076 DEBUG:boto:Method: GET 2023-10-17T19:18:17.077 DEBUG:boto:Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.077 DEBUG:boto:Data: 2023-10-17T19:18:17.077 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.078 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:17.078 DEBUG:boto:Port: 8001 2023-10-17T19:18:17.079 DEBUG:boto:Params: {} 2023-10-17T19:18:17.079 DEBUG:boto:Token: None 2023-10-17T19:18:17.080 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/ 2023-10-17T19:18:17.080 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A= 2023-10-17T19:18:17.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A=', 'Content-Length': '0'} 2023-10-17T19:18:17.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1f20117a6aa330-00652eddf9-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.084 DEBUG:boto:b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:17.085 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:17.085 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:17.086 DEBUG:boto:path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.086 DEBUG:boto:auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.087 DEBUG:boto:Method: GET 2023-10-17T19:18:17.087 DEBUG:boto:Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:18:17.087 DEBUG:boto:Data: 2023-10-17T19:18:17.088 DEBUG:boto:Headers: {} 2023-10-17T19:18:17.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:17.089 DEBUG:boto:Port: 8000 2023-10-17T19:18:17.089 DEBUG:boto:Params: {} 2023-10-17T19:18:17.089 DEBUG:boto:Token: None 2023-10-17T19:18:17.090 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:18:17 GMT /hbmuog-64/ 2023-10-17T19:18:17.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A= 2023-10-17T19:18:17.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A=', 'Content-Length': '0'} 2023-10-17T19:18:17.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedfba94a92a05a4-00652eddf9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:17.094 DEBUG:boto:b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:18:17.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hbmuog-64 zones={a2, a1} 2023-10-17T19:18:17.095 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T19:18:17.096 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:18:17.096 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:18:17.095+0000 7f47e63fd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 169349) UID: 0 2023-10-17T19:18:17.097 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:18:17.095+0000 7f47ea6667c0 -1 shutting down 2023-10-17T19:18:17.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:18:23.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2023-10-17T19:18:23.194 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T19:18:23.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.196 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:23.258 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:23.258 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:23.310 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:23.311 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:23.314 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 5 2023-10-17T19:18:23.354 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.355 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.355 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.356 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.356 DEBUG:boto:Data: 2023-10-17T19:18:23.356 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.357 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.357 DEBUG:boto:Params: {} 2023-10-17T19:18:23.358 DEBUG:boto:Token: None 2023-10-17T19:18:23.358 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef392578e4b5877e-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.363 DEBUG:boto:path=/hbmuog-64/obj17 2023-10-17T19:18:23.363 DEBUG:boto:auth_path=/hbmuog-64/obj17 2023-10-17T19:18:23.364 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.364 DEBUG:boto:Path: /hbmuog-64/obj17 2023-10-17T19:18:23.364 DEBUG:boto:Data: 2023-10-17T19:18:23.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.366 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.366 DEBUG:boto:Params: {} 2023-10-17T19:18:23.367 DEBUG:boto:Token: None 2023-10-17T19:18:23.367 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj17 2023-10-17T19:18:23.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tcZk/uqAQctBvZllr6aJbiCwnzw= 2023-10-17T19:18:23.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tcZk/uqAQctBvZllr6aJbiCwnzw='} 2023-10-17T19:18:23.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab32273926943e44-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.375 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.375 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.376 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.376 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.377 DEBUG:boto:Data: 2023-10-17T19:18:23.377 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.377 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.378 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.378 DEBUG:boto:Params: {} 2023-10-17T19:18:23.379 DEBUG:boto:Token: None 2023-10-17T19:18:23.379 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.380 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da735ff52f77ce07-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.384 DEBUG:boto:path=/hbmuog-64/obj34 2023-10-17T19:18:23.384 DEBUG:boto:auth_path=/hbmuog-64/obj34 2023-10-17T19:18:23.384 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.385 DEBUG:boto:Path: /hbmuog-64/obj34 2023-10-17T19:18:23.385 DEBUG:boto:Data: 2023-10-17T19:18:23.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.386 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.386 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.387 DEBUG:boto:Params: {} 2023-10-17T19:18:23.388 DEBUG:boto:Token: None 2023-10-17T19:18:23.388 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj34 2023-10-17T19:18:23.389 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dAUjsJGEf33SA/BLVbWeWvIxFBs= 2023-10-17T19:18:23.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dAUjsJGEf33SA/BLVbWeWvIxFBs='} 2023-10-17T19:18:23.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074d5714dfeaf6aa1-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.435 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.436 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.436 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.436 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.437 DEBUG:boto:Data: 2023-10-17T19:18:23.437 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.438 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.438 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.439 DEBUG:boto:Params: {} 2023-10-17T19:18:23.439 DEBUG:boto:Token: None 2023-10-17T19:18:23.440 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.440 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4953475cd0ff0a-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.444 DEBUG:boto:path=/hbmuog-64/obj51 2023-10-17T19:18:23.444 DEBUG:boto:auth_path=/hbmuog-64/obj51 2023-10-17T19:18:23.445 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.445 DEBUG:boto:Path: /hbmuog-64/obj51 2023-10-17T19:18:23.445 DEBUG:boto:Data: 2023-10-17T19:18:23.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.446 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.447 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.447 DEBUG:boto:Params: {} 2023-10-17T19:18:23.448 DEBUG:boto:Token: None 2023-10-17T19:18:23.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj51 2023-10-17T19:18:23.448 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BA+gHP4oZ2Cm/hqnRCffRaUf+HA= 2023-10-17T19:18:23.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BA+gHP4oZ2Cm/hqnRCffRaUf+HA='} 2023-10-17T19:18:23.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096d2bfd271169098-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.455 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.455 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.455 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.456 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.456 DEBUG:boto:Data: 2023-10-17T19:18:23.457 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.457 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.457 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.458 DEBUG:boto:Params: {} 2023-10-17T19:18:23.458 DEBUG:boto:Token: None 2023-10-17T19:18:23.459 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.459 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a1b2bce33471aaf-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.463 DEBUG:boto:path=/hbmuog-64/obj68 2023-10-17T19:18:23.463 DEBUG:boto:auth_path=/hbmuog-64/obj68 2023-10-17T19:18:23.464 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.464 DEBUG:boto:Path: /hbmuog-64/obj68 2023-10-17T19:18:23.465 DEBUG:boto:Data: 2023-10-17T19:18:23.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.465 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.466 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.466 DEBUG:boto:Params: {} 2023-10-17T19:18:23.467 DEBUG:boto:Token: None 2023-10-17T19:18:23.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj68 2023-10-17T19:18:23.468 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4://TUWdmMa3DOqKBw2WWbMG4w5kI= 2023-10-17T19:18:23.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4://TUWdmMa3DOqKBw2WWbMG4w5kI='} 2023-10-17T19:18:23.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098e0192b5daaf16d-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.515 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.516 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.516 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.516 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.517 DEBUG:boto:Data: 2023-10-17T19:18:23.517 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.518 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.518 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.518 DEBUG:boto:Params: {} 2023-10-17T19:18:23.519 DEBUG:boto:Token: None 2023-10-17T19:18:23.519 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.520 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b71ae44fb0b02ed-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.524 DEBUG:boto:path=/hbmuog-64/obj85 2023-10-17T19:18:23.524 DEBUG:boto:auth_path=/hbmuog-64/obj85 2023-10-17T19:18:23.524 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.525 DEBUG:boto:Path: /hbmuog-64/obj85 2023-10-17T19:18:23.525 DEBUG:boto:Data: 2023-10-17T19:18:23.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.526 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.527 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.527 DEBUG:boto:Params: {} 2023-10-17T19:18:23.529 DEBUG:boto:Token: None 2023-10-17T19:18:23.529 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj85 2023-10-17T19:18:23.530 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uaKguJVSi2lOtIAx3cRlUbOe00E= 2023-10-17T19:18:23.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uaKguJVSi2lOtIAx3cRlUbOe00E='} 2023-10-17T19:18:23.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6ef587d5b7a959c-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.538 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:23.593 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:23.594 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:23.638 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:23.638 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:23.644 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 10 2023-10-17T19:18:23.689 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.690 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.690 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.690 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.691 DEBUG:boto:Data: 2023-10-17T19:18:23.691 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.692 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.692 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.692 DEBUG:boto:Params: {} 2023-10-17T19:18:23.693 DEBUG:boto:Token: None 2023-10-17T19:18:23.693 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.694 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001747b5686c8f433c-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.697 DEBUG:boto:path=/hbmuog-64/obj19 2023-10-17T19:18:23.698 DEBUG:boto:auth_path=/hbmuog-64/obj19 2023-10-17T19:18:23.698 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.698 DEBUG:boto:Path: /hbmuog-64/obj19 2023-10-17T19:18:23.699 DEBUG:boto:Data: 2023-10-17T19:18:23.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.700 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.700 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.700 DEBUG:boto:Params: {} 2023-10-17T19:18:23.701 DEBUG:boto:Token: None 2023-10-17T19:18:23.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj19 2023-10-17T19:18:23.702 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F6H1i2Z6COrPSeWoCnhQB1kHniI= 2023-10-17T19:18:23.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6H1i2Z6COrPSeWoCnhQB1kHniI='} 2023-10-17T19:18:23.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013fb6f1f708b93c0-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.708 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.708 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.709 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.709 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.709 DEBUG:boto:Data: 2023-10-17T19:18:23.710 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.710 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.711 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.711 DEBUG:boto:Params: {} 2023-10-17T19:18:23.711 DEBUG:boto:Token: None 2023-10-17T19:18:23.712 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.712 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48c03ac565d1bbf-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.715 DEBUG:boto:path=/hbmuog-64/obj38 2023-10-17T19:18:23.716 DEBUG:boto:auth_path=/hbmuog-64/obj38 2023-10-17T19:18:23.716 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.717 DEBUG:boto:Path: /hbmuog-64/obj38 2023-10-17T19:18:23.717 DEBUG:boto:Data: 2023-10-17T19:18:23.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.718 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.718 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.719 DEBUG:boto:Params: {} 2023-10-17T19:18:23.719 DEBUG:boto:Token: None 2023-10-17T19:18:23.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj38 2023-10-17T19:18:23.720 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RGn7C6DjI+6jMZ4Dzlpe32vheVk= 2023-10-17T19:18:23.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RGn7C6DjI+6jMZ4Dzlpe32vheVk='} 2023-10-17T19:18:23.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000300d50e734404c5b-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.763 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.764 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.764 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.764 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.765 DEBUG:boto:Data: 2023-10-17T19:18:23.765 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.766 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.767 DEBUG:boto:Params: {} 2023-10-17T19:18:23.767 DEBUG:boto:Token: None 2023-10-17T19:18:23.768 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.768 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b0d60f2aa7cc9d4-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.772 DEBUG:boto:path=/hbmuog-64/obj57 2023-10-17T19:18:23.772 DEBUG:boto:auth_path=/hbmuog-64/obj57 2023-10-17T19:18:23.773 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.773 DEBUG:boto:Path: /hbmuog-64/obj57 2023-10-17T19:18:23.773 DEBUG:boto:Data: 2023-10-17T19:18:23.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.774 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.775 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.775 DEBUG:boto:Params: {} 2023-10-17T19:18:23.776 DEBUG:boto:Token: None 2023-10-17T19:18:23.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj57 2023-10-17T19:18:23.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/JJRtjpJU9cjpXDF7AqiT48/15Q= 2023-10-17T19:18:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/JJRtjpJU9cjpXDF7AqiT48/15Q='} 2023-10-17T19:18:23.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f7684c0de32dd5d-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.782 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.782 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.782 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.783 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.783 DEBUG:boto:Data: 2023-10-17T19:18:23.784 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.784 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.785 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.785 DEBUG:boto:Params: {} 2023-10-17T19:18:23.786 DEBUG:boto:Token: None 2023-10-17T19:18:23.786 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.787 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db814fdd1f42e02c-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.791 DEBUG:boto:path=/hbmuog-64/obj76 2023-10-17T19:18:23.791 DEBUG:boto:auth_path=/hbmuog-64/obj76 2023-10-17T19:18:23.792 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.792 DEBUG:boto:Path: /hbmuog-64/obj76 2023-10-17T19:18:23.792 DEBUG:boto:Data: 2023-10-17T19:18:23.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.793 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.793 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.794 DEBUG:boto:Params: {} 2023-10-17T19:18:23.794 DEBUG:boto:Token: None 2023-10-17T19:18:23.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj76 2023-10-17T19:18:23.795 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+ZIp9xzy60td4YbUCxuLC7WGwk8= 2023-10-17T19:18:23.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+ZIp9xzy60td4YbUCxuLC7WGwk8='} 2023-10-17T19:18:23.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037ada24ef240143e-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.839 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:23.839 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:23.840 DEBUG:boto:Method: HEAD 2023-10-17T19:18:23.840 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:23.841 DEBUG:boto:Data: 2023-10-17T19:18:23.841 DEBUG:boto:Headers: {} 2023-10-17T19:18:23.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.842 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.842 DEBUG:boto:Params: {} 2023-10-17T19:18:23.843 DEBUG:boto:Token: None 2023-10-17T19:18:23.843 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/ 2023-10-17T19:18:23.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:18:23.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:18:23.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000788cee9af6d80db3-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.848 DEBUG:boto:path=/hbmuog-64/obj95 2023-10-17T19:18:23.848 DEBUG:boto:auth_path=/hbmuog-64/obj95 2023-10-17T19:18:23.849 DEBUG:boto:Method: PUT 2023-10-17T19:18:23.849 DEBUG:boto:Path: /hbmuog-64/obj95 2023-10-17T19:18:23.850 DEBUG:boto:Data: 2023-10-17T19:18:23.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:23.851 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:23.851 DEBUG:boto:Port: 8000 2023-10-17T19:18:23.851 DEBUG:boto:Params: {} 2023-10-17T19:18:23.852 DEBUG:boto:Token: None 2023-10-17T19:18:23.852 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:23 GMT /hbmuog-64/obj95 2023-10-17T19:18:23.853 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:k0CzdsHpn0H5PDEfWEcUaf/9jig= 2023-10-17T19:18:23.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0CzdsHpn0H5PDEfWEcUaf/9jig='} 2023-10-17T19:18:23.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bf3aee9e6f2fe05-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:23.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:23.859 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:23.959 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:23.960 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 15 2023-10-17T19:18:24.013 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.013 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.014 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.014 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.015 DEBUG:boto:Data: 2023-10-17T19:18:24.015 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.015 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.016 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.016 DEBUG:boto:Params: {} 2023-10-17T19:18:24.017 DEBUG:boto:Token: None 2023-10-17T19:18:24.017 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.017 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065725bfbd821f97f-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.022 DEBUG:boto:path=/hbmuog-64/obj23 2023-10-17T19:18:24.022 DEBUG:boto:auth_path=/hbmuog-64/obj23 2023-10-17T19:18:24.022 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.023 DEBUG:boto:Path: /hbmuog-64/obj23 2023-10-17T19:18:24.023 DEBUG:boto:Data: 2023-10-17T19:18:24.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.024 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.025 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.025 DEBUG:boto:Params: {} 2023-10-17T19:18:24.025 DEBUG:boto:Token: None 2023-10-17T19:18:24.026 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj23 2023-10-17T19:18:24.026 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:/PzHQM7QtGNgvlUpmtV57Svqwy4= 2023-10-17T19:18:24.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/PzHQM7QtGNgvlUpmtV57Svqwy4='} 2023-10-17T19:18:24.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fbeda7376f5fac0-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.032 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.033 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.033 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.033 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.034 DEBUG:boto:Data: 2023-10-17T19:18:24.034 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.035 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.035 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.035 DEBUG:boto:Params: {} 2023-10-17T19:18:24.036 DEBUG:boto:Token: None 2023-10-17T19:18:24.036 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.036 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7201acfbd03adb0-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.040 DEBUG:boto:path=/hbmuog-64/obj46 2023-10-17T19:18:24.040 DEBUG:boto:auth_path=/hbmuog-64/obj46 2023-10-17T19:18:24.041 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.041 DEBUG:boto:Path: /hbmuog-64/obj46 2023-10-17T19:18:24.041 DEBUG:boto:Data: 2023-10-17T19:18:24.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.042 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.043 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.043 DEBUG:boto:Params: {} 2023-10-17T19:18:24.043 DEBUG:boto:Token: None 2023-10-17T19:18:24.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj46 2023-10-17T19:18:24.044 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:trTcHmdE+wC9CrgWPLWD4X9lijQ= 2023-10-17T19:18:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:trTcHmdE+wC9CrgWPLWD4X9lijQ='} 2023-10-17T19:18:24.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000232a9887f65e56ae-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.087 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.087 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.088 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.088 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.088 DEBUG:boto:Data: 2023-10-17T19:18:24.089 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.089 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.089 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.090 DEBUG:boto:Params: {} 2023-10-17T19:18:24.090 DEBUG:boto:Token: None 2023-10-17T19:18:24.091 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.091 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c09658949aecfc8-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.095 DEBUG:boto:path=/hbmuog-64/obj69 2023-10-17T19:18:24.096 DEBUG:boto:auth_path=/hbmuog-64/obj69 2023-10-17T19:18:24.096 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.096 DEBUG:boto:Path: /hbmuog-64/obj69 2023-10-17T19:18:24.097 DEBUG:boto:Data: 2023-10-17T19:18:24.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.098 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.098 DEBUG:boto:Params: {} 2023-10-17T19:18:24.099 DEBUG:boto:Token: None 2023-10-17T19:18:24.099 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj69 2023-10-17T19:18:24.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3MUselRa1VMCs3nvh3/SntbtOG4= 2023-10-17T19:18:24.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3MUselRa1VMCs3nvh3/SntbtOG4='} 2023-10-17T19:18:24.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047d35870e95ab3ef-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.105 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.106 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.106 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.107 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.107 DEBUG:boto:Data: 2023-10-17T19:18:24.107 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.108 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.108 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.109 DEBUG:boto:Params: {} 2023-10-17T19:18:24.109 DEBUG:boto:Token: None 2023-10-17T19:18:24.110 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.111 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092633e8119ae75fc-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.114 DEBUG:boto:path=/hbmuog-64/obj92 2023-10-17T19:18:24.114 DEBUG:boto:auth_path=/hbmuog-64/obj92 2023-10-17T19:18:24.115 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.115 DEBUG:boto:Path: /hbmuog-64/obj92 2023-10-17T19:18:24.116 DEBUG:boto:Data: 2023-10-17T19:18:24.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.116 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.117 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.117 DEBUG:boto:Params: {} 2023-10-17T19:18:24.117 DEBUG:boto:Token: None 2023-10-17T19:18:24.118 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj92 2023-10-17T19:18:24.118 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nL97bv00LTOD4/FA5cuLPQd0jo0= 2023-10-17T19:18:24.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nL97bv00LTOD4/FA5cuLPQd0jo0='} 2023-10-17T19:18:24.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd0adc21b3d346b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.167 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.167 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.168 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.168 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.168 DEBUG:boto:Data: 2023-10-17T19:18:24.169 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.169 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.169 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.170 DEBUG:boto:Params: {} 2023-10-17T19:18:24.170 DEBUG:boto:Token: None 2023-10-17T19:18:24.171 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.171 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073ac3ac611de0e20-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.175 DEBUG:boto:path=/hbmuog-64/obj115 2023-10-17T19:18:24.175 DEBUG:boto:auth_path=/hbmuog-64/obj115 2023-10-17T19:18:24.176 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.176 DEBUG:boto:Path: /hbmuog-64/obj115 2023-10-17T19:18:24.176 DEBUG:boto:Data: 2023-10-17T19:18:24.177 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.177 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.178 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.178 DEBUG:boto:Params: {} 2023-10-17T19:18:24.178 DEBUG:boto:Token: None 2023-10-17T19:18:24.179 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj115 2023-10-17T19:18:24.179 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8D0mFah4IhNX5Y/0n7XbHm8zwns= 2023-10-17T19:18:24.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8D0mFah4IhNX5Y/0n7XbHm8zwns='} 2023-10-17T19:18:24.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b0eabf03439d05d-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.187 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:24.241 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:24.242 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:24.290 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:24.290 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:24.298 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 20 2023-10-17T19:18:24.354 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.355 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.355 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.356 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.356 DEBUG:boto:Data: 2023-10-17T19:18:24.356 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.357 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.357 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.357 DEBUG:boto:Params: {} 2023-10-17T19:18:24.358 DEBUG:boto:Token: None 2023-10-17T19:18:24.358 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.359 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3f49b219019b18c-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.363 DEBUG:boto:path=/hbmuog-64/obj29 2023-10-17T19:18:24.363 DEBUG:boto:auth_path=/hbmuog-64/obj29 2023-10-17T19:18:24.364 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.364 DEBUG:boto:Path: /hbmuog-64/obj29 2023-10-17T19:18:24.365 DEBUG:boto:Data: 2023-10-17T19:18:24.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.365 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.366 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.366 DEBUG:boto:Params: {} 2023-10-17T19:18:24.366 DEBUG:boto:Token: None 2023-10-17T19:18:24.367 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj29 2023-10-17T19:18:24.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oGc8v5zgpmjokQSyD21ziIsVF+U= 2023-10-17T19:18:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oGc8v5zgpmjokQSyD21ziIsVF+U='} 2023-10-17T19:18:24.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da9a2ef3169b9d9b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.411 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.412 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.412 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.412 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.413 DEBUG:boto:Data: 2023-10-17T19:18:24.413 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.413 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.414 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.414 DEBUG:boto:Params: {} 2023-10-17T19:18:24.415 DEBUG:boto:Token: None 2023-10-17T19:18:24.415 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.416 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d1a906081918a15-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.420 DEBUG:boto:path=/hbmuog-64/obj58 2023-10-17T19:18:24.420 DEBUG:boto:auth_path=/hbmuog-64/obj58 2023-10-17T19:18:24.421 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.421 DEBUG:boto:Path: /hbmuog-64/obj58 2023-10-17T19:18:24.421 DEBUG:boto:Data: 2023-10-17T19:18:24.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.422 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.423 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.423 DEBUG:boto:Params: {} 2023-10-17T19:18:24.423 DEBUG:boto:Token: None 2023-10-17T19:18:24.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj58 2023-10-17T19:18:24.424 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nksdvK9gkLVH1fStdNxFYOLJxe8= 2023-10-17T19:18:24.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nksdvK9gkLVH1fStdNxFYOLJxe8='} 2023-10-17T19:18:24.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008457d7e534e0ea7c-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.430 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.431 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.431 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.432 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.432 DEBUG:boto:Data: 2023-10-17T19:18:24.432 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.433 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.433 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.434 DEBUG:boto:Params: {} 2023-10-17T19:18:24.434 DEBUG:boto:Token: None 2023-10-17T19:18:24.434 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.435 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da38a72011f1d79-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.439 DEBUG:boto:path=/hbmuog-64/obj87 2023-10-17T19:18:24.439 DEBUG:boto:auth_path=/hbmuog-64/obj87 2023-10-17T19:18:24.440 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.440 DEBUG:boto:Path: /hbmuog-64/obj87 2023-10-17T19:18:24.440 DEBUG:boto:Data: 2023-10-17T19:18:24.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.441 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.441 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.442 DEBUG:boto:Params: {} 2023-10-17T19:18:24.442 DEBUG:boto:Token: None 2023-10-17T19:18:24.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj87 2023-10-17T19:18:24.443 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:awzZz2q8JtetkzQvAntvLYYgCIE= 2023-10-17T19:18:24.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:awzZz2q8JtetkzQvAntvLYYgCIE='} 2023-10-17T19:18:24.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8be247333a1752b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.487 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.487 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.488 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.488 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.488 DEBUG:boto:Data: 2023-10-17T19:18:24.489 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.489 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.489 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.490 DEBUG:boto:Params: {} 2023-10-17T19:18:24.490 DEBUG:boto:Token: None 2023-10-17T19:18:24.491 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.491 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002796066bb6a78979-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.495 DEBUG:boto:path=/hbmuog-64/obj116 2023-10-17T19:18:24.496 DEBUG:boto:auth_path=/hbmuog-64/obj116 2023-10-17T19:18:24.496 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.496 DEBUG:boto:Path: /hbmuog-64/obj116 2023-10-17T19:18:24.497 DEBUG:boto:Data: 2023-10-17T19:18:24.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.497 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.498 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.498 DEBUG:boto:Params: {} 2023-10-17T19:18:24.499 DEBUG:boto:Token: None 2023-10-17T19:18:24.499 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj116 2023-10-17T19:18:24.499 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:qX/9z97wmDQSZv8A1OwIONPViWE= 2023-10-17T19:18:24.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qX/9z97wmDQSZv8A1OwIONPViWE='} 2023-10-17T19:18:24.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcff96efac21e48d-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.505 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.506 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.506 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.506 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.507 DEBUG:boto:Data: 2023-10-17T19:18:24.507 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.508 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.508 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.508 DEBUG:boto:Params: {} 2023-10-17T19:18:24.509 DEBUG:boto:Token: None 2023-10-17T19:18:24.509 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.509 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82eea0bdf57294b-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.514 DEBUG:boto:path=/hbmuog-64/obj145 2023-10-17T19:18:24.514 DEBUG:boto:auth_path=/hbmuog-64/obj145 2023-10-17T19:18:24.515 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.515 DEBUG:boto:Path: /hbmuog-64/obj145 2023-10-17T19:18:24.515 DEBUG:boto:Data: 2023-10-17T19:18:24.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.516 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.516 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.517 DEBUG:boto:Params: {} 2023-10-17T19:18:24.517 DEBUG:boto:Token: None 2023-10-17T19:18:24.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj145 2023-10-17T19:18:24.518 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MxeJFVjoQe6vcjKMxIHbqZgZkj8= 2023-10-17T19:18:24.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MxeJFVjoQe6vcjKMxIHbqZgZkj8='} 2023-10-17T19:18:24.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d775050ca96c8a40-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.564 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:24.666 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:24.667 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:24.677 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 25 2023-10-17T19:18:24.735 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.735 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.736 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.736 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.736 DEBUG:boto:Data: 2023-10-17T19:18:24.737 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.737 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.737 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.738 DEBUG:boto:Params: {} 2023-10-17T19:18:24.738 DEBUG:boto:Token: None 2023-10-17T19:18:24.739 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.739 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d971562159d979b6-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.744 DEBUG:boto:path=/hbmuog-64/obj31 2023-10-17T19:18:24.744 DEBUG:boto:auth_path=/hbmuog-64/obj31 2023-10-17T19:18:24.744 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.745 DEBUG:boto:Path: /hbmuog-64/obj31 2023-10-17T19:18:24.745 DEBUG:boto:Data: 2023-10-17T19:18:24.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.746 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.746 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.747 DEBUG:boto:Params: {} 2023-10-17T19:18:24.747 DEBUG:boto:Token: None 2023-10-17T19:18:24.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj31 2023-10-17T19:18:24.748 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XB/5MQIo9tf1m3fSD8+HYPoDTGA= 2023-10-17T19:18:24.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XB/5MQIo9tf1m3fSD8+HYPoDTGA='} 2023-10-17T19:18:24.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf6823d2be727794-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.754 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.754 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.755 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.755 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.755 DEBUG:boto:Data: 2023-10-17T19:18:24.756 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.756 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.756 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.757 DEBUG:boto:Params: {} 2023-10-17T19:18:24.757 DEBUG:boto:Token: None 2023-10-17T19:18:24.757 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.758 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004532a25169b3abfc-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.762 DEBUG:boto:path=/hbmuog-64/obj62 2023-10-17T19:18:24.763 DEBUG:boto:auth_path=/hbmuog-64/obj62 2023-10-17T19:18:24.763 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.764 DEBUG:boto:Path: /hbmuog-64/obj62 2023-10-17T19:18:24.764 DEBUG:boto:Data: 2023-10-17T19:18:24.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.765 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.765 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.765 DEBUG:boto:Params: {} 2023-10-17T19:18:24.766 DEBUG:boto:Token: None 2023-10-17T19:18:24.766 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj62 2023-10-17T19:18:24.767 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Hq0ZK/6JmjCopHLT4MJL51TJEd4= 2023-10-17T19:18:24.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hq0ZK/6JmjCopHLT4MJL51TJEd4='} 2023-10-17T19:18:24.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee2f8eda2fdc9f77-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.772 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.773 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.773 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.773 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.774 DEBUG:boto:Data: 2023-10-17T19:18:24.774 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.775 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.775 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.775 DEBUG:boto:Params: {} 2023-10-17T19:18:24.776 DEBUG:boto:Token: None 2023-10-17T19:18:24.776 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.777 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f7f8e1dc0a550f-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.781 DEBUG:boto:path=/hbmuog-64/obj93 2023-10-17T19:18:24.781 DEBUG:boto:auth_path=/hbmuog-64/obj93 2023-10-17T19:18:24.782 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.782 DEBUG:boto:Path: /hbmuog-64/obj93 2023-10-17T19:18:24.782 DEBUG:boto:Data: 2023-10-17T19:18:24.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.783 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.784 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.784 DEBUG:boto:Params: {} 2023-10-17T19:18:24.784 DEBUG:boto:Token: None 2023-10-17T19:18:24.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj93 2023-10-17T19:18:24.785 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Awu4vSm7EZxYNByeg7It4z6/64g= 2023-10-17T19:18:24.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Awu4vSm7EZxYNByeg7It4z6/64g='} 2023-10-17T19:18:24.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bca5208f078ac88-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.831 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.831 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.832 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.832 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.832 DEBUG:boto:Data: 2023-10-17T19:18:24.833 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.833 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.833 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.834 DEBUG:boto:Params: {} 2023-10-17T19:18:24.834 DEBUG:boto:Token: None 2023-10-17T19:18:24.835 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.835 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bcd53269ae86309-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.839 DEBUG:boto:path=/hbmuog-64/obj124 2023-10-17T19:18:24.840 DEBUG:boto:auth_path=/hbmuog-64/obj124 2023-10-17T19:18:24.840 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.840 DEBUG:boto:Path: /hbmuog-64/obj124 2023-10-17T19:18:24.841 DEBUG:boto:Data: 2023-10-17T19:18:24.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.841 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.842 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.842 DEBUG:boto:Params: {} 2023-10-17T19:18:24.843 DEBUG:boto:Token: None 2023-10-17T19:18:24.843 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj124 2023-10-17T19:18:24.843 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ewLBn3DyQ33Oxztw/u/RRVgPw00= 2023-10-17T19:18:24.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewLBn3DyQ33Oxztw/u/RRVgPw00='} 2023-10-17T19:18:24.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073e4f840ab3a8343-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.849 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:24.850 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:24.850 DEBUG:boto:Method: HEAD 2023-10-17T19:18:24.851 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:24.851 DEBUG:boto:Data: 2023-10-17T19:18:24.851 DEBUG:boto:Headers: {} 2023-10-17T19:18:24.852 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.852 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.852 DEBUG:boto:Params: {} 2023-10-17T19:18:24.853 DEBUG:boto:Token: None 2023-10-17T19:18:24.853 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/ 2023-10-17T19:18:24.854 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:18:24.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:18:24.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000939d52acacca8e51-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.858 DEBUG:boto:path=/hbmuog-64/obj155 2023-10-17T19:18:24.858 DEBUG:boto:auth_path=/hbmuog-64/obj155 2023-10-17T19:18:24.859 DEBUG:boto:Method: PUT 2023-10-17T19:18:24.859 DEBUG:boto:Path: /hbmuog-64/obj155 2023-10-17T19:18:24.859 DEBUG:boto:Data: 2023-10-17T19:18:24.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:24.860 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:24.860 DEBUG:boto:Port: 8000 2023-10-17T19:18:24.861 DEBUG:boto:Params: {} 2023-10-17T19:18:24.861 DEBUG:boto:Token: None 2023-10-17T19:18:24.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:24 GMT /hbmuog-64/obj155 2023-10-17T19:18:24.862 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7AOylJ5J+/l2vtdD+DeAelunfvM= 2023-10-17T19:18:24.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7AOylJ5J+/l2vtdD+DeAelunfvM='} 2023-10-17T19:18:24.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007407a18bf1e115ba-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:24.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:24.908 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:24.961 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:24.962 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:25.012 INFO:teuthology.orchestra.run.smithi049.stdout:tenant: 2023-10-17T19:18:25.012 INFO:teuthology.orchestra.run.smithi049.stdout:bucket name: hbmuog-64 2023-10-17T19:18:25.023 INFO:teuthology.orchestra.run.smithi049.stdout:total entries: 30 2023-10-17T19:18:25.085 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:25.086 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:25.086 DEBUG:boto:Method: HEAD 2023-10-17T19:18:25.086 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:25.087 DEBUG:boto:Data: 2023-10-17T19:18:25.087 DEBUG:boto:Headers: {} 2023-10-17T19:18:25.088 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.088 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.088 DEBUG:boto:Params: {} 2023-10-17T19:18:25.089 DEBUG:boto:Token: None 2023-10-17T19:18:25.089 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/ 2023-10-17T19:18:25.090 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:18:25.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:18:25.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b51fe35fac9cd57-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.094 DEBUG:boto:path=/hbmuog-64/obj37 2023-10-17T19:18:25.095 DEBUG:boto:auth_path=/hbmuog-64/obj37 2023-10-17T19:18:25.095 DEBUG:boto:Method: PUT 2023-10-17T19:18:25.096 DEBUG:boto:Path: /hbmuog-64/obj37 2023-10-17T19:18:25.096 DEBUG:boto:Data: 2023-10-17T19:18:25.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:25.097 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.097 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.097 DEBUG:boto:Params: {} 2023-10-17T19:18:25.098 DEBUG:boto:Token: None 2023-10-17T19:18:25.098 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/obj37 2023-10-17T19:18:25.099 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:KOAg+3ai0Hk/xzSgMcE9edG2ELg= 2023-10-17T19:18:25.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KOAg+3ai0Hk/xzSgMcE9edG2ELg='} 2023-10-17T19:18:25.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000691507f2a32717a0-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.105 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:25.105 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:25.105 DEBUG:boto:Method: HEAD 2023-10-17T19:18:25.106 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:25.106 DEBUG:boto:Data: 2023-10-17T19:18:25.107 DEBUG:boto:Headers: {} 2023-10-17T19:18:25.107 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.107 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.108 DEBUG:boto:Params: {} 2023-10-17T19:18:25.108 DEBUG:boto:Token: None 2023-10-17T19:18:25.109 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/ 2023-10-17T19:18:25.109 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:18:25.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:18:25.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aa6f161e1c8b261-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.113 DEBUG:boto:path=/hbmuog-64/obj74 2023-10-17T19:18:25.114 DEBUG:boto:auth_path=/hbmuog-64/obj74 2023-10-17T19:18:25.114 DEBUG:boto:Method: PUT 2023-10-17T19:18:25.115 DEBUG:boto:Path: /hbmuog-64/obj74 2023-10-17T19:18:25.115 DEBUG:boto:Data: 2023-10-17T19:18:25.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:25.116 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.116 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.117 DEBUG:boto:Params: {} 2023-10-17T19:18:25.117 DEBUG:boto:Token: None 2023-10-17T19:18:25.117 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/obj74 2023-10-17T19:18:25.118 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:F29kESgwKlQcLx1P/NwOC/RWclI= 2023-10-17T19:18:25.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F29kESgwKlQcLx1P/NwOC/RWclI='} 2023-10-17T19:18:25.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddaea423727d10a3-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.124 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:25.124 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:25.125 DEBUG:boto:Method: HEAD 2023-10-17T19:18:25.125 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:25.125 DEBUG:boto:Data: 2023-10-17T19:18:25.126 DEBUG:boto:Headers: {} 2023-10-17T19:18:25.126 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.126 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.127 DEBUG:boto:Params: {} 2023-10-17T19:18:25.127 DEBUG:boto:Token: None 2023-10-17T19:18:25.128 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/ 2023-10-17T19:18:25.128 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:18:25.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:18:25.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa9499a2799663a3-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.132 DEBUG:boto:path=/hbmuog-64/obj111 2023-10-17T19:18:25.133 DEBUG:boto:auth_path=/hbmuog-64/obj111 2023-10-17T19:18:25.133 DEBUG:boto:Method: PUT 2023-10-17T19:18:25.133 DEBUG:boto:Path: /hbmuog-64/obj111 2023-10-17T19:18:25.134 DEBUG:boto:Data: 2023-10-17T19:18:25.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:25.135 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.135 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.135 DEBUG:boto:Params: {} 2023-10-17T19:18:25.136 DEBUG:boto:Token: None 2023-10-17T19:18:25.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/obj111 2023-10-17T19:18:25.137 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XNEm7hKq6AdsqYnCbxnze0PIsgs= 2023-10-17T19:18:25.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XNEm7hKq6AdsqYnCbxnze0PIsgs='} 2023-10-17T19:18:25.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fb95e6d06e8bcf0-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.183 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:25.183 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:25.184 DEBUG:boto:Method: HEAD 2023-10-17T19:18:25.184 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:25.184 DEBUG:boto:Data: 2023-10-17T19:18:25.185 DEBUG:boto:Headers: {} 2023-10-17T19:18:25.185 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.186 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.186 DEBUG:boto:Params: {} 2023-10-17T19:18:25.186 DEBUG:boto:Token: None 2023-10-17T19:18:25.187 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/ 2023-10-17T19:18:25.187 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:18:25.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:18:25.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad0af2c09f683df-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.192 DEBUG:boto:path=/hbmuog-64/obj148 2023-10-17T19:18:25.192 DEBUG:boto:auth_path=/hbmuog-64/obj148 2023-10-17T19:18:25.193 DEBUG:boto:Method: PUT 2023-10-17T19:18:25.193 DEBUG:boto:Path: /hbmuog-64/obj148 2023-10-17T19:18:25.193 DEBUG:boto:Data: 2023-10-17T19:18:25.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:25.194 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.195 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.195 DEBUG:boto:Params: {} 2023-10-17T19:18:25.195 DEBUG:boto:Token: None 2023-10-17T19:18:25.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/obj148 2023-10-17T19:18:25.196 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JgMp2F0bWUm/SFH0qJjhJ6J1peI= 2023-10-17T19:18:25.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JgMp2F0bWUm/SFH0qJjhJ6J1peI='} 2023-10-17T19:18:25.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000369992c85e85183-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.202 DEBUG:boto:path=/hbmuog-64/ 2023-10-17T19:18:25.203 DEBUG:boto:auth_path=/hbmuog-64/ 2023-10-17T19:18:25.203 DEBUG:boto:Method: HEAD 2023-10-17T19:18:25.203 DEBUG:boto:Path: /hbmuog-64/ 2023-10-17T19:18:25.204 DEBUG:boto:Data: 2023-10-17T19:18:25.204 DEBUG:boto:Headers: {} 2023-10-17T19:18:25.205 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.205 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.205 DEBUG:boto:Params: {} 2023-10-17T19:18:25.206 DEBUG:boto:Token: None 2023-10-17T19:18:25.206 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/ 2023-10-17T19:18:25.207 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:18:25.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:18:25.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c722424017d1433f-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.211 DEBUG:boto:path=/hbmuog-64/obj185 2023-10-17T19:18:25.211 DEBUG:boto:auth_path=/hbmuog-64/obj185 2023-10-17T19:18:25.212 DEBUG:boto:Method: PUT 2023-10-17T19:18:25.212 DEBUG:boto:Path: /hbmuog-64/obj185 2023-10-17T19:18:25.213 DEBUG:boto:Data: 2023-10-17T19:18:25.213 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:18:25.213 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:18:25.214 DEBUG:boto:Port: 8000 2023-10-17T19:18:25.214 DEBUG:boto:Params: {} 2023-10-17T19:18:25.215 DEBUG:boto:Token: None 2023-10-17T19:18:25.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 17 Oct 2023 19:18:25 GMT /hbmuog-64/obj185 2023-10-17T19:18:25.215 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bWHJLL9BKITdRzMtOhn1+Z7iljM= 2023-10-17T19:18:25.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWHJLL9BKITdRzMtOhn1+Z7iljM='} 2023-10-17T19:18:25.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5f3f4d80ed2a9f9-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:18:25.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:25.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:18:25.260 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:18:25.313 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:25.314 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:25.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:25.344+0000 7f8abe7a8780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-17T19:18:25.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:25.345+0000 7f8abe7a8780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-17T19:18:25.370 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-17T19:18:25.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T19:18:25.373 INFO:tasks.rgw.c1.client.1:Started 2023-10-17T19:18:25.373 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:25.394 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:18:25.394 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:18:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-17T19:18:25.396 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T19:18:26.397 DEBUG:teuthology.orchestra.run.smithi049:> curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:18:26.420 INFO:teuthology.orchestra.run.smithi049.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:18:26.420 INFO:teuthology.orchestra.run.smithi049.stderr: Dload Upload Total Spent Left Speed 2023-10-17T19:18:26.421 INFO:teuthology.orchestra.run.smithi049.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T19:18:26.422 INFO:teuthology.orchestra.run.smithi049.stdout:anonymous 2023-10-17T19:18:26.423 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hbmuog-64 2023-10-17T19:18:26.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:18:26.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:18:26.424 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:18:26.520 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:18:26.521 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:18:26.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:26.557+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:28.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T19:18:31.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:31.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:36.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:36.560+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:41.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:41.558+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:46.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:46.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:51.561 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:51.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:18:56.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:18:56.560+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:01.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:01.561+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:06.563 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:06.562+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:11.563 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:11.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:16.564 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:16.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:21.565 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:21.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:26.565 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:26.564+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:31.566 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:31.564+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:36.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:36.565+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:41.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:41.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:46.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:46.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:51.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:51.568+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:19:56.569 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:19:56.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:20:01.569 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:01.568+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:06.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:06.569+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:11.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:11.570+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:16.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:16.571+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:21.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:21.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:26.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:26.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:31.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:31.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:36.573+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:41.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:41.573+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:46.574+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:51.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:51.575+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:20:56.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:20:56.576+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:01.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:01.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:06.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:06.576+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:11.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:11.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:16.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:16.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:21.578+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:26.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:26.579+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:31.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:31.580+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:36.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:36.581+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:41.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:41.580+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:46.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:46.581+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:51.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:51.582+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:21:56.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:21:56.582+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:01.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:01.583+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:06.584+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:11.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:11.585+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:16.586+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:21.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:21.585+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:26.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:26.586+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:31.587+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:36.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:36.587+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:41.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:46.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:46.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:51.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:51.590+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:22:56.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:22:56.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:23:01.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:01.590+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:23:06.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:06.591+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:23:11.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:11.591+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:23:16.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:16.592+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:23:21.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:21.593+0000 7f79c97e6780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-17T19:23:21.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:21.593+0000 7f79c97e6780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:23:21.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:21.593+0000 7f79c97e6780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:23:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-17T19:23:21.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-10-17T19:23:21.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:21.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:21.618 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:21.674 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:21.675 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:23:21.723 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:23:21.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.724 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:21.724 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:25.032212Z" 2023-10-17T19:23:21.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.725 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:21.725 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:23:21.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.727 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:21.727 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:23:21.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.728 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:21.728 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:23:21.729 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.729 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:23:21.731 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:21.732 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:23:21.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:21.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:21.733 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:21.814+0000 7fefc7b04780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:21.815+0000 7fefc7b04780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:21.829 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:21.829 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:21.830 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:21.832 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:21.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:21.834 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:21.835 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:21.837 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:21.838 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:21.839 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:21.840 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:21.842 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:21.843 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:21Z" 2023-10-17T19:23:21.843 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:21.844 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:21Z" } 2023-10-17T19:23:21.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:21.845 DEBUG:rgw_multi.tests:log_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:21.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:21.846 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:23:21.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:21.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:21.847 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:21.894 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:21.894 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:21.940 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:23:21.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:21.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:21.942 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:21.993 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:21.993 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.039 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:23:22.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.041 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.138 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:22.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.139 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.188 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.188 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.236 INFO:teuthology.orchestra.run.smithi049.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-17T19:23:22.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.237 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.284 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.284 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.331 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:22.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.332 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.389 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.389 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.437 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:22.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.439 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.496 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.497 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.545 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:22.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.546 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.600 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.601 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.646 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:23:22.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.648 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.696 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.697 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.742 INFO:teuthology.orchestra.run.smithi049.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-10-17T19:23:22.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:22.744 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:22.793 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:22.793 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:22.845 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 13, 2023-10-17T19:23:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:23:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:23:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:23:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:23:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:23:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:23:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:23:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:23:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:23:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:23:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:23:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:23:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:23:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:23:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:23:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:23:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:23:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:23:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:23:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:23:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:23:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:23:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:23:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:23:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:23:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:23:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:23:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:23:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:23:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:23:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:23:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:23:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:23:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:23:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:23:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:23:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:23:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:23:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:23:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:23:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:23:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:23:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:23:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:23:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:23:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:23:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:23:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:23:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:23:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:23:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:23:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:23:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "directional": [ 2023-10-17T19:23:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "source_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "dest_zone": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:23:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:23:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:23:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:23:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:23:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:23:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:23:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:23:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:23:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:23:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:23:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:23:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:23:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:23:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:23:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:23:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:23:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:23:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:23:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:23:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:23:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:23:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:23:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:23:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:23:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:23:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:23:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:23:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:23:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:23:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:23:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:23:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:23:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:23:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:23:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:23:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:23:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:23:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:23:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:23:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:23:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:23:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:23:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:23:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:23:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:23:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:23:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:23:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:23:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:23:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:23:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:23:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:23:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:23:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:23:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:23:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:23:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:23:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:23:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:23:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:23:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:23:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:23:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:23:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:23:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:23:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:23:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:23:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:23:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:23:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:23:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:23:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:23:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:23:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:23:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:23:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:23:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:23:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:23:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:23:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:23:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:23:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:23:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:23:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:23:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:23:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:23:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:23:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:23:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:23:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:23:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:23:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:23:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:23:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:23:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:23:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:23:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:23:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:23:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:23:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:23:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:23:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:23:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:23:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:23:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:23:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:23:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:23:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:22.904 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:22.905 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-65 2023-10-17T19:23:22.905 DEBUG:boto:path=/hbmuog-65/ 2023-10-17T19:23:22.906 DEBUG:boto:auth_path=/hbmuog-65/ 2023-10-17T19:23:22.906 DEBUG:boto:Method: PUT 2023-10-17T19:23:22.907 DEBUG:boto:Path: /hbmuog-65/ 2023-10-17T19:23:22.907 DEBUG:boto:Data: 2023-10-17T19:23:22.908 DEBUG:boto:Headers: {} 2023-10-17T19:23:22.908 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:23:22.908 DEBUG:boto:Port: 8000 2023-10-17T19:23:22.909 DEBUG:boto:Params: {} 2023-10-17T19:23:22.910 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:23:22.910 DEBUG:boto:Token: None 2023-10-17T19:23:22.911 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:23:22 GMT /hbmuog-65/ 2023-10-17T19:23:22.911 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VEp9FCMX8y5LS+wNPvmmEfm3z5M= 2023-10-17T19:23:22.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:23:22 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VEp9FCMX8y5LS+wNPvmmEfm3z5M=', 'Content-Length': '0'} 2023-10-17T19:23:23.118 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050d2f1224e5fb4f9-00652edf2b-1eae-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:23:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:23:23.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.120 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.203 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.204 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.320 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:23:23.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.322 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.400 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.401 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.483 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:23.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.484 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.547 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.547 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.596 INFO:teuthology.orchestra.run.smithi049.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-17T19:23:23.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.598 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.694 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:23.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.695 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.752 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.752 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.802 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:23.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.804 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:23.912 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:23:23.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:23.928 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:23.981 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:23.982 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:24.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:23:24.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:24.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:23:24.034 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:23:24.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:24.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:24.134 INFO:teuthology.orchestra.run.smithi049.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-10-17T19:23:24.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:24.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:24.136 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:24.185 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:24.185 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:24.218 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:23:24.235 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:23:24.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.235 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570604.018172_940.1", 2023-10-17T19:23:24.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:23:24.018172Z" 2023-10-17T19:23:24.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.236 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:24.236 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:23:24.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.237 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:24.237 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:23:24.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.238 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:24.238 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:23:24.238 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.239 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:23:24.240 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:24.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:23:24.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:24.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:24.241 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:24.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:24.292 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:24.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:24.321+0000 7fdf75f3e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:24.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:24.322+0000 7fdf75f3e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:24.336 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:24.336 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:24.336 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:24.337 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:24.337 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:24.337 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:24.337 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:24.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:24.339 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:24.339 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:24.339 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:24.339 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:24.339 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:24.340 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:24.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.340 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:24.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:24.342 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:24.342 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:24.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.343 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:24.343 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:24.343 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:24.343 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:24.345 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:24.346 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:24.346 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:24.346 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:24.346 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:24.347 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:24.347 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:24.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:24.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:24.348 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:24Z" 2023-10-17T19:23:24.349 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:24.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:24Z" } 2023-10-17T19:23:24.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:24.351 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:24.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:24.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:29.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:29.354 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:29.413 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:29.413 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:29.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:29.447+0000 7fcddb604780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:29.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:29.448+0000 7fcddb604780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:29.462 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:29.463 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:29.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:29.465 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:29.468 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:29.469 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:29.470 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:29.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:29.472 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:29.473 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:29.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:29.475 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:29Z" 2023-10-17T19:23:29.476 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:29.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:29Z" } 2023-10-17T19:23:29.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:29.478 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:29.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:29.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:34.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:34.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:34.481 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:34.540 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:34.541 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:34.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:34.573+0000 7f62d89fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:34.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:34.574+0000 7f62d89fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:34.590 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:34.590 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:34.591 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:34.591 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:34.591 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:34.591 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:34.592 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:34.593 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:34.593 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:34.593 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:34.593 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:34.593 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:34.594 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:34.594 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:34.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:34.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:34.595 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:34.595 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:34.595 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:34.595 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:34.595 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:34.596 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:34.596 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:34.596 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:34.596 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:34.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:34.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:34.597 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:34.597 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:34.597 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:34.598 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:34.598 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:34.598 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:34.598 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:34.598 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:34.599 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:34.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:34.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:34.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:34.600 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:34.600 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:34.600 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:34.600 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:34.600 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:34.601 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:34.601 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:34.601 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:34.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:34.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:34.602 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:34.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.602 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:34.602 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:34.602 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:34.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:34.603 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:34Z" 2023-10-17T19:23:34.603 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:34.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:34Z" } 2023-10-17T19:23:34.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:34.605 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:34.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:34.606 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:39.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:39.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:39.608 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:39.667 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:39.668 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:39.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:39.700+0000 7feb84c27780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:39.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:39.701+0000 7feb84c27780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:39.717 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:39.719 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:39.720 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:39.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:39.722 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:39.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:39.725 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:39.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:39.728 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:39.729 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:39.730 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:39.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:39.732 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:39Z" 2023-10-17T19:23:39.732 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:39.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:39Z" } 2023-10-17T19:23:39.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:39.734 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:39.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:39.735 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:44.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:44.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:44.737 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:44.796 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:44.796 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:44.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:44.828+0000 7f1142a30780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:44.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:44.829+0000 7f1142a30780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:44.845 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:44.845 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:44.846 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:44.846 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:44.846 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:44.846 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:44.847 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:44.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.847 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:44.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:44.848 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:44.848 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:44.848 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:44.848 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:44.849 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:44.849 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:44.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:44.849 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:44.850 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:44.850 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:44.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:44.851 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:44.851 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:44.851 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:44.851 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:44.852 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:44.852 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:44.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:44.852 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:44.852 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:44.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:44.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:44.853 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:44.854 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:44.854 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:44.854 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:44.854 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:44.855 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:44.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:44.855 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:44.855 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:44.855 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:44.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:44.856 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:44.856 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:44.857 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:44.857 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:44.857 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:44.857 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:44.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:44.858 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:44.858 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:44.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:44.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:44.859 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:44.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.859 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:44.859 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:44.860 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:44.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:44.860 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:44Z" 2023-10-17T19:23:44.860 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:44.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:44Z" } 2023-10-17T19:23:44.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:44.862 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:44.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:44.866 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:49.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:49.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:49.869 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:49.961+0000 7f24937e3780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:49.962+0000 7f24937e3780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:49.977 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:49.978 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:49.978 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:49.978 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:49.979 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:49.979 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:49.979 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:49.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.979 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:49.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:49.980 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:49.980 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:49.980 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:49.981 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:49.982 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:49.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:49.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:49.983 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:49.983 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:49.983 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:49.983 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:49.983 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:49.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:49.985 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:49.985 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:49.985 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:49.986 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:49.986 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:49.986 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:49.986 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:49.987 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:49.987 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:49.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:49.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:49.988 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:49.988 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:49.988 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:49.988 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:49.989 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:49.989 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:49.989 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:49.990 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:49.990 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:49.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:49.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:49.991 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:49.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.993 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:49.994 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:49.994 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:49.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:49.994 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:49Z" 2023-10-17T19:23:49.994 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:49.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:49Z" } 2023-10-17T19:23:49.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:49.996 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:49.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:49.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:23:54.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:54.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:23:55.000 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:23:55.061 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:23:55.061 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:23:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:55.094+0000 7f8ce9443780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:55.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:23:55.095+0000 7f8ce9443780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:23:55.110 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:23:55.110 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:23:55.111 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:23:55.111 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:23:55.111 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:23:55.111 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:23:55.112 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:55.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.112 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:23:55.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:55.112 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:23:55.113 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:55.114 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:23:55.114 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:55.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:55.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:23:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:23:55.116 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:55.116 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:55.116 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:55.116 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:23:55.116 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:55.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:55.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:55.117 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:23:55.117 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:55.118 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:55.118 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:23:55.118 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:55.118 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:55.118 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:55.119 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:23:55.119 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:55.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:55.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:23:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:23:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:23:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:23:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:23:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:23:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:23:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:23:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:23:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:23:55.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:55.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:23:55.122 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:23:55.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.123 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:23:55.123 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:23:55.123 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:23:55.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:23:55.123 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:23:55Z" 2023-10-17T19:23:55.124 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:23:55.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:23:55Z" } 2023-10-17T19:23:55.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:23:55.126 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:55.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:23:55.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:00.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:00.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:00.129 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:00.188 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:00.189 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:00.229+0000 7ff65ef27780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:00.230+0000 7ff65ef27780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:00.245 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:00.245 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:00.245 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:00.245 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:00.246 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:00.246 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:00.246 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:00.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.247 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:00.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:00.248 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:00.248 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:00.248 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:00.248 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:00.249 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:00.249 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:00.249 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:00.250 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:00.250 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:00.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:00.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:00.252 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:00.252 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:00.252 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:00.253 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:00.253 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:00.253 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:00.254 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:00.254 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:00.254 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:00.254 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:00.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:00.255 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:00.255 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:00.256 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:00.256 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:00.256 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:00.256 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:00.256 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:00.257 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:00.257 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:00.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:00.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:00.258 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:00.258 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:00.258 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:00.258 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:00.259 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:00.259 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:00.259 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:00.259 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:00.259 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:00.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:00.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:00.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:00.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.260 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:00.261 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:00.261 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:00.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:00.261 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:00Z" 2023-10-17T19:24:00.261 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:00.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:00Z" } 2023-10-17T19:24:00.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:00.263 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:00.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:00.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:05.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:05.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:05.267 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:05.326 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:05.327 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:05.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:05.361+0000 7f437e73e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:05.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:05.362+0000 7f437e73e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:05.376 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:05.376 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.388 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:05.389 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:05.389 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:05.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:05.389 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:05Z" 2023-10-17T19:24:05.389 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:05.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:05Z" } 2023-10-17T19:24:05.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:05.391 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:05.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:05.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:10.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:10.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:10.395 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:10.454 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:10.454 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:10.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:10.487+0000 7fae81ccd780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:10.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:10.488+0000 7fae81ccd780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:10.504 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:10Z" 2023-10-17T19:24:10.519 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:10.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:10Z" } 2023-10-17T19:24:10.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:10.521 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:10.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:10.522 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:15.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:15.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:15.524 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:15.583 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:15.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:15.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:15.617+0000 7fcc13156780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:15.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:15.618+0000 7fcc13156780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:15.633 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:15.634 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:15.634 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:15.634 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:15.635 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:15.635 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:15.635 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:15.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.635 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:15.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:15.636 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:15.636 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:15.636 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:15.636 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:15.637 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:15.637 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:15.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:15.637 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:15.638 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:15.638 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:15.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:15.638 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:15.639 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:15.639 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:15.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:15.639 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:15.639 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:15.641 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:15.641 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:15.641 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:15.641 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:15.641 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:15.642 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:15.642 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:15.642 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:15.642 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:15.643 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:15.644 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:15.644 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:15.644 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:15.644 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:15.644 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:15.645 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:15.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:15.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:15.645 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:15.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.646 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:15.646 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:15.646 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:15.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:15.646 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:15Z" 2023-10-17T19:24:15.647 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:15.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:15Z" } 2023-10-17T19:24:15.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:15.648 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:15.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:15.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:20.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:20.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:20.651 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:20.744+0000 7f3239d98780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:20.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:20.745+0000 7f3239d98780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:20.761 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:20.761 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:20.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:20.763 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:20.764 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:20.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:20.766 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:20.767 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:20.769 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:20.770 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:20.772 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:20.773 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:20.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:20.775 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:20Z" 2023-10-17T19:24:20.776 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:20.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:20Z" } 2023-10-17T19:24:20.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:20.778 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:20.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:20.779 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:25.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:25.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:25.781 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:25.840 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:25.841 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:25.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:25.872+0000 7f4ea6937780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:25.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:25.873+0000 7f4ea6937780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:25.889 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:25.890 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:25.890 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:25.890 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:25.890 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:25.890 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:25.891 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:25.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.891 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:25.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:25.893 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:25.893 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:25.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:25.893 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:25.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:25.894 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:25.895 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:25.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:25.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:25.902 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:25Z" 2023-10-17T19:24:25.903 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:25.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:25Z" } 2023-10-17T19:24:25.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:25.905 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:25.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:25.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:30.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:30.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:30.907 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:30.966 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:30.966 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:31.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:31.000+0000 7f6f916b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:31.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:31.001+0000 7f6f916b2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:31.016 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:31.016 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:31.016 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:31.016 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:31.017 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:31.017 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:31.017 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:31.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.017 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:31.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:31.018 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:31.018 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:31.018 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:31.019 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:31.019 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:31.019 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:31.019 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:31.019 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:31.020 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:31.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:31.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:31.021 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:31.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:31.023 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:31.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:31.026 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:31.027 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.028 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:31.029 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:31Z" 2023-10-17T19:24:31.029 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:31.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:31Z" } 2023-10-17T19:24:31.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:31.031 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:31.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:31.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:36.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:36.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:36.034 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:36.094 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:36.094 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:36.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:36.127+0000 7fd77edab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:36.129 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:36.129+0000 7fd77edab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:36.143 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:36Z" 2023-10-17T19:24:36.157 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:36.158 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:36Z" } 2023-10-17T19:24:36.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:36.159 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:36.159 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:36.160 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:41.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:41.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:41.162 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:41.220 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:41.220 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:41.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:41.253+0000 7fc16bc0f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:41.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:41.254+0000 7fc16bc0f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:41Z" 2023-10-17T19:24:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:41.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:41Z" } 2023-10-17T19:24:41.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:41.283 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:41.284 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:41.284 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:46.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:46.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:46.286 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:46.345 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:46.345 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:46.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:46.379+0000 7f3f8147f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:46.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:46.380+0000 7f3f8147f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:46.399 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:46.412 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:46Z" 2023-10-17T19:24:46.412 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:46.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:46Z" } 2023-10-17T19:24:46.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:46.413 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:46.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:46.414 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:51.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:51.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:51.416 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:51.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:51.507+0000 7f2a59d67780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:51.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:51.509+0000 7f2a59d67780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:51.525 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:51.526 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:51.526 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:51.526 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:51.527 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:51.527 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:51.527 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:51.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.528 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:51.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:51.528 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:51.528 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:51.528 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:51.529 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:51.529 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:51.529 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:51.529 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:51.529 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:51.530 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:51.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:51.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:51.531 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:51.531 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:51.531 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:51.531 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:51.531 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:51.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:51.533 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:51.534 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:51.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:51.536 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:51.537 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:51.538 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:51.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:51.540 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:51Z" 2023-10-17T19:24:51.540 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:51.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:51Z" } 2023-10-17T19:24:51.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:51.542 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:51.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:51.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:24:56.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:56.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:24:56.545 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:24:56.604 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:24:56.605 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:24:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:56.637+0000 7f7995b9a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:24:56.638+0000 7f7995b9a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:24:56.654 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:24:56.654 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:24:56.655 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:24:56.655 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:24:56.655 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:24:56.655 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:24:56.656 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:56.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.656 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:24:56.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:56.656 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:24:56.657 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:56.657 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:56.657 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:24:56.657 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:56.657 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:24:56.658 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:56.658 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:24:56.658 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:56.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:56.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:24:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:56.659 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:24:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:24:56.660 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:24:56.661 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:24:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:56.662 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:24:56.663 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:24:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:24:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:24:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:24:56.666 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:24:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:24:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:24:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:24:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:24:56.667 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:24:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:24:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:24:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.668 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:24:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:24:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:24:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:24:56.669 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:24:56Z" 2023-10-17T19:24:56.670 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:24:56.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:24:56Z" } 2023-10-17T19:24:56.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:24:56.672 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:56.672 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:24:56.673 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:25:01.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:01.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:01.675 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:01.734 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:01.767+0000 7fdec1ef7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:01.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:01.768+0000 7fdec1ef7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:01.784 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:01.785 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:01.785 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:01.785 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:01.785 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:01.785 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:01.786 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570305.032212_898.1", 2023-10-17T19:25:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:01.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.032212Z", 2023-10-17T19:25:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:01.789 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:01.790 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:01.791 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:01.792 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:01.794 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:01.795 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.796 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:01.797 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:01Z" 2023-10-17T19:25:01.797 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:01.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570305.032212_898.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:18:25.032212Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:01Z" } 2023-10-17T19:25:01.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570305.032212_898.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.032212Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:01.799 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:01.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570305.032212_898.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:01.800 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570305.032212_898.1 2023-10-17T19:25:06.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:06.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:06.803 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:06.862 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:06.863 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:06.899 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:06.898+0000 7f28aea1c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:06.900 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:06.900+0000 7f28aea1c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:06.925 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:06.925 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:06.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:06.927 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570603.087973_913.1", 2023-10-17T19:25:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:23:23.087973Z", 2023-10-17T19:25:06.928 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:06.929 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:06.930 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:06.932 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:06.933 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:06.934 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:06.935 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:06.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:06.937 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:06Z" 2023-10-17T19:25:06.938 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:06.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570603.087973_913.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:23:23.087973Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:06Z" } 2023-10-17T19:25:06.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570603.087973_913.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:23:23.087973Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:06.939 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:06.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570603.087973_913.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:06.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570604.018172_940.1 target=1_1697570603.087973_913.1 2023-10-17T19:25:11.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:11.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:11.943 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:12.002 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.002 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:12.035+0000 7ff199766780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:12.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:12.036+0000 7ff199766780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:12.051 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570604.018172_940.1", 2023-10-17T19:25:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:23:24.018172Z", 2023-10-17T19:25:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.056 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.056 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:12.056 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.057 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.057 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:12.057 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.057 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.057 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.058 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:12.058 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.059 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:12.059 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.059 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.059 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:12.060 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.060 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.060 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.060 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:12.060 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.061 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.062 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.063 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.063 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:12.063 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:12.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.064 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:12.064 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:12.064 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:12.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.065 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:12Z" 2023-10-17T19:25:12.065 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:12.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570604.018172_940.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:23:24.018172Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:12Z" } 2023-10-17T19:25:12.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570604.018172_940.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:23:24.018172Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:12.067 DEBUG:rgw_multi.tests:log_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:12.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:12.068 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:25:12.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.069 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.174 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:12.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.176 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:12.235 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.287 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:12.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.288 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:12.348 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.349 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.403 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:12.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.404 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.454 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.504 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:25:12.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hbmuog-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:12.507 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hbmuog-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.555 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.601 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:25:12.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:12.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:12.603 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:12.652 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.653 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.684 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:25:12.697 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:25:12.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.698 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:12.698 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:25:12.489592Z" 2023-10-17T19:25:12.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.699 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:12.699 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:25:12.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.700 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:12.700 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:25:12.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.701 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:12.701 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:25:12.702 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.702 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:25:12.703 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:12.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:25:12.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:12.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:12.705 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:12.755 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:12.756 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:12.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:12.785+0000 7f4fee8f0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:12.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:12.786+0000 7f4fee8f0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:12.800 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:12.800 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:12.800 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:12.800 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:12.801 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:12.801 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:12.801 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.802 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:12.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.803 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:12.803 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.803 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.803 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570604.018172_940.1", 2023-10-17T19:25:12.804 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.804 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:12.804 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.804 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:23:24.018172Z", 2023-10-17T19:25:12.805 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.806 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:12.806 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.806 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.806 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:12.807 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.807 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.807 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.809 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.810 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:12.811 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:12.813 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:12.814 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:12.814 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:12.815 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:12.815 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:12.815 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:12.816 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:12.816 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:12.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:12.817 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:12.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.818 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:12.819 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:12Z" 2023-10-17T19:25:12.820 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:12.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570604.018172_940.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:23:24.018172Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:12Z" } 2023-10-17T19:25:12.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570604.018172_940.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:23:24.018172Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:12.822 DEBUG:rgw_multi.tests:log_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:12.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570604.018172_940.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:12.823 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570712.489592_955.1 target=1_1697570604.018172_940.1 2023-10-17T19:25:17.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:17.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:17.826 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:17.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:17.918+0000 7efd54cde780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:17.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:17.920+0000 7efd54cde780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:17.935 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:17.935 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:17.935 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:17.936 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:17.936 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:17.936 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:17.936 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:17.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.937 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:17.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:17.937 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:17.938 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:17.938 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:17.938 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:17.938 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:17.939 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:17.940 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:17.941 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:17.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:17.943 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:17.944 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:17.945 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:17.946 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:17.948 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:17.949 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:17Z" 2023-10-17T19:25:17.949 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:17.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:17Z" } 2023-10-17T19:25:17.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:17.951 DEBUG:rgw_multi.tests:log_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:17.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:17.952 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:25:17.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:17.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:17.953 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.007 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.007 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.054 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:18.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.056 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.113 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.113 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.160 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:18.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.162 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.216 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.216 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.263 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-17T19:25:18.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.265 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.313 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.314 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.361 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:25:18.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.362 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.409 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.409 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.455 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:25:18.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.457 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:18.507 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.508 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.559 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:18.559 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:18.560 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 14, 2023-10-17T19:25:18.560 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:25:18.560 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:25:18.560 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:25:18.561 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:18.561 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:25:18.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.561 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:25:18.562 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:25:18.562 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:25:18.562 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:25:18.562 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.562 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:25:18.563 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.563 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:25:18.563 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:25:18.563 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:18.564 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:18.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.564 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:18.564 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:25:18.565 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.565 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:25:18.565 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.565 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:18.568 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.568 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.569 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:25:18.569 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:25:18.569 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.569 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:25:18.569 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.570 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:18.570 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:18.570 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:18.571 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:18.571 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:18.571 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:18.571 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:18.571 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:18.572 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:18.572 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:18.572 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.572 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:25:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:25:18.574 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:25:18.574 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:25:18.574 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:25:18.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.575 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.575 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:25:18.575 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:18.575 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:25:18.576 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:25:18.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.576 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:25:18.577 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.577 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.578 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:25:18.578 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:25:18.578 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:25:18.579 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:25:18.579 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.579 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:25:18.579 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.579 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:25:18.580 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:25:18.580 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:18.581 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:18.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.581 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:25:18.581 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:25:18.582 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.582 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:25:18.582 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.583 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:18.583 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:18.583 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:18.583 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:18.584 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:18.584 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:18.584 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:18.584 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:18.585 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:18.585 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:18.586 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.586 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.587 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:18.587 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:25:18.588 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:18.588 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:25:18.588 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.588 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:18.589 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:18.589 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:18.589 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:18.589 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:18.590 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:18.590 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:18.590 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:18.590 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:18.591 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:18.591 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.591 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.592 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.592 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:25:18.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.593 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:25:18.593 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:25:18.593 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:25:18.593 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:25:18.593 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.594 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.594 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:25:18.594 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:18.595 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:25:18.595 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:25:18.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.595 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:25:18.595 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:18.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.596 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:18.596 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:25:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:25:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:25:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.598 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:18.598 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:25:18.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.599 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:25:18.599 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:25:18.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.600 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:18.600 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:25:18.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.601 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.601 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:25:18.601 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:18.601 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:25:18.602 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:25:18.602 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:25:18.602 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:25:18.602 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:25:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:25:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:25:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:25:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:25:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:25:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:25:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:25:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:25:18.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.605 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:25:18.605 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:18.605 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:18.605 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:18.606 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:18.606 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:18.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.606 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:25:18.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:18.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:18.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:18.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:18.607 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:18.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.608 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:25:18.608 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:18.609 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:18.609 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:18.609 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:18.609 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:18.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.610 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:18.611 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:25:18.611 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.611 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:18.612 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-10-17T19:25:18.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:18.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:18.615 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:18.670 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.670 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.718 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:25:18.752 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:25:18.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.752 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:18.753 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:25:12.489592Z" 2023-10-17T19:25:18.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.753 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:18.754 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:25:18.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.754 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:18.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:25:18.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.755 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:18.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:25:18.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.756 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:25:18.757 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:18.758 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:25:18.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:18.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:18.759 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:18.827 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:18.828 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:18.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:18.925+0000 7f5215502780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:18.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:18.928+0000 7f5215502780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:18.965 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:18.965 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:18.965 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:18.965 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:18.966 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:18.966 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:18.966 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.967 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:18.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.967 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:18.967 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:18.967 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:18.968 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:18.968 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:18.968 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:18.968 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:18.968 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:18.969 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.969 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:18.970 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:18.970 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:18.970 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:18.970 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:18.970 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:18.971 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:18.971 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:18.971 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.972 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:18.972 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:18.972 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:18.973 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:18.973 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:18.973 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:18.973 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:18.973 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:18.974 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:18.974 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:18.975 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:18.975 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:18.975 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:18.975 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:18.975 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:18.976 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:18.976 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:18.976 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:18.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:18.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:18.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.977 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:18.977 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:18.978 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:18.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:18.978 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:18Z" 2023-10-17T19:25:18.978 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:18.979 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:18Z" } 2023-10-17T19:25:18.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:18.980 DEBUG:rgw_multi.tests:log_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:18.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:18.981 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:25:18.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:18.983 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.053 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.053 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.148 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:25:19.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.150 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.220 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.221 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.293 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:25:19.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.294 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.412 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:25:19.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.414 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.468 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.469 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.515 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:25:19.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.517 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.567 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.614 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:25:19.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.616 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.666 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.666 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.721 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:19.722 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:19.722 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 15, 2023-10-17T19:25:19.722 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:25:19.723 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:25:19.723 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:25:19.723 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:19.723 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:25:19.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.724 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:25:19.724 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:25:19.724 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:25:19.725 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:25:19.725 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.725 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:25:19.725 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.726 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:25:19.726 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:25:19.726 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.726 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:19.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.727 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.727 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:25:19.727 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.727 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:25:19.728 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.728 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:19.728 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:19.728 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:19.729 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:19.729 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:19.729 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:19.729 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:19.729 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:19.730 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:19.730 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:19.730 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.730 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.731 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:25:19.731 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:25:19.732 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.732 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:25:19.732 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.732 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:19.732 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:19.733 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:19.734 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:19.734 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:19.734 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:19.734 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.735 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.735 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.735 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.735 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:25:19.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.736 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:25:19.736 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:25:19.736 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:25:19.736 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:25:19.737 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.737 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.737 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:25:19.737 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:19.738 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:25:19.738 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:25:19.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.738 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:25:19.739 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:25:19.739 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:25:19.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.739 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:25:19.740 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:19.740 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.740 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:25:19.740 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.740 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.741 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.741 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:25:19.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.742 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:25:19.742 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:25:19.742 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:25:19.742 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:19.742 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.743 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:25:19.743 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.743 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:25:19.743 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:25:19.744 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:19.744 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.744 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:25:19.744 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.745 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:25:19.745 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:25:19.745 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:25:19.745 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:25:19.745 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.746 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:25:19.746 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:25:19.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:25:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:25:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.747 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:25:19.748 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.748 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.748 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:25:19.748 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.749 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.749 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:25:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:25:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:25:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:25:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:25:19.751 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.751 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:25:19.751 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:25:19.751 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:19.751 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:25:19.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.752 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:25:19.752 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:25:19.752 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.753 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:25:19.753 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.753 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:19.753 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:19.753 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:19.754 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:19.754 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:19.754 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:19.754 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:19.754 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:19.755 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:19.755 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:19.755 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.755 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.756 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:19.756 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:25:19.756 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:25:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:25:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:25:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:25:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:25:19.758 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:25:19.758 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:25:19.758 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:25:19.759 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:25:19.759 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:25:19.759 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:25:19.759 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:25:19.759 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.760 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.760 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.760 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.760 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:25:19.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.761 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:25:19.761 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:25:19.761 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:25:19.762 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:25:19.762 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.762 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.762 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:25:19.763 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:19.763 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:25:19.763 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:25:19.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.763 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:25:19.764 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:25:19.764 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.764 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.764 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:25:19.765 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:25:19.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.765 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:25:19.765 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:25:19.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:25:19.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.768 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:25:19.768 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:25:19.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:19.769 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:25:19.769 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:25:19.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.769 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:19.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.770 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:25:19.770 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:25:19.770 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:25:19.770 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:25:19.771 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:25:19.771 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:25:19.771 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:25:19.771 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:25:19.771 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:25:19.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.772 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:25:19.772 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:25:19.772 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:25:19.773 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:25:19.773 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:25:19.773 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:25:19.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.773 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:25:19.774 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:19.774 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:19.774 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:19.774 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:19.775 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:19.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.775 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:25:19.775 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:19.775 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:19.776 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:19.776 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:19.776 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:19.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.776 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:25:19.777 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:25:19.777 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:25:19.777 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:25:19.777 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:25:19.778 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:25:19.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:19.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:19.778 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:25:19.779 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:25:19.779 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:19.779 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:19.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:25:19.781 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:25:19.837 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:19.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:19.938 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:25:19.939 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-66 2023-10-17T19:25:19.940 DEBUG:boto:path=/hbmuog-66/ 2023-10-17T19:25:19.940 DEBUG:boto:auth_path=/hbmuog-66/ 2023-10-17T19:25:19.941 DEBUG:boto:Method: PUT 2023-10-17T19:25:19.941 DEBUG:boto:Path: /hbmuog-66/ 2023-10-17T19:25:19.941 DEBUG:boto:Data: 2023-10-17T19:25:19.942 DEBUG:boto:Headers: {} 2023-10-17T19:25:19.942 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:25:19.943 DEBUG:boto:Port: 8000 2023-10-17T19:25:19.943 DEBUG:boto:Params: {} 2023-10-17T19:25:19.944 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:25:19.945 DEBUG:boto:Token: None 2023-10-17T19:25:19.945 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:25:19 GMT /hbmuog-66/ 2023-10-17T19:25:19.946 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:SLo8KWovXvq3Ptm3bAcPYeP68Ek= 2023-10-17T19:25:19.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:25:19 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLo8KWovXvq3Ptm3bAcPYeP68Ek=', 'Content-Length': '0'} 2023-10-17T19:25:20.040 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e69db0155ab363c1-00652edfa0-206d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.041 DEBUG:boto:path=/hbmuog-66/ 2023-10-17T19:25:20.042 DEBUG:boto:auth_path=/hbmuog-66/ 2023-10-17T19:25:20.042 DEBUG:boto:Method: HEAD 2023-10-17T19:25:20.042 DEBUG:boto:Path: /hbmuog-66/ 2023-10-17T19:25:20.043 DEBUG:boto:Data: 2023-10-17T19:25:20.043 DEBUG:boto:Headers: {} 2023-10-17T19:25:20.044 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:25:20.044 DEBUG:boto:Port: 8000 2023-10-17T19:25:20.045 DEBUG:boto:Params: {} 2023-10-17T19:25:20.045 DEBUG:boto:Token: None 2023-10-17T19:25:20.046 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:25:20 GMT /hbmuog-66/ 2023-10-17T19:25:20.046 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3V15zNWBKPb7xBR7hsy8+2QqRps= 2023-10-17T19:25:20.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:25:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3V15zNWBKPb7xBR7hsy8+2QqRps=', 'Content-Length': '0'} 2023-10-17T19:25:20.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001e24b23d7dd2657-00652edfa0-206d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.051 DEBUG:boto:path=/hbmuog-66/obj1 2023-10-17T19:25:20.052 DEBUG:boto:auth_path=/hbmuog-66/obj1 2023-10-17T19:25:20.052 DEBUG:boto:Method: PUT 2023-10-17T19:25:20.053 DEBUG:boto:Path: /hbmuog-66/obj1 2023-10-17T19:25:20.053 DEBUG:boto:Data: 2023-10-17T19:25:20.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:25:20.054 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:25:20.054 DEBUG:boto:Port: 8000 2023-10-17T19:25:20.055 DEBUG:boto:Params: {} 2023-10-17T19:25:20.055 DEBUG:boto:Token: None 2023-10-17T19:25:20.056 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:25:20 GMT /hbmuog-66/obj1 2023-10-17T19:25:20.056 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:EDuEXfRjrC0G1tBPcOXuvQ9iLPI= 2023-10-17T19:25:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:25:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EDuEXfRjrC0G1tBPcOXuvQ9iLPI='} 2023-10-17T19:25:20.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf6ae83f09c27264-00652edfa0-206d-a1'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.070 INFO:rgw_multi.tests:create bucket zone=a2 name=hbmuog-67 2023-10-17T19:25:20.071 DEBUG:boto:path=/hbmuog-67/ 2023-10-17T19:25:20.071 DEBUG:boto:auth_path=/hbmuog-67/ 2023-10-17T19:25:20.071 DEBUG:boto:Method: PUT 2023-10-17T19:25:20.072 DEBUG:boto:Path: /hbmuog-67/ 2023-10-17T19:25:20.072 DEBUG:boto:Data: 2023-10-17T19:25:20.073 DEBUG:boto:Headers: {} 2023-10-17T19:25:20.073 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:25:20.073 DEBUG:boto:Port: 8001 2023-10-17T19:25:20.074 DEBUG:boto:Params: {} 2023-10-17T19:25:20.075 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:25:20.075 DEBUG:boto:Token: None 2023-10-17T19:25:20.076 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:25:20 GMT /hbmuog-67/ 2023-10-17T19:25:20.076 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:0UY1UCdePkrAIeEP2frstJEqaKI= 2023-10-17T19:25:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:25:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0UY1UCdePkrAIeEP2frstJEqaKI=', 'Content-Length': '0'} 2023-10-17T19:25:20.116 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e395ab4e5ff02a1f-00652edfa0-2061-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.117 DEBUG:boto:path=/hbmuog-67/ 2023-10-17T19:25:20.117 DEBUG:boto:auth_path=/hbmuog-67/ 2023-10-17T19:25:20.118 DEBUG:boto:Method: HEAD 2023-10-17T19:25:20.118 DEBUG:boto:Path: /hbmuog-67/ 2023-10-17T19:25:20.119 DEBUG:boto:Data: 2023-10-17T19:25:20.119 DEBUG:boto:Headers: {} 2023-10-17T19:25:20.120 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:25:20.120 DEBUG:boto:Port: 8001 2023-10-17T19:25:20.120 DEBUG:boto:Params: {} 2023-10-17T19:25:20.121 DEBUG:boto:Token: None 2023-10-17T19:25:20.121 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:25:20 GMT /hbmuog-67/ 2023-10-17T19:25:20.122 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:3Nnce/M7CHKV5yDLOKdyVlaYJbM= 2023-10-17T19:25:20.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:25:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3Nnce/M7CHKV5yDLOKdyVlaYJbM=', 'Content-Length': '0'} 2023-10-17T19:25:20.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d06bcc1569370d-00652edfa0-2061-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.126 DEBUG:boto:path=/hbmuog-67/obj2 2023-10-17T19:25:20.126 DEBUG:boto:auth_path=/hbmuog-67/obj2 2023-10-17T19:25:20.127 DEBUG:boto:Method: PUT 2023-10-17T19:25:20.127 DEBUG:boto:Path: /hbmuog-67/obj2 2023-10-17T19:25:20.128 DEBUG:boto:Data: 2023-10-17T19:25:20.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:25:20.128 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:25:20.129 DEBUG:boto:Port: 8001 2023-10-17T19:25:20.129 DEBUG:boto:Params: {} 2023-10-17T19:25:20.130 DEBUG:boto:Token: None 2023-10-17T19:25:20.130 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:25:20 GMT /hbmuog-67/obj2 2023-10-17T19:25:20.131 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Mzhc+YrwVSQaMCw24crz3CP/NTs= 2023-10-17T19:25:20.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:25:20 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mzhc+YrwVSQaMCw24crz3CP/NTs='} 2023-10-17T19:25:20.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000472e1ddcac9dc6d8-00652edfa0-2061-a2'), ('Date', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:25:20.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:20.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:20.145 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:20.214 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:20.214 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:20.263 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:25:20.281 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:25:20.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.282 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570720.102476_987.1", 2023-10-17T19:25:20.282 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:25:20.102476Z" 2023-10-17T19:25:20.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.283 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:20.283 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:25:20.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.283 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:20.284 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:25:20.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.284 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:20.284 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:25:20.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.285 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:25:20.286 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:20.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:25:20.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:20.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:20.288 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:20.350 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:20.350 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:20.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:20.391+0000 7f3f70ef4780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:20.392+0000 7f3f70ef4780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:20.407 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:20Z" 2023-10-17T19:25:20.422 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:20.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:20Z" } 2023-10-17T19:25:20.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:20.424 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:20.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:20.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:25.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:25.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:25.427 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:25.487 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:25.488 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:25.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:25.519+0000 7f92e8890780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:25.520+0000 7f92e8890780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:25.536 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:25.536 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:25.537 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:25.538 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:25.539 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:25.541 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:25.541 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:25.541 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:25.541 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:25.542 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:25.542 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:25.542 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:25.542 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:25.543 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:25.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:25.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:25.544 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:25.544 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:25.544 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:25.544 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:25.544 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:25.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:25.546 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:25.546 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:25.546 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:25.547 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:25.547 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:25.547 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:25.547 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:25.547 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:25.548 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:25.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:25.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:25.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:25.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.549 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:25.549 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:25.549 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:25.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:25.550 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:25Z" 2023-10-17T19:25:25.550 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:25.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:25Z" } 2023-10-17T19:25:25.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:25.552 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:25.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:25.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:30.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:30.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:30.555 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:30.615 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:30.616 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:30.650 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:30.648+0000 7f284dc3f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:30.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:30.649+0000 7f284dc3f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:30.665 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:30.665 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:30.666 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:30.666 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:30.666 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:30.666 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:30.666 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:30.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.667 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:30.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:30.667 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:30.668 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:30.669 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:30.669 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:30.669 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:30.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:30.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:30.670 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:30.670 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:30.670 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:30.671 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:30.671 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:30.671 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:30.671 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:30.671 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:30.672 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:30.672 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:30.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:30.672 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:30.673 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:30.673 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:30.673 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:30.673 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:30.673 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:30.674 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:30.674 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:30.674 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:30.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:30.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:30.675 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:30.675 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:30.675 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:30.676 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:30.676 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:30.676 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:30.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:30.677 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:30.677 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:30.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:30.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:30.677 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:30.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.678 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:30.678 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:30.678 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:30.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:30.679 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:30Z" 2023-10-17T19:25:30.679 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:30.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:30Z" } 2023-10-17T19:25:30.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:30.681 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:30.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:30.682 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:35.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:35.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:35.684 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:35.745 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:35.745 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:35.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:35.778+0000 7f6102d67780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:35.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:35.779+0000 7f6102d67780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:35.795 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:35.795 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:35.795 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:35.795 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:35.796 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:35.796 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:35.796 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:35.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.797 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:35.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:35.797 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:35.797 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:35.798 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:35.798 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:35.798 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:35.798 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:35.798 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:35.799 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:35.799 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:35.799 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:35.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:35.801 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:35.801 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:35.802 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:35.802 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:35.802 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:35.802 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:35.802 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:35.803 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:35.803 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:35.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:35.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:35.804 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:35.804 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:35.805 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:35.805 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:35.805 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:35.805 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:35.806 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:35.806 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:35.806 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:35.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:35.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:35.807 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:35.807 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:35.807 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:35.808 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:35.808 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:35.808 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:35.808 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:35.809 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:35.809 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:35.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:35.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:35.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:35.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.810 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:35.810 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:35.810 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:35.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:35.811 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:35Z" 2023-10-17T19:25:35.811 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:35.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:35Z" } 2023-10-17T19:25:35.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:35.813 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:35.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:35.814 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:40.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:40.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:40.817 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:40.876 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:40.876 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:40.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:40.908+0000 7f07632d0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:40.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:40.909+0000 7f07632d0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:40.925 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:40.926 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:40.926 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:40.926 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:40.926 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:40.926 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:40.927 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:40.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.927 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:40.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:40.928 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:40.928 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:40.928 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:40.928 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:40.928 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:40.929 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:40.929 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:40.929 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:40.929 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:40.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:40.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:40.930 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:40.930 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:40.931 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:40.931 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:40.931 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:40.931 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:40.932 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:40.932 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:40.932 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:40.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:40.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:40.933 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:40.933 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:40.933 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:40.934 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:40.934 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:40.934 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:40.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:40.934 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:40.935 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:40.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:40.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:40.936 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:40.936 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:40.936 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:40.936 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:40.937 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:40.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:40.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:40.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:40.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.939 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:40.939 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:40.939 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:40.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:40.939 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:40Z" 2023-10-17T19:25:40.940 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:40.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:40Z" } 2023-10-17T19:25:40.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:40.942 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:40.942 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:40.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:45.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:45.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:45.945 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:46.004 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:46.004 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:46.037+0000 7f7f6e06b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:46.038+0000 7f7f6e06b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:46.053 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:46.054 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:46.054 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:46.054 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:46.054 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:46.055 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:46.055 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:46.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.055 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:46.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:46.057 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:46.057 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:46.057 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:46.058 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:46.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:46.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:46.059 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:46.059 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:46.059 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:46.059 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:46.059 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:46.060 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:46.060 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:46.060 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:46.060 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:46.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:46.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:46.061 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:46.061 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:46.061 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:46.062 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:46.062 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:46.062 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:46.062 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:46.063 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:46.063 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:46.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:46.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:46.064 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:46.064 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:46.064 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:46.064 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:46.064 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:46.065 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:46.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:46.065 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:46.065 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:46.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:46.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:46.066 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:46.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.066 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:46.067 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:46.067 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:46.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:46.067 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:46Z" 2023-10-17T19:25:46.067 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:46.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:46Z" } 2023-10-17T19:25:46.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:46.070 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:46.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:46.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:51.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:51.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:51.073 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:51.132 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:51.133 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:51.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:51.165+0000 7f281c43a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:51.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:51.167+0000 7f281c43a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:51.181 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:51.182 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:51.182 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:51.182 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:51.182 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:51.183 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:51.183 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:51.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.183 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:51.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:51.184 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:51.184 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:51.184 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:51.184 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:51.184 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:51.185 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:51.185 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:51.185 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:51.185 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:51.186 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:51.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:51.186 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:51.187 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:51.188 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:51.188 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:51.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:51.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:51.189 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:51.189 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:51.189 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:51.189 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:51.189 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:51.190 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:51.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:51.190 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:51.190 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:51.190 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:51.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:51.191 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:51.191 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:51.191 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:51.192 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:51.192 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:51.192 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:51.192 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:51.192 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:51.193 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:51.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:51.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:51.193 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:51.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.194 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:51.194 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:51.194 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:51.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:51.194 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:51Z" 2023-10-17T19:25:51.195 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:51.196 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:51Z" } 2023-10-17T19:25:51.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:51.197 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:51.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:51.197 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:25:56.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:56.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:25:56.199 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:25:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:25:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:25:56.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:56.291+0000 7f30af42d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:56.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:25:56.292+0000 7f30af42d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:25:56.308 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:25:56.308 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:25:56.308 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:25:56.308 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:25:56.309 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:25:56.309 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:25:56.309 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:56.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.309 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:25:56.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:56.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:25:56.310 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:56.310 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:56.311 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:25:56.311 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:56.311 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:25:56.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:56.311 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:25:56.312 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:56.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:56.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:56.312 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:25:56.313 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:56.313 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:56.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:25:56.313 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:56.314 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:56.314 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:56.314 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:25:56.314 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:56.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:56.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:56.315 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:25:56.315 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:56.315 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:56.316 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:25:56.316 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:56.316 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:56.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:56.317 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:25:56.317 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:56.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:56.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:25:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:25:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:25:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:25:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:25:56.318 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:25:56.319 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:25:56.319 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:25:56.319 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:25:56.319 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:25:56.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:56.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:25:56.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:25:56.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.320 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:25:56.321 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:25:56.321 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:25:56.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:25:56.321 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:25:56Z" 2023-10-17T19:25:56.321 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:25:56.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:25:56Z" } 2023-10-17T19:25:56.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:25:56.323 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:56.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:25:56.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:01.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:01.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:01.327 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:01.419+0000 7fee823e2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:01.420+0000 7fee823e2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:01.436 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:01.436 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:01.437 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:01.437 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:01.437 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:01.437 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:01.437 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:01.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.438 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:01.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:01.438 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:01.438 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:01.439 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:01.439 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:01.439 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:01.439 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:01.439 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:01.440 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:01.441 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:01.441 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:01.441 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:01.441 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:01.441 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:01.442 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:01.442 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:01.442 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:01.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:01.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:01.443 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:01.443 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:01.443 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:01.444 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:01.444 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:01.444 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:01.444 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:01.444 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:01.445 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:01.446 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:01.446 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:01.446 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:01.446 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:01.446 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:01.447 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:01.447 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:01.447 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:01.447 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:01.447 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:01.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.448 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:01.448 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:01.448 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:01.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:01.449 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:01Z" 2023-10-17T19:26:01.449 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:01.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:01Z" } 2023-10-17T19:26:01.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:01.451 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:01.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:01.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:06.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:06.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:06.453 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:06.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:06.512 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:06.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:06.545+0000 7f1b21543780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:06.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:06.546+0000 7f1b21543780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:06.562 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:06.562 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:06.562 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:06.563 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:06.563 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:06.563 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:06.563 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:06.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.564 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:06.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:06.564 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:06.564 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:06.565 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:06.565 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:06.565 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:06.565 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:06.566 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:06.566 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:06.566 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:06.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:06.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:06.567 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:06.567 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:06.567 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:06.568 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:06.568 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:06.568 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:06.568 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:06.568 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:06.569 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:06.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:06.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:06.569 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:06.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:06.570 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:06.570 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:06.570 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:06.571 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:06.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:06.571 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:06.571 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:06.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:06.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:06.572 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:06.572 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:06.572 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:06.573 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:06.573 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:06.573 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:06.573 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:06.574 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:06.574 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:06.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:06.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:06.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:06.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.575 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:06.575 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:06.575 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:06.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:06.576 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:06Z" 2023-10-17T19:26:06.576 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:06.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:06Z" } 2023-10-17T19:26:06.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:06.578 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:06.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:06.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:11.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:11.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:11.581 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:11.640 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:11.641 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:11.674 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:11.672+0000 7f3159041780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:11.675 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:11.673+0000 7f3159041780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:11.688 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:11.689 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:11.689 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:11.689 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:11.690 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:11.690 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:11.691 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:11.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.691 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:11.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:11.692 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:11.693 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:11.693 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:11.693 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:11.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:11.694 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:11.695 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:11.695 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:11.695 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:11.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:11.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:11.697 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:11.697 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:11.697 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:11.697 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:11.698 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:11.698 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:11.698 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:11.698 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:11.699 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:11.699 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:11.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:11.700 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:11.700 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:11.700 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:11.701 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:11.701 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:11.701 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:11.702 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:11.702 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:11.702 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:11.703 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:11.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:11.704 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:11.704 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:11.705 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:11.705 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:11.705 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:11.706 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:11.706 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:11.707 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:11.707 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:11.707 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:11.708 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:11.708 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:11.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.709 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:11.709 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:11.712 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:11.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:11.714 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:11Z" 2023-10-17T19:26:11.715 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:11.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:11Z" } 2023-10-17T19:26:11.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:11.717 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:11.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:11.717 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:16.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:16.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:16.719 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:16.778 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:16.778 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:16.810+0000 7f5c02a86780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:16.812+0000 7f5c02a86780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:16.828 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:16.828 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:16.828 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:16.829 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:16.829 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:16.829 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:16.829 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:16.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.830 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:16.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:16.830 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:16.830 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:16.831 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:16.831 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:16.831 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:16.831 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:16.832 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:16.832 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:16.832 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:16.832 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:16.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:16.833 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:16.833 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:16.833 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:16.834 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:16.834 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:16.834 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:16.834 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:16.834 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:16.835 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:16.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:16.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:16.835 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:16.836 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:16.836 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:16.836 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:16.836 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:16.837 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:16.837 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:16.837 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:16.837 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:16.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:16.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:16.838 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:16.838 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:16.839 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:16.839 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:16.839 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:16.839 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:16.839 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:16.840 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:16.840 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:16.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:16.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:16.840 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:16.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.841 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:16.841 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:16.841 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:16.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:16.842 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:16Z" 2023-10-17T19:26:16.842 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:16.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:16Z" } 2023-10-17T19:26:16.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:16.844 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:16.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:16.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:21.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:21.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:21.847 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:21.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:21.907 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:21.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:21.941+0000 7fcfb206d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:21.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:21.942+0000 7fcfb206d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:21.957 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:21.957 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:21.957 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:21.957 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:21.958 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:21.958 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:21.958 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:21.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.959 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:21.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:21.959 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:21.959 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:21.959 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:21.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:21.960 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:21.960 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:21.960 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:21.961 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:21.961 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:21.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:21.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:21.962 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:21.962 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:21.962 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:21.963 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:21.963 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:21.963 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:21.964 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:21.964 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:21.964 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:21.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:21.966 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:21.966 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:21.966 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:21.967 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:21.967 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:21.967 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:21.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:21.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:21.968 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:21.968 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:21.969 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:21.969 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:21.969 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:21.969 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:21.969 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:21.970 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:21.970 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:21.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:21.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:21.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:21.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.971 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:21.971 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:21.972 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:21.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:21.972 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:21Z" 2023-10-17T19:26:21.972 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:21.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:21Z" } 2023-10-17T19:26:21.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:21.974 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:21.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:21.975 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:26.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:26.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:26.977 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:27.036 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:27.037 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:27.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:27.073+0000 7f402c19c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:27.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:27.075+0000 7f402c19c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:27.091 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:27.091 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:27.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:27.093 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:27.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:27.096 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:27.097 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:27.098 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:27.099 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:27.101 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:27.102 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:27.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:27.105 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:27Z" 2023-10-17T19:26:27.105 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:27.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:27Z" } 2023-10-17T19:26:27.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:27.107 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:27.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:27.108 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:32.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:32.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:32.110 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:32.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:32.202+0000 7f2551b63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:32.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:32.207+0000 7f2551b63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:32.223 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:32.223 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:32.223 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:32.224 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:32.224 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:32.224 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:32.224 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:32.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.225 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:32.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:32.225 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:32.225 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:32.225 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:32.226 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:32.226 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:32.226 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:32.226 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:32.227 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:32.227 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:32.227 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:32.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:32.228 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:32.228 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:32.228 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:32.228 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:32.229 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:32.229 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:32.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:32.229 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:32.229 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:32.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:32.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:32.230 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:32.230 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:32.231 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:32.231 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:32.231 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:32.231 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:32.231 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:32.232 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:32.232 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:32.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:32.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:32.233 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:32.233 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:32.233 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:32.233 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:32.234 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:32.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:32.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:32.235 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:32.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.235 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:32.236 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:32.236 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:32.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:32.236 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:32Z" 2023-10-17T19:26:32.236 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:32.237 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:32Z" } 2023-10-17T19:26:32.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:32.239 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:32.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:32.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:37.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:37.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:37.243 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:37.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:37.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:37.336 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:37.335+0000 7f535af25780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:37.337 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:37.336+0000 7f535af25780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:37.352 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:37.352 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:37.353 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:37.355 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:37.356 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:37.357 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:37.358 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:37.359 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:37.360 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:37.361 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:37.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:37.363 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:37.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:37.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:37.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:37.367 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:37.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:37.368 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:37Z" 2023-10-17T19:26:37.368 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:37.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:37Z" } 2023-10-17T19:26:37.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:37.370 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:37.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:37.371 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:42.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:42.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:42.374 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:42.434 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:42.434 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:42.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:42.466+0000 7f935d11b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:42.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:42.467+0000 7f935d11b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:42.483 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:42.484 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:42.484 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:42.484 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:42.484 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:42.490 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:42.490 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:42.490 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:42.490 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:42.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:42.491 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:42.491 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:42.491 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:42.492 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:42.492 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:42.492 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:42.492 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:42.492 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:42.493 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:42.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:42.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:42.494 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:42.494 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:42.494 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:42.494 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:42.494 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:42.495 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:42.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:42.495 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:42.495 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:42.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:42.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:42.496 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:42.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.496 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:42.497 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:42.497 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:42.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:42.497 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:42Z" 2023-10-17T19:26:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:42.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:42Z" } 2023-10-17T19:26:42.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:42.500 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:42.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:42.500 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:47.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:47.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:47.502 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:47.562 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:47.562 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:47.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:47.596+0000 7fcb5f0ca780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:47.597+0000 7fcb5f0ca780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:47.613 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:47.614 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.615 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:47.616 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:47.617 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:47.619 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:47.620 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:47.621 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:47.622 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:47.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:47.624 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:47.625 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:47.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.627 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:47.628 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:47.629 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:47Z" 2023-10-17T19:26:47.629 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:47.631 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:47Z" } 2023-10-17T19:26:47.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:47.632 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:47.633 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:47.633 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:52.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:52.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:52.635 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:52.727+0000 7f47d8d4e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:52.728+0000 7f47d8d4e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:52.744 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:52.744 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:52.744 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:52.745 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:52.745 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:52.745 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:52.745 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:52.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.746 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:52.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:52.746 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:52.747 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:52.747 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:52.747 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:52.747 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:52.747 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:52.748 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:52.748 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:52.748 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:52.748 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:52.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:52.749 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:52.749 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:52.749 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:52.749 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:52Z" 2023-10-17T19:26:52.757 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:52.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:52Z" } 2023-10-17T19:26:52.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:52.760 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:52.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:52.760 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:26:57.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:57.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:26:57.763 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:26:57.822 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:26:57.822 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:26:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:57.855+0000 7f140e096780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:26:57.856+0000 7f140e096780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:26:57.871 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:26:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:26:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:26:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:26:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:26:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:26:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:26:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:26:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:26:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:26:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:26:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:26:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:26:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:26:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:26:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:26:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:26:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:26:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:26:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:26:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:26:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:26:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:26:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:26:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:26:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:26:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:26:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:26:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:26:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:26:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:26:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:26:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:26:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:26:57Z" 2023-10-17T19:26:57.885 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:26:57.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:26:57Z" } 2023-10-17T19:26:57.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:26:57.887 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:57.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:26:57.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:02.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:02.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:02.890 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:02.950 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:02.950 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:02.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:02.983+0000 7f3fbe492780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:02.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:02.984+0000 7f3fbe492780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:02.999 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:03.000 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:03.000 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:03.001 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:03.001 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:03.002 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:03.002 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:03.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.003 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:03.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:03.004 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:03.004 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:03.004 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:03.005 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:03.005 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:03.005 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:03.006 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:03.006 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:03.006 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:03.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:03.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:03.007 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:03.007 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:03.008 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:03.008 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:03.008 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:03.009 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:03.009 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:03.009 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:03.009 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:03.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:03.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:03.010 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:03.010 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:03.011 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:03.011 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:03.011 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:03.011 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:03.011 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:03.012 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:03.012 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:03.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:03.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:03.013 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:03.013 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:03.013 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:03.013 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:03.014 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:03.014 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:03.014 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:03.014 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:03.014 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:03.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:03.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:03.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:03.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.016 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:03.016 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:03.016 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:03.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:03.016 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:02Z" 2023-10-17T19:27:03.017 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:03.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:02Z" } 2023-10-17T19:27:03.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:03.019 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:03.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:03.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:08.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:08.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:08.023 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:08.115 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:08.113+0000 7f4ba85ab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:08.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:08.115+0000 7f4ba85ab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:08.131 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:08Z" 2023-10-17T19:27:08.146 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:08.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:08Z" } 2023-10-17T19:27:08.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:08.148 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:08.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:08.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:13.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:13.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:13.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:13.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:13.242+0000 7fa598292780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:13.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:13.244+0000 7fa598292780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:13.260 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:13.260 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:13.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:13.262 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:13.263 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:13.265 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:13.266 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:13.267 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:13.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:13.270 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:13.271 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:13.272 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:13.273 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:13Z" 2023-10-17T19:27:13.273 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:13.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:13Z" } 2023-10-17T19:27:13.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:13.275 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:13.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:13.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:18.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:18.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:18.279 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:18.338 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:18.338 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:18.373 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:18.372+0000 7f31be7eb780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:18.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:18.373+0000 7f31be7eb780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:18.389 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:18Z" 2023-10-17T19:27:18.403 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:18.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:18Z" } 2023-10-17T19:27:18.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:18.405 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:18.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:18.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:21.019 INFO:tasks.rgw.c1.client.0.smithi049.stdout:2023-10-17T19:27:21.017+0000 7f95bd2fb700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-17T19:27:21.040 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:27:21.038+0000 7fc70d904700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-17T19:27:23.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:23.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:23.408 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:23.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:23.500+0000 7f4e4f55f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:23.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:23.501+0000 7f4e4f55f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:23.517 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:23.521 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:23.521 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:23.521 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:23.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:23.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:23.522 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:23.522 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:23.522 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:23.522 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:23.523 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:23.523 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:23.523 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:23.523 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:23.524 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:23.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:23.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:23.524 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:23.525 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:23.525 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:23.525 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:23.525 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:23.525 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:23.526 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:23.526 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:23.526 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:23.526 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:23.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:23.527 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:23.527 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:23.527 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:23.527 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:23.528 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:23.528 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:23.528 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:23.528 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:23.528 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:23.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:23.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:23.529 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:23.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.529 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:23.530 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:23.530 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:23.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:23.530 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:23Z" 2023-10-17T19:27:23.531 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:23.532 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:23Z" } 2023-10-17T19:27:23.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:23.533 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:23.533 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:23.533 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:28.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:28.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:28.536 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:28.594 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:28.594 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:28.625+0000 7f2003db0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:28.626+0000 7f2003db0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:28.642 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:28.642 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:28.643 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:28.643 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:28.643 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:28.643 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:28.643 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:28.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.644 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:28.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:28.644 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:28.645 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:28.645 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:28.645 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570712.489592_955.1", 2023-10-17T19:27:28.645 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:28.645 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:28.646 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:28.646 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:12.489592Z", 2023-10-17T19:27:28.646 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:28.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:28.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:28.647 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:28.647 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:28.648 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:28.648 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:28.648 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:28.648 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:28.649 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:28.649 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:28.649 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:28.649 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:28.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:28.650 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:28.650 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:28.650 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:28.651 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:28.651 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:28.651 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:28.651 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:28.652 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:28.652 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:28.652 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:28.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:28.653 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:28.653 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:28.653 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:28.653 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:28.654 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:28.654 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:28.654 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:28.654 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:28.654 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:28.655 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:28.655 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:28.655 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:28.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.656 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:28.656 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:28.656 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:28.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:28.656 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:28Z" 2023-10-17T19:27:28.657 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:28.658 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570712.489592_955.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:12.489592Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:28Z" } 2023-10-17T19:27:28.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570712.489592_955.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:12.489592Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:28.659 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:28.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570712.489592_955.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:28.660 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570720.102476_987.1 target=1_1697570712.489592_955.1 2023-10-17T19:27:33.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.662 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:33.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:33.752+0000 7f4d6ebc8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:33.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:33.754+0000 7f4d6ebc8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:33.768 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:33.769 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:33.769 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:33.769 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:33.769 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:33.770 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:33.770 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:33.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.770 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:33.771 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570720.102476_987.1", 2023-10-17T19:27:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.102476Z", 2023-10-17T19:27:33.772 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:33.773 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.774 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:33.776 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:33.777 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:33.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.778 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:33.778 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.779 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:33.779 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:33.779 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.779 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:33.779 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:33.780 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:33.780 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:33.780 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.780 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.780 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:33.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.781 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:33.781 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:33.781 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:33.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.782 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:33Z" 2023-10-17T19:27:33.782 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:33.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570720.102476_987.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:20.102476Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:33Z" } 2023-10-17T19:27:33.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570720.102476_987.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.102476Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:33.784 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:33.784 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:33.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:27:33.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.786 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:33.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:33.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:33.890 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:27:33.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.891 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:33.891 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:33.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:27:33.892 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:27:33.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:33.893 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:33.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:33.893 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:33.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.894 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:27:33.894 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:33.895 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:27:33.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:33.896 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:33.948 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:33.949 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:33.994 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:33.994 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:33.994 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:33.995 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:33.995 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:33.995 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:27:33.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.995 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:33.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.996 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:33.996 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.996 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:33.997 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:33.997 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.997 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:33.997 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:33.997 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:33.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:33.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:33.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:33.998 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:33.999 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:33.999 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:33.999 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040561", 2023-10-17T19:27:33.999 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:33.999 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:34.000 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:34.000 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.141672Z" 2023-10-17T19:27:34.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:34.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:34.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:34.001 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:34.001 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:34.001 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:34.001 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:34.002 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:34.002 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:27:34.002 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:34.002 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:34.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:34.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:34.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:34.003 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:34.004 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:34.004 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:34.004 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:34.004 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:34.004 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:27:34.005 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:34.005 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:34.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:34.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:34.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:34.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:34.006 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:34.006 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:27:34.006 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:27:34.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:34.007 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:33Z" 2023-10-17T19:27:34.007 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:34.008 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040561", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.141672Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:27:33Z" } 2023-10-17T19:27:34.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040561', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.141672Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:27:34.009 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:34.009 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040561', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:34.010 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040702 target=00000000000000000000:00000000000000040561 2023-10-17T19:27:39.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:39.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:39.012 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:39.073 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:39.073 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:39.123 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:39.124 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:39.124 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:39.124 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:39.124 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:39.124 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:27:39.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.125 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:39.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:39.125 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:39.126 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:39.126 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:39.126 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:39.126 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:39.126 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:39.127 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:39.127 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:39.127 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:39.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:39.128 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:39.128 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:39.128 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:39.128 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040561", 2023-10-17T19:27:39.128 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:39.129 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:39.129 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:39.129 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.141672Z" 2023-10-17T19:27:39.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:39.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:39.130 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:39.130 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:39.130 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:39.131 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:39.131 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:39.131 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:27:39.131 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:39.131 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:39.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:39.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:39.132 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:39.133 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:39.133 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:39.133 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:39.133 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:39.133 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:27:39.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:39.134 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:39.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:39.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:39.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:39.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.135 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:39.135 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:27:39.135 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:27:39.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:39.136 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:39Z" 2023-10-17T19:27:39.136 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:39.137 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040561", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.141672Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:27:39Z" } 2023-10-17T19:27:39.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040561', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.141672Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:27:39.138 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:39.138 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040561', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:39.139 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040702 target=00000000000000000000:00000000000000040561 2023-10-17T19:27:44.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.142 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:44.200 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.201 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.252 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:44.252 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:44.253 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:44.253 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:44.253 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:44.253 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:27:44.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.254 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:44.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.254 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:44.255 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.255 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:44.255 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:44.255 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.256 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:44.256 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.256 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:44.256 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.257 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:44.257 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.258 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:44.258 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:27:44.258 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.258 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:44.258 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.259 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:27:44.259 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.260 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:44.260 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.261 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:44.261 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:44.262 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.262 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:27:44.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.263 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:44.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.264 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:44.265 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.265 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:44.266 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:44.266 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.266 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:27:44.267 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.267 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:44.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.268 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.268 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:44.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.269 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:44.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:27:44.270 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:27:44.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.270 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:44Z" 2023-10-17T19:27:44.270 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:44.271 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:27:44Z" } 2023-10-17T19:27:44.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:27:44.273 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:44.273 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:44.273 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:27:44.274 DEBUG:boto:path=/hbmuog-66/ 2023-10-17T19:27:44.274 DEBUG:boto:auth_path=/hbmuog-66/ 2023-10-17T19:27:44.275 DEBUG:boto:Method: HEAD 2023-10-17T19:27:44.275 DEBUG:boto:Path: /hbmuog-66/ 2023-10-17T19:27:44.276 DEBUG:boto:Data: 2023-10-17T19:27:44.276 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.277 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:27:44.277 DEBUG:boto:Port: 8001 2023-10-17T19:27:44.277 DEBUG:boto:Params: {} 2023-10-17T19:27:44.278 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:27:44.279 DEBUG:boto:Token: None 2023-10-17T19:27:44.279 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-66/ 2023-10-17T19:27:44.280 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wRJTdeQDzqeNhAhT8uuZpORK+sI= 2023-10-17T19:27:44.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wRJTdeQDzqeNhAhT8uuZpORK+sI=', 'Content-Length': '0'} 2023-10-17T19:27:44.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bea56315bd85b71a-00652ee030-2061-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.285 DEBUG:boto:path=/hbmuog-66/obj1 2023-10-17T19:27:44.285 DEBUG:boto:auth_path=/hbmuog-66/obj1 2023-10-17T19:27:44.286 DEBUG:boto:Method: HEAD 2023-10-17T19:27:44.286 DEBUG:boto:Path: /hbmuog-66/obj1 2023-10-17T19:27:44.287 DEBUG:boto:Data: 2023-10-17T19:27:44.287 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.287 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:27:44.288 DEBUG:boto:Port: 8001 2023-10-17T19:27:44.288 DEBUG:boto:Params: {} 2023-10-17T19:27:44.289 DEBUG:boto:Token: None 2023-10-17T19:27:44.289 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-66/obj1 2023-10-17T19:27:44.290 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2Yw5FYrvAQwzJXtdcDMO8uKvd1M= 2023-10-17T19:27:44.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2Yw5FYrvAQwzJXtdcDMO8uKvd1M=', 'Content-Length': '0'} 2023-10-17T19:27:44.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-66:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abb3c629a06141ce-00652ee030-2061-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.292 DEBUG:boto:path=/hbmuog-66/obj1 2023-10-17T19:27:44.293 DEBUG:boto:auth_path=/hbmuog-66/obj1 2023-10-17T19:27:44.293 DEBUG:boto:Method: GET 2023-10-17T19:27:44.294 DEBUG:boto:Path: /hbmuog-66/obj1 2023-10-17T19:27:44.294 DEBUG:boto:Data: 2023-10-17T19:27:44.294 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.295 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:27:44.295 DEBUG:boto:Port: 8001 2023-10-17T19:27:44.296 DEBUG:boto:Params: {} 2023-10-17T19:27:44.296 DEBUG:boto:Token: None 2023-10-17T19:27:44.297 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-66/obj1 2023-10-17T19:27:44.297 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4+Cmni3moahIvA4tuW/JgAB1ORM= 2023-10-17T19:27:44.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4+Cmni3moahIvA4tuW/JgAB1ORM=', 'Content-Length': '0'} 2023-10-17T19:27:44.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-66:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d010a0d8a720adfd-00652ee030-2061-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.300 DEBUG:boto:path=/hbmuog-67/ 2023-10-17T19:27:44.301 DEBUG:boto:auth_path=/hbmuog-67/ 2023-10-17T19:27:44.301 DEBUG:boto:Method: HEAD 2023-10-17T19:27:44.301 DEBUG:boto:Path: /hbmuog-67/ 2023-10-17T19:27:44.302 DEBUG:boto:Data: 2023-10-17T19:27:44.302 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:44.303 DEBUG:boto:Port: 8000 2023-10-17T19:27:44.303 DEBUG:boto:Params: {} 2023-10-17T19:27:44.304 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:27:44.304 DEBUG:boto:Token: None 2023-10-17T19:27:44.305 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-67/ 2023-10-17T19:27:44.305 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GS6ifEh39VuY4ihmYJknqCGcVcg= 2023-10-17T19:27:44.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GS6ifEh39VuY4ihmYJknqCGcVcg=', 'Content-Length': '0'} 2023-10-17T19:27:44.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5e827f3418f9c9b-00652ee030-206d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.310 DEBUG:boto:path=/hbmuog-67/obj2 2023-10-17T19:27:44.311 DEBUG:boto:auth_path=/hbmuog-67/obj2 2023-10-17T19:27:44.311 DEBUG:boto:Method: HEAD 2023-10-17T19:27:44.312 DEBUG:boto:Path: /hbmuog-67/obj2 2023-10-17T19:27:44.312 DEBUG:boto:Data: 2023-10-17T19:27:44.313 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.313 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:44.313 DEBUG:boto:Port: 8000 2023-10-17T19:27:44.314 DEBUG:boto:Params: {} 2023-10-17T19:27:44.314 DEBUG:boto:Token: None 2023-10-17T19:27:44.315 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-67/obj2 2023-10-17T19:27:44.315 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FwmfLELDQiV8TpZbuqsR2Rt/8+U= 2023-10-17T19:27:44.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FwmfLELDQiV8TpZbuqsR2Rt/8+U=', 'Content-Length': '0'} 2023-10-17T19:27:44.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-67:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003efba3a4797b3ef3-00652ee030-206d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.318 DEBUG:boto:path=/hbmuog-67/obj2 2023-10-17T19:27:44.319 DEBUG:boto:auth_path=/hbmuog-67/obj2 2023-10-17T19:27:44.319 DEBUG:boto:Method: GET 2023-10-17T19:27:44.320 DEBUG:boto:Path: /hbmuog-67/obj2 2023-10-17T19:27:44.320 DEBUG:boto:Data: 2023-10-17T19:27:44.321 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.321 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:44.321 DEBUG:boto:Port: 8000 2023-10-17T19:27:44.322 DEBUG:boto:Params: {} 2023-10-17T19:27:44.322 DEBUG:boto:Token: None 2023-10-17T19:27:44.323 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-67/obj2 2023-10-17T19:27:44.323 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zyNntesV8NDj/bCHvyr+Qm9Qeec= 2023-10-17T19:27:44.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zyNntesV8NDj/bCHvyr+Qm9Qeec=', 'Content-Length': '0'} 2023-10-17T19:27:44.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:25:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '8f644c5a-db9c-4565-b32b-e3a6790e3bcc:hbmuog-67:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022be4673690ddd2c-00652ee030-206d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.327 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:44.381 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.381 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.428 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:27:44.428 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-10-17T19:27:44.430 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-17T19:27:44.431 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-17T19:27:44.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.432 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:44.481 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.482 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.514 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:27:44.528 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:27:44.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.529 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570720.102476_987.1", 2023-10-17T19:27:44.529 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:25:20.102476Z" 2023-10-17T19:27:44.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.530 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:44.530 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:27:44.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.531 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:44.532 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:27:44.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.533 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:44.533 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:27:44.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.533 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:27:44.535 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:44.535 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:27:44.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.537 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:44.588 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.588 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:44.616+0000 7ff74af7d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:44.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:44.617+0000 7ff74af7d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:44.632 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:44.633 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:44.633 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:44.633 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:44.634 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:44.634 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:44.634 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:44.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.635 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:44.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.635 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:44.635 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.636 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:44.636 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570720.102476_987.1", 2023-10-17T19:27:44.636 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.636 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:44.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.637 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.102476Z", 2023-10-17T19:27:44.637 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:44.637 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.638 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:44.638 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.639 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:44.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:44.639 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.639 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:44.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:44.640 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:44.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:44.642 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:44.642 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.642 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:44.642 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.643 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:44.644 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:44.644 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:44.644 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:44.644 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.646 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:44.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.646 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:44.647 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:44.647 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:44.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.647 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:44Z" 2023-10-17T19:27:44.647 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:44.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570720.102476_987.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:25:20.102476Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:44Z" } 2023-10-17T19:27:44.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570720.102476_987.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.102476Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:44.650 DEBUG:rgw_multi.tests:log_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:44.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570720.102476_987.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:44.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:27:44.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.652 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:44.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.746 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:27:44.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:44.749 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:44.796 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.796 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.841 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:27:44.842 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-68 2023-10-17T19:27:44.843 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:27:44.843 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:27:44.843 DEBUG:boto:Method: PUT 2023-10-17T19:27:44.844 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:27:44.844 DEBUG:boto:Data: 2023-10-17T19:27:44.845 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.845 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:44.846 DEBUG:boto:Port: 8000 2023-10-17T19:27:44.846 DEBUG:boto:Params: {} 2023-10-17T19:27:44.847 DEBUG:boto:Token: None 2023-10-17T19:27:44.847 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-68/ 2023-10-17T19:27:44.848 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VZx/xo5jmdwe5HAe1ujJ+Nqbhk8= 2023-10-17T19:27:44.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VZx/xo5jmdwe5HAe1ujJ+Nqbhk8=', 'Content-Length': '0'} 2023-10-17T19:27:44.865 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076f69e851669a3b6-00652ee030-206d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.865 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-69 2023-10-17T19:27:44.865 DEBUG:boto:path=/hbmuog-69/ 2023-10-17T19:27:44.866 DEBUG:boto:auth_path=/hbmuog-69/ 2023-10-17T19:27:44.866 DEBUG:boto:Method: PUT 2023-10-17T19:27:44.867 DEBUG:boto:Path: /hbmuog-69/ 2023-10-17T19:27:44.867 DEBUG:boto:Data: 2023-10-17T19:27:44.868 DEBUG:boto:Headers: {} 2023-10-17T19:27:44.868 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:44.868 DEBUG:boto:Port: 8000 2023-10-17T19:27:44.869 DEBUG:boto:Params: {} 2023-10-17T19:27:44.869 DEBUG:boto:Token: None 2023-10-17T19:27:44.870 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:27:44 GMT /hbmuog-69/ 2023-10-17T19:27:44.871 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:oFQhiAogH9265KdGvgADm2DjXJA= 2023-10-17T19:27:44.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:44 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oFQhiAogH9265KdGvgADm2DjXJA=', 'Content-Length': '0'} 2023-10-17T19:27:44.886 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005b40700459f4d7f-00652ee030-206d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:44.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.888 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:44.941 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:44.941 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:44.972 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:27:44.986 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:27:44.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.987 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:27:44.987 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:44.884595Z" 2023-10-17T19:27:44.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.988 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:44.988 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:27:44.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.989 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:44.989 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:27:44.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:44.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:44.990 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:44.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:27:44.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:44.990 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:27:44.991 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:44.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:27:44.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:44.993 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:45.044 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:45.044 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:45.080+0000 7fe786714780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:45.081+0000 7fe786714780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:45.096 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.859713_1004.1", 2023-10-17T19:27:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:44.859713Z", 2023-10-17T19:27:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:45.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:45.110 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:45Z" 2023-10-17T19:27:45.110 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:45.111 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570864.859713_1004.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:27:44.859713Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:45Z" } 2023-10-17T19:27:45.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570864.859713_1004.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:27:44.859713Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:45.112 DEBUG:rgw_multi.tests:log_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:45.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570864.859713_1004.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:45.113 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570864.884595_1009.1 target=1_1697570864.859713_1004.1 2023-10-17T19:27:50.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:50.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:50.116 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:50.176 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.176 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:50.208+0000 7f82a3b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:50.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:50.209+0000 7f82a3b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:50.225 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:50.225 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:50.226 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:50.226 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:50.226 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:50.226 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:50.226 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.227 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:50.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.227 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:50.227 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:50.228 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:50.228 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:27:50.228 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:50.228 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:50.228 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:50.229 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:44.884595Z", 2023-10-17T19:27:50.229 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.230 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:50.230 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:50.230 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:50.230 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.232 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:50.232 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:50.233 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:50.233 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:50.233 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:50.233 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:50.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:50.234 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:50.234 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.235 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:50.235 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:50.236 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:50.236 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:50.236 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:50.236 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:50.236 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.237 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.238 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:50.238 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:50.238 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:50.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.239 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:50Z" 2023-10-17T19:27:50.239 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:50.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570864.884595_1009.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:27:44.884595Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:50Z" } 2023-10-17T19:27:50.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570864.884595_1009.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:27:44.884595Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:50.241 DEBUG:rgw_multi.tests:log_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:50.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:50.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:27:50.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.243 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.297 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.297 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.344 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:27:50.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.346 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.403 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.451 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:27:50.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hbmuog-68', '--dest-bucket=hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hbmuog-68', '--dest-bucket=hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.452 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=hbmuog-68 --dest-bucket=hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.509 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.510 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.558 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-68","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:27:50.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.560 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.612 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.613 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.664 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-68","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:27:50.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.665 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.712 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.712 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.762 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-68","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:27:50.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.763 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.813 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.813 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:50.865 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 16, 2023-10-17T19:27:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:27:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:27:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:27:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:27:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:27:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:27:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:27:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:27:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:27:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:27:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:27:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:27:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:27:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:27:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:27:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:27:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:27:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:27:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:27:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:27:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:27:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:27:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:27:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:27:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:27:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:27:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:27:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:27:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:27:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:27:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:27:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:27:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:27:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:27:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:27:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:27:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:27:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:27:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:27:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:27:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:27:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:27:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:27:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:27:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:27:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:27:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:27:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:27:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:27:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:27:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:27:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:27:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:27:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:27:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:27:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-68", 2023-10-17T19:27:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:27:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:27:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:27:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-68", 2023-10-17T19:27:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:27:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:27:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:27:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:27:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:27:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:27:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:27:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:27:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:27:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:27:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:27:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:27:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:27:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:27:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:27:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:27:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:27:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:27:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:27:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:27:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:27:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:27:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:27:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:27:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:27:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:27:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:27:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:27:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:27:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:27:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:27:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:27:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:27:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:27:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:27:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:27:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:27:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:27:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:27:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:27:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:27:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:27:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:27:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:27:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:27:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:27:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:27:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:27:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:27:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:27:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:27:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:27:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:27:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:27:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:27:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:27:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:27:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:27:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:27:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:27:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:27:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:27:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:27:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:27:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:27:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:27:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:27:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:27:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:27:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:27:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:27:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:27:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:27:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:27:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:27:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:27:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:27:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.918 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:27:50.918 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:27:50.918 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:27:50.918 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:27:50.918 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:27:50.919 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:27:50.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.919 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:27:50.920 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:27:50.920 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:27:50.920 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:27:50.920 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:27:50.920 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:27:50.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.921 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:27:50.921 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:27:50.921 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:27:50.921 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:27:50.922 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:27:50.922 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:27:50.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.923 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:27:50.923 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:27:50.923 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:27:50.923 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:27:50.924 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:27:50.924 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:27:50.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:50.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:50.925 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:27:50.925 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:27:50.926 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:50.926 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:50.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:27:50.928 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:27:50.980 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:50.981 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:51.074 INFO:teuthology.orchestra.run.smithi049.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"hbmuog-68"},"dest":{"zone":"a1","bucket":"hbmuog-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"hbmuog-68"},"dest":{"zone":"a2","bucket":"hbmuog-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-10-17T19:27:51.075 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:27:51.075 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:27:51.076 DEBUG:boto:Method: HEAD 2023-10-17T19:27:51.076 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:27:51.077 DEBUG:boto:Data: 2023-10-17T19:27:51.077 DEBUG:boto:Headers: {} 2023-10-17T19:27:51.077 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.078 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.078 DEBUG:boto:Params: {} 2023-10-17T19:27:51.079 DEBUG:boto:Token: None 2023-10-17T19:27:51.079 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/ 2023-10-17T19:27:51.080 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U= 2023-10-17T19:27:51.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U=', 'Content-Length': '0'} 2023-10-17T19:27:51.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a4faa1dbf3c926-00652ee037-21ed-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.233 DEBUG:boto:path=/hbmuog-68/obj1 2023-10-17T19:27:51.233 DEBUG:boto:auth_path=/hbmuog-68/obj1 2023-10-17T19:27:51.233 DEBUG:boto:Method: PUT 2023-10-17T19:27:51.234 DEBUG:boto:Path: /hbmuog-68/obj1 2023-10-17T19:27:51.234 DEBUG:boto:Data: 2023-10-17T19:27:51.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:27:51.235 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.236 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.236 DEBUG:boto:Params: {} 2023-10-17T19:27:51.237 DEBUG:boto:Token: None 2023-10-17T19:27:51.237 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/obj1 2023-10-17T19:27:51.238 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AhRafmm1cE52Vj6Y9dx+CzW0bBU= 2023-10-17T19:27:51.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AhRafmm1cE52Vj6Y9dx+CzW0bBU='} 2023-10-17T19:27:51.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ff7c8dbc96a6f19-00652ee037-21ed-a1'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.259 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:27:51.260 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:27:51.260 DEBUG:boto:Method: HEAD 2023-10-17T19:27:51.260 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:27:51.261 DEBUG:boto:Data: 2023-10-17T19:27:51.261 DEBUG:boto:Headers: {} 2023-10-17T19:27:51.263 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.263 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.264 DEBUG:boto:Params: {} 2023-10-17T19:27:51.264 DEBUG:boto:Token: None 2023-10-17T19:27:51.265 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/ 2023-10-17T19:27:51.265 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U= 2023-10-17T19:27:51.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U=', 'Content-Length': '0'} 2023-10-17T19:27:51.270 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d28312de6c8eb83c-00652ee037-21ed-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.271 DEBUG:boto:path=/hbmuog-68/obj2 2023-10-17T19:27:51.271 DEBUG:boto:auth_path=/hbmuog-68/obj2 2023-10-17T19:27:51.272 DEBUG:boto:Method: PUT 2023-10-17T19:27:51.272 DEBUG:boto:Path: /hbmuog-68/obj2 2023-10-17T19:27:51.272 DEBUG:boto:Data: 2023-10-17T19:27:51.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:27:51.273 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.274 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.274 DEBUG:boto:Params: {} 2023-10-17T19:27:51.274 DEBUG:boto:Token: None 2023-10-17T19:27:51.275 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/obj2 2023-10-17T19:27:51.276 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zkSH75chJoDfSSvshrYpE4v5KS0= 2023-10-17T19:27:51.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zkSH75chJoDfSSvshrYpE4v5KS0='} 2023-10-17T19:27:51.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007a0ed1c0dda2791-00652ee037-21ed-a1'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.323 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:27:51.324 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:27:51.324 DEBUG:boto:Method: HEAD 2023-10-17T19:27:51.325 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:27:51.326 DEBUG:boto:Data: 2023-10-17T19:27:51.326 DEBUG:boto:Headers: {} 2023-10-17T19:27:51.327 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.327 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.327 DEBUG:boto:Params: {} 2023-10-17T19:27:51.328 DEBUG:boto:Token: None 2023-10-17T19:27:51.328 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/ 2023-10-17T19:27:51.329 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U= 2023-10-17T19:27:51.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s0mFjUjK+cr+7Xx/6oNUxCfJo8U=', 'Content-Length': '0'} 2023-10-17T19:27:51.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1520f8f1fb26597-00652ee037-21ed-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.335 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:27:51.335 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:27:51.336 DEBUG:boto:Method: PUT 2023-10-17T19:27:51.336 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:27:51.336 DEBUG:boto:Data: 2023-10-17T19:27:51.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:27:51.337 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.338 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.338 DEBUG:boto:Params: {} 2023-10-17T19:27:51.339 DEBUG:boto:Token: None 2023-10-17T19:27:51.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-68/obj3 2023-10-17T19:27:51.340 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:9o6RZm0JJoDWz2qVKD1nmFWyev4= 2023-10-17T19:27:51.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9o6RZm0JJoDWz2qVKD1nmFWyev4='} 2023-10-17T19:27:51.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7e70a1082f2b7ce-00652ee037-21ed-a1'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.353 DEBUG:boto:path=/hbmuog-69/ 2023-10-17T19:27:51.353 DEBUG:boto:auth_path=/hbmuog-69/ 2023-10-17T19:27:51.354 DEBUG:boto:Method: HEAD 2023-10-17T19:27:51.354 DEBUG:boto:Path: /hbmuog-69/ 2023-10-17T19:27:51.354 DEBUG:boto:Data: 2023-10-17T19:27:51.355 DEBUG:boto:Headers: {} 2023-10-17T19:27:51.355 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.356 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.356 DEBUG:boto:Params: {} 2023-10-17T19:27:51.356 DEBUG:boto:Token: None 2023-10-17T19:27:51.357 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-69/ 2023-10-17T19:27:51.357 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eXlmrD9n1/FgxDH/lTLNmwXyi6I= 2023-10-17T19:27:51.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eXlmrD9n1/FgxDH/lTLNmwXyi6I=', 'Content-Length': '0'} 2023-10-17T19:27:51.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b1143bd88a9e777-00652ee037-21ed-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.364 DEBUG:boto:path=/hbmuog-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-17T19:27:51.364 DEBUG:boto:auth_path=/hbmuog-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-17T19:27:51.365 DEBUG:boto:Method: PUT 2023-10-17T19:27:51.365 DEBUG:boto:Path: /hbmuog-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-17T19:27:51.365 DEBUG:boto:Data: 2023-10-17T19:27:51.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:27:51.366 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:27:51.367 DEBUG:boto:Port: 8000 2023-10-17T19:27:51.367 DEBUG:boto:Params: {} 2023-10-17T19:27:51.368 DEBUG:boto:Token: None 2023-10-17T19:27:51.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:27:51 GMT /hbmuog-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-17T19:27:51.369 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NvlSItnN7nhWq5dykd3sYVMwT90= 2023-10-17T19:27:51.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:27:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NvlSItnN7nhWq5dykd3sYVMwT90='} 2023-10-17T19:27:51.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006666913615f64ce3-00652ee037-21ed-a1'), ('Date', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:27:51.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.421 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:51.494 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:51.494 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:51.546 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:27:51.565 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:27:51.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.566 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:27:51.566 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:44.884595Z" 2023-10-17T19:27:51.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.567 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:51.568 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:27:51.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.569 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:51.569 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:27:51.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.570 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:51.570 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:27:51.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.571 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:27:51.572 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:51.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:27:51.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.575 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:51.635 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:51.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:51.673 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:51.672+0000 7f4b5f6e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:51.674 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:27:51.673+0000 7f4b5f6e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:27:51.691 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:51.691 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:51.692 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:51.692 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:51.692 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:51.692 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:27:51.693 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:51.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.693 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:51.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.693 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:51.694 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.694 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:51.694 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:27:51.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.695 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:27:51.695 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.695 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:44.884595Z", 2023-10-17T19:27:51.695 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:51.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:51.697 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:27:51.697 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.697 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:51.697 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.698 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:27:51.698 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:51.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.699 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:51.699 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.699 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:51.699 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:27:51.700 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.700 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:51.700 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.700 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:27:51.700 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.702 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:27:51.702 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:27:51.702 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.702 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:51.703 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.703 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:27:51.703 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:27:51.703 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.703 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.704 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:51.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.704 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:51.704 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:27:51.704 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:27:51.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.705 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:51Z" 2023-10-17T19:27:51.705 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:51.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570864.884595_1009.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:27:44.884595Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:27:51Z" } 2023-10-17T19:27:51.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570864.884595_1009.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:27:44.884595Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:27:51.707 DEBUG:rgw_multi.tests:log_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:51.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:27:51.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:27:51.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.709 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:51.759 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:51.759 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:51.808 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:27:51.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.809 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:27:51.809 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:27:51.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.809 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:27:51.810 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:27:51.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.810 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:51.811 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:51.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.811 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:27:51.811 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:27:51.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.812 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:27:51.813 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:27:51.813 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:27:51.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:51.815 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:51.868 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:51.868 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:51.913 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:51.914 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:51.914 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:51.914 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:51.914 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:51.914 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:27:51.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.915 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:51.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.915 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:51.916 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.916 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:51.916 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:51.916 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.917 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:51.917 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.917 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:51.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.918 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:51.918 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.918 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:51.919 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:27:51.919 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.919 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:51.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.919 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:27:51.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.920 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:51.920 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.921 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:51.921 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:51.921 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.921 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:27:51.922 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.922 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:51.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:51.923 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:51.924 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:51.924 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:51.925 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:51.925 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:51.925 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:27:51.925 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:51.925 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:51.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:51.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:51.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.927 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:51.927 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:27:51.927 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:27:51.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:51.927 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:51Z" 2023-10-17T19:27:51.928 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:51.929 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:27:51Z" } 2023-10-17T19:27:51.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:27:51.930 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:27:51.930 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:51.931 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:27:56.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:56.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:27:56.934 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:27:56.994 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:27:56.995 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:27:57.045 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:27:57.045 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:27:57.046 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:27:57.046 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:27:57.046 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:27:57.046 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:27:57.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.047 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:27:57.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:57.047 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:27:57.047 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:57.047 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:57.048 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:27:57.048 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:57.048 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:57.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:57.049 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:27:57.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:57.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:57.049 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:27:57.050 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:57.050 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:57.050 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:27:57.050 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:57.051 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:27:57.051 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:57.051 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:27:57.051 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:57.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:57.052 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:27:57.052 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:57.052 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:57.052 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:27:57.053 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:57.053 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:27:57.053 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:57.053 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:27:57.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:57.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:27:57.054 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:27:57.054 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:27:57.054 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:27:57.055 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:27:57.055 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:27:57.055 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:27:57.056 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:27:57.056 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:27:57.056 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:57.056 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:27:57.056 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:27:57.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.057 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:27:57.057 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:27:57.057 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:27:57.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:27:57.058 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:27:57Z" 2023-10-17T19:27:57.058 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:27:57.059 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:27:57Z" } 2023-10-17T19:27:57.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:27:57.060 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:27:57.060 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:27:57.061 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:02.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:02.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:02.063 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:02.124 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:02.125 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:02.175 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:02.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.177 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:02.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:02.177 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:02.177 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:02.178 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:02.178 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:02.178 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:02.178 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:02.179 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:02.179 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:02.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:02.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:02.180 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:02.180 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:02.180 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:02.181 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:02.181 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:02.181 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:02.181 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:02.181 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:02.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:02.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:02.182 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:02.183 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:02.184 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:02.184 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:02.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:02.185 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:02.185 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:02.185 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:02.185 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:02.185 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:02.186 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:02.186 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:02.186 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:02.186 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:02.186 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:02.187 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:02.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.187 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:02.187 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:02.187 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:02.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:02.188 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:02Z" 2023-10-17T19:28:02.188 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:02.189 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:02Z" } 2023-10-17T19:28:02.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:02.190 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:02.191 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:02.191 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:07.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:07.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:07.194 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:07.252 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:07.253 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:07.302 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:07Z" 2023-10-17T19:28:07.316 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:07.317 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:07Z" } 2023-10-17T19:28:07.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:07.318 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:07.319 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:07.319 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:12.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:12.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:12.321 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:12.384 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:12.385 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:12.438 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:12.439 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:12.439 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:12.439 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:12.439 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:12.440 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:12.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.440 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:12.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:12.441 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:12.441 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:12.441 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:12.441 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:12.441 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:12.442 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:12.442 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:12.442 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:12.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:12.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:12.443 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:12.443 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:12.443 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:12.444 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:12.444 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:12.444 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:12.444 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:12.445 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:12.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:12.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:12.446 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:12.446 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:12.446 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:12.446 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:12.446 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:12.447 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:12.447 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:12.447 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:12.447 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:12.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:12.448 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:12.448 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:12.448 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:12.449 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:12.449 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:12.449 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:12.449 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:12.450 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:12.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:12.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:12.450 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:12.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:12.452 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:12Z" 2023-10-17T19:28:12.452 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:12.453 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:12Z" } 2023-10-17T19:28:12.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:12.454 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:12.454 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:12.455 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:17.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:17.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:17.457 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:17.568 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:17Z" 2023-10-17T19:28:17.581 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:17.582 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:17Z" } 2023-10-17T19:28:17.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:17.583 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:17.584 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:17.584 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:22.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:22.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:22.587 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:22.650 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:22.651 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:22.700 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:22Z" 2023-10-17T19:28:22.713 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:22.715 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:22Z" } 2023-10-17T19:28:22.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:22.716 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:22.716 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:22.717 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:27.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:27.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:27.719 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:27.781 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:27.781 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:27.834 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:27.836 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:27.836 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:27.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.836 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:27.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:27.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:27.837 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:27.837 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:27.837 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:27.838 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:27.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:27.838 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:27.838 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:27.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:27.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:27.839 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:27.839 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:27.840 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:27.840 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:27.840 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:27.840 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:27.841 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:27.841 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:27.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:27.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:27.842 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:27.842 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:27.842 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:27.842 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:27.843 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:27.843 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:27.843 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:27.843 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:27.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:27.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:27.844 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:27.844 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:27.844 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:27.845 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:27.845 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:27.845 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:27.845 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:27.846 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:27.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:27.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:27.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:27.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.847 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:27.847 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:27.847 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:27.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:27.847 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:27Z" 2023-10-17T19:28:27.848 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:27.849 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:27Z" } 2023-10-17T19:28:27.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:27.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:27.850 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:27.850 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:32.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:32.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:32.853 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:32.913 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:32.914 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:32Z" 2023-10-17T19:28:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:32.978 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:32Z" } 2023-10-17T19:28:32.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:32.979 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:32.980 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:32.980 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:37.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:37.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:37.983 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:38.044 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:38.044 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:38.094 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:38.097 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:38.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:38.099 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:38.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:38.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:38.102 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:38.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:38.104 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:38.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.106 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:38.107 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:38.107 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:38.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:38.107 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:38Z" 2023-10-17T19:28:38.107 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:38.109 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:38Z" } 2023-10-17T19:28:38.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:38.110 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:38.110 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:38.110 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:43.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:43.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:43.112 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:43.172 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:43.173 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:43.224 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:43Z" 2023-10-17T19:28:43.237 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:43.239 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:43Z" } 2023-10-17T19:28:43.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:43.240 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:43.240 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:43.240 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:48.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:48.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:48.243 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:48.306 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:48.309 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:48.364 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:48.365 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:48.366 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:48.367 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:48.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:48.369 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:48.370 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:48.371 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:48.373 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:48.374 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:48.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:48.376 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:48.377 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:48Z" 2023-10-17T19:28:48.377 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:48.378 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:48Z" } 2023-10-17T19:28:48.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:48.379 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:48.379 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:48.380 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:53.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:53.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:53.382 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:53.443 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:53.443 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:53.495 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:53.495 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:53.496 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:53.496 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:53.496 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:53.496 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:53.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.497 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:53.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:53.497 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:53.497 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:53.498 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:53.498 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:53.498 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:53.498 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:53.499 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:53.499 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:53.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:53.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:53.500 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:53.500 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:53.500 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:53.500 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:53.501 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:53.501 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:53.501 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:53.501 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:53.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:53Z" 2023-10-17T19:28:53.508 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:53.509 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:53Z" } 2023-10-17T19:28:53.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:53.510 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:53.511 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:53.511 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:28:58.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:58.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:28:58.514 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:28:58.575 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:28:58.575 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:28:58.629 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:28:58.629 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:28:58.630 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:28:58.630 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:28:58.630 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:28:58.630 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:28:58.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.631 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:28:58.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:58.631 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:28:58.631 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:58.632 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:58.632 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:28:58.632 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:58.632 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:58.632 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:58.633 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:28:58.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:58.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:58.633 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:28:58.634 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:58.634 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:58.634 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:28:58.634 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:58.634 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:28:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:28:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:28:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:28:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:28:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:28:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:28:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:28:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:28:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:28:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:28:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:28:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:28:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:28:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:28:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:28:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:28:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:28:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:28:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:28:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:28:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:28:58Z" 2023-10-17T19:28:58.642 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:28:58.643 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:28:58Z" } 2023-10-17T19:28:58.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:28:58.644 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:28:58.644 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:28:58.644 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:29:03.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:03.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:03.646 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:03.709 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:03.709 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:03.760 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:03.760 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:03.760 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:03.760 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:03.761 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:03.761 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:03.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.761 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:03.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:03.762 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:03.762 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:03.762 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:03.762 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:29:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:29:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:29:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:29:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:29:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:29:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:03Z" 2023-10-17T19:29:03.772 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:03.773 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:03Z" } 2023-10-17T19:29:03.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:29:03.774 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:03.774 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:29:03.775 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:29:08.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:08.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:08.777 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:08.889 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:08.889 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:08.889 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:08.890 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:08.890 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:08.890 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:08.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.891 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:08.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:08.891 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:08.891 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:08.891 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:08.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:29:08.892 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:08.892 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:08.892 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:08.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:29:08.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:08.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:08.893 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:08.894 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:08.894 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:08.894 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:29:08.894 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:08.895 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:08.895 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:08.895 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:29:08.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:08.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:08.896 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:08.896 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:08.896 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:29:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:29:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:08.902 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:08Z" 2023-10-17T19:29:08.902 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:08.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:08Z" } 2023-10-17T19:29:08.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:29:08.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:08.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:29:08.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:29:13.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:13.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:13.907 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:13.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:13.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:14.019 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:14.020 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:14.020 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:14.020 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:14.020 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:14.020 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:14.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.021 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:14.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:14.021 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:14.022 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:14.022 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:14.022 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:29:14.022 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:14.023 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:14.023 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:14.023 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:29:14.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:14.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:14.024 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:14.024 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:14.024 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:14.024 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:29:14.025 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:14.025 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:14.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:14.025 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:29:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:29:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:29:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:14Z" 2023-10-17T19:29:14.032 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:14.033 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:14Z" } 2023-10-17T19:29:14.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:29:14.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:14.035 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:29:14.035 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:29:19.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:19.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:19.037 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:19.098 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:19.098 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:19.149 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:19.149 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:19.149 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:19.150 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:19.150 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:19.150 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:19.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.151 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:19.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:19.151 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:19.151 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:19.151 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:19.152 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040074", 2023-10-17T19:29:19.152 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:19.152 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:19.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:19.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:25.103689Z" 2023-10-17T19:29:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040702", 2023-10-17T19:29:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:21.123698Z" 2023-10-17T19:29:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046156", 2023-10-17T19:29:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:25:20.068512Z" 2023-10-17T19:29:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:19Z" 2023-10-17T19:29:19.162 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:19.163 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:18:25.103689Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040702", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:21.123698Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046156", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:25:20.068512Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:19Z" } 2023-10-17T19:29:19.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:18:25.103689Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040702', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:21.123698Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046156', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:25:20.068512Z'}}] 2023-10-17T19:29:19.164 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:19.164 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040074', 1: '00000000000000000000:00000000000000040702', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046156'} 2023-10-17T19:29:19.164 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000040216 target=00000000000000000000:00000000000000040074, shard=1 master=00000000000000000000:00000000000000040842 target=00000000000000000000:00000000000000040702, shard=3 master=00000000000000000000:00000000000000046436 target=00000000000000000000:00000000000000046156 2023-10-17T19:29:24.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:24.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:24.167 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:24.230 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.230 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:24.281 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.295 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.295 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:24Z" 2023-10-17T19:29:24.296 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:24.298 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:24Z" } 2023-10-17T19:29:24.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:24.299 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:24.299 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:24.300 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:29:24.300 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:29:24.301 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:29:24.301 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.302 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:29:24.302 DEBUG:boto:Data: 2023-10-17T19:29:24.302 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.303 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.303 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.304 DEBUG:boto:Params: {} 2023-10-17T19:29:24.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:29:24.305 DEBUG:boto:Token: None 2023-10-17T19:29:24.306 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/ 2023-10-17T19:29:24.307 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FS+1MGRN5IP49Tk6IrvB7UqfUFk= 2023-10-17T19:29:24.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FS+1MGRN5IP49Tk6IrvB7UqfUFk=', 'Content-Length': '0'} 2023-10-17T19:29:24.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084725d28cebcfa9a-00652ee094-21e1-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.314 DEBUG:boto:path=/hbmuog-68/obj1 2023-10-17T19:29:24.314 DEBUG:boto:auth_path=/hbmuog-68/obj1 2023-10-17T19:29:24.314 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.315 DEBUG:boto:Path: /hbmuog-68/obj1 2023-10-17T19:29:24.315 DEBUG:boto:Data: 2023-10-17T19:29:24.316 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.316 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.317 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.317 DEBUG:boto:Params: {} 2023-10-17T19:29:24.317 DEBUG:boto:Token: None 2023-10-17T19:29:24.318 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj1 2023-10-17T19:29:24.318 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:eH+RmGZqcjLhNEziMWtCn05cEww= 2023-10-17T19:29:24.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eH+RmGZqcjLhNEziMWtCn05cEww=', 'Content-Length': '0'} 2023-10-17T19:29:24.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc0f50bcb2c0dbb9-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.322 DEBUG:boto:path=/hbmuog-68/obj1 2023-10-17T19:29:24.322 DEBUG:boto:auth_path=/hbmuog-68/obj1 2023-10-17T19:29:24.323 DEBUG:boto:Method: GET 2023-10-17T19:29:24.323 DEBUG:boto:Path: /hbmuog-68/obj1 2023-10-17T19:29:24.324 DEBUG:boto:Data: 2023-10-17T19:29:24.324 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.324 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.325 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.325 DEBUG:boto:Params: {} 2023-10-17T19:29:24.326 DEBUG:boto:Token: None 2023-10-17T19:29:24.326 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj1 2023-10-17T19:29:24.327 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:A7YPHDshBhOFQay5M0Df/IYdvkg= 2023-10-17T19:29:24.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:A7YPHDshBhOFQay5M0Df/IYdvkg=', 'Content-Length': '0'} 2023-10-17T19:29:24.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000979d342b4525be22-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.330 DEBUG:boto:path=/hbmuog-68/obj2 2023-10-17T19:29:24.330 DEBUG:boto:auth_path=/hbmuog-68/obj2 2023-10-17T19:29:24.331 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.331 DEBUG:boto:Path: /hbmuog-68/obj2 2023-10-17T19:29:24.331 DEBUG:boto:Data: 2023-10-17T19:29:24.332 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.332 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.333 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.333 DEBUG:boto:Params: {} 2023-10-17T19:29:24.334 DEBUG:boto:Token: None 2023-10-17T19:29:24.334 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj2 2023-10-17T19:29:24.335 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MChwqh/1WUzLbXeu7ePbK34nUFc= 2023-10-17T19:29:24.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MChwqh/1WUzLbXeu7ePbK34nUFc=', 'Content-Length': '0'} 2023-10-17T19:29:24.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be64bb0894d67c16-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.337 DEBUG:boto:path=/hbmuog-68/obj2 2023-10-17T19:29:24.338 DEBUG:boto:auth_path=/hbmuog-68/obj2 2023-10-17T19:29:24.338 DEBUG:boto:Method: GET 2023-10-17T19:29:24.339 DEBUG:boto:Path: /hbmuog-68/obj2 2023-10-17T19:29:24.339 DEBUG:boto:Data: 2023-10-17T19:29:24.340 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.340 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.341 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.341 DEBUG:boto:Params: {} 2023-10-17T19:29:24.341 DEBUG:boto:Token: None 2023-10-17T19:29:24.342 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj2 2023-10-17T19:29:24.342 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HWCwH079kyviSZcp6sFpI+NXTk4= 2023-10-17T19:29:24.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HWCwH079kyviSZcp6sFpI+NXTk4=', 'Content-Length': '0'} 2023-10-17T19:29:24.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8eb613026c95883-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.345 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:29:24.346 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:29:24.346 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.347 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:29:24.347 DEBUG:boto:Data: 2023-10-17T19:29:24.348 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.348 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.349 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.349 DEBUG:boto:Params: {} 2023-10-17T19:29:24.349 DEBUG:boto:Token: None 2023-10-17T19:29:24.350 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj3 2023-10-17T19:29:24.350 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:FNyOHk52cvo9xRuJGc8Mf3PwO+0= 2023-10-17T19:29:24.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FNyOHk52cvo9xRuJGc8Mf3PwO+0=', 'Content-Length': '0'} 2023-10-17T19:29:24.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bea9ca31ffa56a4-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.353 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:29:24.354 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:29:24.354 DEBUG:boto:Method: GET 2023-10-17T19:29:24.355 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:29:24.355 DEBUG:boto:Data: 2023-10-17T19:29:24.356 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.356 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.357 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.357 DEBUG:boto:Params: {} 2023-10-17T19:29:24.357 DEBUG:boto:Token: None 2023-10-17T19:29:24.358 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-68/obj3 2023-10-17T19:29:24.358 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:uS2dOfZ40LHWg4cma2j7vanPzmg= 2023-10-17T19:29:24.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uS2dOfZ40LHWg4cma2j7vanPzmg=', 'Content-Length': '0'} 2023-10-17T19:29:24.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b15bd81d44265424-00652ee094-21e1-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.362 DEBUG:boto:path=/hbmuog-69/ 2023-10-17T19:29:24.362 DEBUG:boto:auth_path=/hbmuog-69/ 2023-10-17T19:29:24.363 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.363 DEBUG:boto:Path: /hbmuog-69/ 2023-10-17T19:29:24.363 DEBUG:boto:Data: 2023-10-17T19:29:24.364 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.364 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.365 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.365 DEBUG:boto:Params: {} 2023-10-17T19:29:24.366 DEBUG:boto:Token: None 2023-10-17T19:29:24.366 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-69/ 2023-10-17T19:29:24.367 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AVepNJ/acqDbj6qe7k7OAai9o4Q= 2023-10-17T19:29:24.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AVepNJ/acqDbj6qe7k7OAai9o4Q=', 'Content-Length': '0'} 2023-10-17T19:29:24.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b87844f2de0f94f-00652ee094-21e1-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.370 DEBUG:boto:path=/hbmuog-69/obj1 2023-10-17T19:29:24.371 DEBUG:boto:auth_path=/hbmuog-69/obj1 2023-10-17T19:29:24.371 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.371 DEBUG:boto:Path: /hbmuog-69/obj1 2023-10-17T19:29:24.372 DEBUG:boto:Data: 2023-10-17T19:29:24.372 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.373 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.373 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.373 DEBUG:boto:Params: {} 2023-10-17T19:29:24.374 DEBUG:boto:Token: None 2023-10-17T19:29:24.374 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-69/obj1 2023-10-17T19:29:24.375 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dvtRRRRjSNq7X4j5nMdc8Iym4UI= 2023-10-17T19:29:24.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dvtRRRRjSNq7X4j5nMdc8Iym4UI=', 'Content-Length': '0'} 2023-10-17T19:29:24.377 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007771a83a2ff2532c-00652ee094-21e1-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.377 DEBUG:boto:path=/hbmuog-69/obj2 2023-10-17T19:29:24.378 DEBUG:boto:auth_path=/hbmuog-69/obj2 2023-10-17T19:29:24.378 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.379 DEBUG:boto:Path: /hbmuog-69/obj2 2023-10-17T19:29:24.379 DEBUG:boto:Data: 2023-10-17T19:29:24.380 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.380 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.381 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.381 DEBUG:boto:Params: {} 2023-10-17T19:29:24.382 DEBUG:boto:Token: None 2023-10-17T19:29:24.382 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-69/obj2 2023-10-17T19:29:24.383 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:l75E5gLrcAIA/Zno5+ULpu+mJ+I= 2023-10-17T19:29:24.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:l75E5gLrcAIA/Zno5+ULpu+mJ+I=', 'Content-Length': '0'} 2023-10-17T19:29:24.385 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000280a3f20d1448d42-00652ee094-21e1-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.385 DEBUG:boto:path=/hbmuog-69/obj3 2023-10-17T19:29:24.386 DEBUG:boto:auth_path=/hbmuog-69/obj3 2023-10-17T19:29:24.386 DEBUG:boto:Method: HEAD 2023-10-17T19:29:24.387 DEBUG:boto:Path: /hbmuog-69/obj3 2023-10-17T19:29:24.387 DEBUG:boto:Data: 2023-10-17T19:29:24.387 DEBUG:boto:Headers: {} 2023-10-17T19:29:24.388 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:29:24.388 DEBUG:boto:Port: 8001 2023-10-17T19:29:24.389 DEBUG:boto:Params: {} 2023-10-17T19:29:24.389 DEBUG:boto:Token: None 2023-10-17T19:29:24.390 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:24 GMT /hbmuog-69/obj3 2023-10-17T19:29:24.390 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:VILdTG7KzXJLIRqLLUZzA+sj7ds= 2023-10-17T19:29:24.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VILdTG7KzXJLIRqLLUZzA+sj7ds=', 'Content-Length': '0'} 2023-10-17T19:29:24.392 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f04677b303a2fb56-00652ee094-21e1-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:24.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.394 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:24.447 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.448 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:24.493 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-17T19:29:24.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.495 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:24.548 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.549 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:24.596 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:29:24.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.597 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:24.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.649 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:24.695 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:29:24.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.698 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:24.747 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:24.800 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:24.801 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:29:24.801 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 17, 2023-10-17T19:29:24.801 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:29:24.802 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:29:24.802 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:29:24.802 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:29:24.802 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:29:24.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.803 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:29:24.803 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:29:24.803 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:29:24.803 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:29:24.803 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.804 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:29:24.804 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.804 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:29:24.804 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:29:24.804 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.805 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:29:24.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.805 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.805 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:29:24.806 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.806 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:29:24.806 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.806 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:29:24.806 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:29:24.807 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:29:24.807 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:29:24.807 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:29:24.807 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:29:24.807 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:29:24.808 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:29:24.808 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:29:24.808 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:29:24.808 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.808 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.809 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:29:24.809 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:29:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:29:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:29:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:29:24.811 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:29:24.811 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:29:24.811 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:29:24.811 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:29:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:29:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:29:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:29:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:29:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.813 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.813 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.813 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.813 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:29:24.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.814 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:29:24.814 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:29:24.814 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:29:24.814 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:29:24.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.815 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.815 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:29:24.815 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:29:24.816 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:29:24.816 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:29:24.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.816 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:29:24.816 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:29:24.817 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:29:24.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.817 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:29:24.817 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:29:24.817 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.818 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:29:24.818 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.818 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.818 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.819 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:29:24.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.819 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:29:24.819 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:29:24.820 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:29:24.820 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:29:24.820 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.820 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:29:24.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.822 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:29:24.822 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:29:24.822 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:29:24.823 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.823 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:29:24.823 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.824 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:29:24.824 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:29:24.824 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:29:24.824 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:29:24.824 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.825 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:29:24.825 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:29:24.825 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:29:24.825 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:29:24.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.826 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.827 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "allowed" 2023-10-17T19:29:24.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.827 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.827 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:29:24.828 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.828 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.828 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:29:24.829 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:29:24.829 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:29:24.829 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:29:24.829 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.829 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:29:24.830 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.830 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:29:24.830 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:29:24.830 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:29:24.831 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:29:24.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.831 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:29:24.831 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:29:24.831 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.832 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:29:24.832 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.832 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:29:24.832 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:29:24.833 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:29:24.833 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:29:24.833 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:29:24.833 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:29:24.833 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:29:24.834 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:29:24.834 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:29:24.834 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:29:24.834 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.835 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:29:24.835 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:29:24.836 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:29:24.836 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:29:24.836 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.836 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:29:24.837 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:29:24.837 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:29:24.837 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:29:24.838 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:29:24.838 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:29:24.838 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:29:24.838 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:29:24.839 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:29:24.839 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:29:24.839 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.839 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.840 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.840 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:29:24.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.840 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:29:24.841 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:29:24.841 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:29:24.841 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:29:24.841 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.842 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.842 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:29:24.842 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:29:24.842 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:29:24.843 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:29:24.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.843 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:29:24.843 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:29:24.843 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.844 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.844 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:29:24.844 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:29:24.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.844 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:29:24.845 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:29:24.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.845 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.846 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:29:24.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.847 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:29:24.847 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:29:24.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:24.848 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:29:24.848 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:29:24.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:24.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.849 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:29:24.850 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:29:24.850 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:29:24.850 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:29:24.850 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:29:24.850 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:29:24.851 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:29:24.851 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:29:24.851 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:29:24.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.852 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:29:24.852 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:29:24.852 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:29:24.852 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:29:24.852 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:29:24.853 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:29:24.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.853 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:29:24.853 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:29:24.854 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:29:24.854 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:29:24.854 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:29:24.854 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:29:24.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.855 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:29:24.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:29:24.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:29:24.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:29:24.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:29:24.856 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:29:24.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.856 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:29:24.856 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:29:24.856 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:29:24.857 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:29:24.857 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:29:24.857 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:29:24.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:24.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:24.858 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:29:24.858 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:29:24.858 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:24.858 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:24.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:24.860 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:24.922 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:24.923 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.048 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:29:25.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.049 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:25.128 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.129 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.225 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:29:25.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.227 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:25.307 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.307 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.382 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:29:25.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.384 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:25.456 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.457 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.512 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:29:25.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.514 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:25.566 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.567 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.616 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:29:25.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:29:25.618 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:29:25.666 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.667 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.714 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:29:25.714 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:29:25.715 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:29:25.715 DEBUG:boto:Method: HEAD 2023-10-17T19:29:25.715 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:29:25.716 DEBUG:boto:Data: 2023-10-17T19:29:25.716 DEBUG:boto:Headers: {} 2023-10-17T19:29:25.716 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:29:25.717 DEBUG:boto:Port: 8000 2023-10-17T19:29:25.717 DEBUG:boto:Params: {} 2023-10-17T19:29:25.718 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:29:25.718 DEBUG:boto:Token: None 2023-10-17T19:29:25.719 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:25 GMT /hbmuog-68/ 2023-10-17T19:29:25.719 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QeICcKTtqNEpIjZV2tqUQcllWrQ= 2023-10-17T19:29:25.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QeICcKTtqNEpIjZV2tqUQcllWrQ=', 'Content-Length': '0'} 2023-10-17T19:29:25.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000552f699062bad33e-00652ee095-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:25.726 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:29:25.726 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:29:25.727 DEBUG:boto:Method: PUT 2023-10-17T19:29:25.727 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:29:25.727 DEBUG:boto:Data: 2023-10-17T19:29:25.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:29:25.728 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:29:25.729 DEBUG:boto:Port: 8000 2023-10-17T19:29:25.729 DEBUG:boto:Params: {} 2023-10-17T19:29:25.729 DEBUG:boto:Token: None 2023-10-17T19:29:25.730 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:29:25 GMT /hbmuog-68/obj3 2023-10-17T19:29:25.730 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:5Q69X+7bCJAxETLWNj71wLWLIZ4= 2023-10-17T19:29:25.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:29:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:5Q69X+7bCJAxETLWNj71wLWLIZ4='} 2023-10-17T19:29:25.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005cbee5a87eb7d41-00652ee095-22c3-a1'), ('Date', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:25.740 DEBUG:boto:path=/hbmuog-69/ 2023-10-17T19:29:25.740 DEBUG:boto:auth_path=/hbmuog-69/ 2023-10-17T19:29:25.741 DEBUG:boto:Method: HEAD 2023-10-17T19:29:25.741 DEBUG:boto:Path: /hbmuog-69/ 2023-10-17T19:29:25.741 DEBUG:boto:Data: 2023-10-17T19:29:25.742 DEBUG:boto:Headers: {} 2023-10-17T19:29:25.742 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:29:25.743 DEBUG:boto:Port: 8000 2023-10-17T19:29:25.743 DEBUG:boto:Params: {} 2023-10-17T19:29:25.743 DEBUG:boto:Token: None 2023-10-17T19:29:25.744 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:29:25 GMT /hbmuog-69/ 2023-10-17T19:29:25.744 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zB2HHQw0rB4UqpnFRLkUtuZfLkY= 2023-10-17T19:29:25.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:29:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zB2HHQw0rB4UqpnFRLkUtuZfLkY=', 'Content-Length': '0'} 2023-10-17T19:29:25.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5daccd468126897-00652ee095-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:25.748 DEBUG:boto:path=/hbmuog-69/obj3 2023-10-17T19:29:25.749 DEBUG:boto:auth_path=/hbmuog-69/obj3 2023-10-17T19:29:25.749 DEBUG:boto:Method: PUT 2023-10-17T19:29:25.749 DEBUG:boto:Path: /hbmuog-69/obj3 2023-10-17T19:29:25.750 DEBUG:boto:Data: 2023-10-17T19:29:25.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:29:25.751 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:29:25.751 DEBUG:boto:Port: 8000 2023-10-17T19:29:25.751 DEBUG:boto:Params: {} 2023-10-17T19:29:25.752 DEBUG:boto:Token: None 2023-10-17T19:29:25.752 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:29:25 GMT /hbmuog-69/obj3 2023-10-17T19:29:25.753 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:OB4VNCNIdDPDw/F4zeUt5Nrm67o= 2023-10-17T19:29:25.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:29:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OB4VNCNIdDPDw/F4zeUt5Nrm67o='} 2023-10-17T19:29:25.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008327ccda47571716-00652ee095-22c3-a1'), ('Date', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:29:25.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:25.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:25.800 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:25.850 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.851 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.883 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:29:25.896 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:29:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:25.896 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570965.601721_1052.1", 2023-10-17T19:29:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:29:25.601721Z" 2023-10-17T19:29:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:29:25.897 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:29:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:29:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:29:25.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:29:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:29:25.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:25.899 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:29:25.900 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:25.901 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:29:25.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:25.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:25.902 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:25.952 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:25.952 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:25.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:25.982+0000 7f0901774780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:25.983 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:25.983+0000 7f0901774780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:25.996 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:29:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:29:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:29:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:44.884595Z", 2023-10-17T19:29:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:29:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:29:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:29:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:29:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:29:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:29:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:29:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:25Z" 2023-10-17T19:29:26.010 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:26.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570864.884595_1009.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:27:44.884595Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:29:25Z" } 2023-10-17T19:29:26.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570864.884595_1009.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:27:44.884595Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:29:26.012 DEBUG:rgw_multi.tests:log_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:26.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:26.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570965.601721_1052.1 target=1_1697570864.884595_1009.1 2023-10-17T19:29:31.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:31.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:31.015 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:31.074 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:31.075 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:31.108 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:31.107+0000 7f1c3e21e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:31.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:31.108+0000 7f1c3e21e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:31.123 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:31.124 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:31.124 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:31.124 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:31.124 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:31.125 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:29:31.125 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:31.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.125 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:31.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:31.126 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:31.126 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:31.127 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:31.127 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570864.884595_1009.1", 2023-10-17T19:29:31.127 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:31.127 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:29:31.128 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:31.128 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:44.884595Z", 2023-10-17T19:29:31.128 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:31.128 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:31.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:31.129 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:31.129 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:31.129 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:31.130 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:29:31.130 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:31.130 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:31.130 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:31.130 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:29:31.131 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:31.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:31.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:31.132 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:31.132 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:31.133 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:31.133 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:29:31.133 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:31.134 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:31.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:31.134 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:29:31.135 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:31.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:31.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:31.136 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:31.136 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:31.136 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:31.136 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:29:31.136 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:31.137 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:31.137 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:31.137 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:29:31.137 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:31.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:31.138 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:31.138 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:31.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.138 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:31.139 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:29:31.139 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:29:31.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:31.139 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:31Z" 2023-10-17T19:29:31.139 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:31.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570864.884595_1009.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:27:44.884595Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:29:31Z" } 2023-10-17T19:29:31.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570864.884595_1009.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:27:44.884595Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:29:31.141 DEBUG:rgw_multi.tests:log_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:31.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570864.884595_1009.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:31.142 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697570965.601721_1052.1 target=1_1697570864.884595_1009.1 2023-10-17T19:29:36.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.145 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:36.204 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:36.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:36.242+0000 7fbf73242780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:36.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:29:36.243+0000 7fbf73242780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:29:36.259 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570965.601721_1052.1", 2023-10-17T19:29:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:29:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.601721Z", 2023-10-17T19:29:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:29:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:29:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.266 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:36.266 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.266 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:36.266 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:29:36.267 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.267 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:36.267 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.267 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.268 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:36.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.270 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:29:36.270 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:29:36.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.271 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:36.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.271 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:36.271 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:29:36.272 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:29:36.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.272 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:36Z" 2023-10-17T19:29:36.272 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:36.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570965.601721_1052.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:29:25.601721Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:29:36Z" } 2023-10-17T19:29:36.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570965.601721_1052.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.601721Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:29:36.274 DEBUG:rgw_multi.tests:log_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:36.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:29:36.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:29:36.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.277 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:36.386 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:29:36.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.387 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:36.387 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:36.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.388 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:36.388 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:36.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.388 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:36.389 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:36.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.389 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:29:36.389 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:29:36.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.390 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:29:36.391 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:36.392 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:29:36.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:36.393 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:36.446 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:36.446 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:36.493 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:36.494 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:36.494 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:36.494 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:36.494 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:36.495 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:36.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.495 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:36.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:36.496 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.496 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:36.496 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:36.496 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.498 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:36.498 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.498 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:36.498 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:36.499 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.499 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:36.499 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.499 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.501 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:36.501 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:36.501 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.501 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:36.506 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:36.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:36.506 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:36Z" 2023-10-17T19:29:36.506 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:36.507 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:36Z" } 2023-10-17T19:29:36.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:36.508 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:36.509 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:36.509 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:29:41.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:41.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:41.511 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:41.572 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:41.573 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:41.624 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:41.624 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:41.624 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:41.625 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:41.625 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:41.625 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:41.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.625 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:41.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:41.626 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:41.626 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:41.626 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:41.627 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:41.628 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:41.629 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:41.629 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:41.629 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:41.629 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:41.630 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:41.630 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:41.630 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:41.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:41.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:41.631 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:41.631 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:41.631 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:41.631 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:41.632 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:41.632 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:41.632 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:41.632 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:41.632 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:41.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:41.633 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:41.633 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:41.633 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:41.634 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:41.634 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:41.634 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:41.634 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:41.634 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:41.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:41.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:41.635 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:41.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.635 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:41.636 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:41.636 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:41.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:41.636 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:41Z" 2023-10-17T19:29:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:41.637 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:41Z" } 2023-10-17T19:29:41.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:41.638 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:41.639 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:41.639 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:29:46.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:46.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:46.642 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:46.753 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:46.754 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:46.754 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:46.754 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:46.755 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:46.755 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:46.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.756 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:46.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:46.757 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:46.757 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:46.757 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:46.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:46.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:46.758 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:46.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:46.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:46.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:46.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:46.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:46.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:46.760 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:46.760 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:46.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:46.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:46.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:46.761 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:46.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:46.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:46.762 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:46.762 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:46.763 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:46.763 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:46.763 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:46.764 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:46.764 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:46.764 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:46.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:46.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:46.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:46.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:46.766 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:46.766 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:46.767 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:46.767 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:46.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:46.767 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:46.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:46.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:46.768 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:46.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.769 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:46.769 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:46.769 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:46.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:46.769 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:46Z" 2023-10-17T19:29:46.770 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:46.771 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:46Z" } 2023-10-17T19:29:46.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:46.772 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:46.772 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:46.772 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:29:51.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:51.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:51.775 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:51.837 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:51.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:51.890 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:51.890 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:51.890 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:51.891 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:51.891 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:51.891 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:51.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.891 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:51.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:51.892 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:51.892 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:51.892 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:51.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:51.893 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:51.893 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:51.893 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:51.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:51.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:51.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:51.894 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:51.894 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:51.895 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:51.895 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:51.895 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:51.895 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:51.895 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:51.896 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:51.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:51.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:51.896 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:51.897 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:51.898 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:51.898 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:51.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:51.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:51.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:51.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:51.899 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:51.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:51.899 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:51.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:51.900 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:51.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:51.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:51.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:51.901 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:51.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.901 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:51.901 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:51.902 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:51.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:51.902 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:51Z" 2023-10-17T19:29:51.902 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:51.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:51Z" } 2023-10-17T19:29:51.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:51.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:51.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:51.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:29:56.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:56.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:29:56.907 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:29:56.969 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:29:56.969 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:29:57.020 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:29:57.021 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:29:57.021 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:29:57.021 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:29:57.022 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:29:57.022 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:29:57.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.022 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:29:57.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:57.023 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:29:57.023 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:57.023 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:57.023 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:29:57.024 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:57.024 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:57.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:57.024 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:29:57.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:57.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:57.025 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:29:57.025 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:57.025 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:57.026 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:29:57.026 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:57.026 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:29:57.026 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:57.027 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:29:57.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:57.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:57.027 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:29:57.028 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:57.028 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:57.028 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:29:57.028 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:57.028 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:29:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:29:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:29:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:29:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:29:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:29:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:29:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:29:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:29:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:29:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:29:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:29:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:29:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:29:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:29:57.033 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:29:57.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:29:57.033 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:29:57Z" 2023-10-17T19:29:57.033 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:29:57.034 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:29:57Z" } 2023-10-17T19:29:57.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:29:57.035 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:29:57.036 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:29:57.036 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:02.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:02.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:02.039 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:02.101 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:02.102 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:02.151 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:02Z" 2023-10-17T19:30:02.164 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:02.164 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:02Z" } 2023-10-17T19:30:02.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:02.165 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:02.166 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:02.166 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:07.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:07.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:07.169 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:07.230 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:07.231 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:07.285 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:07.286 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:07.286 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:07.286 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:07.287 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:07.287 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:07.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.287 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:07.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:07.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:07.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:07.288 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:07.288 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:07.288 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:07.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:07.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:07.289 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:07.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:07.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:07.290 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:07.290 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:07Z" 2023-10-17T19:30:07.299 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:07.300 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:07Z" } 2023-10-17T19:30:07.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:07.301 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:07.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:07.302 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:12.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:12.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:12.304 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:12.365 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:12.417 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:12.417 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:12.418 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:12.418 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:12.418 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:12.418 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:12.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.419 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:12.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:12.419 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:12.420 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:12.420 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:12.420 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:12.420 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:12.420 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:12.421 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:12.421 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:12.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:12.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:12.422 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:12.423 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:12.423 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:12.423 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:12.423 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:12.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:12.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:12.424 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:12.424 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:12.425 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:12.425 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:12.425 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:12.425 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:12.426 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:12.426 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:12.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:12.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:12.427 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:12.427 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:12.427 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:12.427 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:12.428 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:12.428 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:12.428 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:12.428 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:12.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:12.429 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:12.429 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:12.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.429 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:12.430 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:12.430 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:12.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:12.430 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:12Z" 2023-10-17T19:30:12.431 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:12.432 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:12Z" } 2023-10-17T19:30:12.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:12.433 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:12.433 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:12.433 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:17.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:17.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:17.436 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:17.498 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:17.547 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:17Z" 2023-10-17T19:30:17.560 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:17.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:17Z" } 2023-10-17T19:30:17.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:17.562 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:17.562 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:17.563 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:22.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:22.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:22.565 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:22.679 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:22.679 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:22.679 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:22.680 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:22.680 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:22.680 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:22.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.681 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:22.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:22.681 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:22.681 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:22.682 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:22.682 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:22.682 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:22.682 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:22.682 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:22.683 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:22.683 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:22.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:22.683 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:22.684 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:22.684 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:22.684 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:22.684 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:22.684 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:22.685 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:22.685 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:22.685 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:22.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:22.686 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:22.686 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:22.686 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:22.686 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:22.687 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:22.687 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:22.687 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:22.687 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:22.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:22.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:22.688 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:22.688 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:22.688 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:22.689 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:22.689 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:22.689 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:22.689 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:22.689 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:22.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:22.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:22.690 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:22.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.691 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:22.691 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:22.691 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:22.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:22.692 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:22Z" 2023-10-17T19:30:22.692 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:22.693 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:22Z" } 2023-10-17T19:30:22.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:22.694 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:22.694 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:22.695 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:27.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:27.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:27.697 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:27.758 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:27.758 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:27.808 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:27.809 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:27.809 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:27.809 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:27.809 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:27.810 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:27.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.810 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:27.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:27.811 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:27.812 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:27.812 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:27.812 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:27.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:27.814 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:27.814 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:27.814 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:27.814 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:27.815 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:27.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:27.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:27.815 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:27.816 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:27.816 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:27.816 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:27.816 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:27.816 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:27.817 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:27.817 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:27.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:27.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:27.818 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:27.818 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:27.818 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:27.818 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:27.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:27.820 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:27.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.820 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:27.820 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:27.820 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:27.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:27.821 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:27Z" 2023-10-17T19:30:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:27.822 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:27Z" } 2023-10-17T19:30:27.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:27.823 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:27.823 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:27.824 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:32.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:32.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:32.826 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:32.887 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:32.887 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:32.937 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:32Z" 2023-10-17T19:30:32.950 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:32.951 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:32Z" } 2023-10-17T19:30:32.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:32.952 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:32.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:32.953 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:37.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:37.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:37.955 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:38.016 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:38.017 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:38.068 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:38.069 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:38.069 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:38.069 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:38.069 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:38.070 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:38.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.070 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:38.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:38.071 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:38.071 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:38.071 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:38.071 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:38.071 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:38.072 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:38.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:38.072 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:38.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:38.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:38.073 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:38.073 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:38.073 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:38.073 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:38.074 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:38.074 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:38.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:38.074 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:38.075 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:38.076 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:38.076 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:38.076 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:38.076 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:38.076 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:38.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:38.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:38.077 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:38.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:38.081 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:38Z" 2023-10-17T19:30:38.081 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:38.082 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:38Z" } 2023-10-17T19:30:38.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:38.083 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:38.083 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:38.084 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:43.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:43.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:43.086 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:43.148 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:43.148 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:43Z" 2023-10-17T19:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:43.212 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:43Z" } 2023-10-17T19:30:43.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:43.213 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:43.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:43.214 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:48.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:48.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:48.216 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:48.275 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:48.276 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:48.325 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:48.326 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:48.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:48.328 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:48.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:48.330 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:48.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:48.333 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:48.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:48.335 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:48.336 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.337 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:48.338 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:48Z" 2023-10-17T19:30:48.338 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:48.340 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:48Z" } 2023-10-17T19:30:48.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:48.341 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:48.341 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:48.341 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:53.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:53.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:53.344 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:53.405 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:53.406 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:53.456 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:53.456 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:53.456 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:53.456 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:53.457 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:53.457 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:53.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.457 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:53.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:53.458 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:53.458 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:53.458 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:53.459 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:53.459 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:53.459 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:53.459 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:53.459 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:53.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:53.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:53.460 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:53.461 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:53.461 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:53.461 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:53.461 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:53.461 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:53.462 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:53.462 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:53.462 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:53.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:53.463 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:53.463 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:53.463 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:53.463 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:53.464 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:53.464 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:53.464 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:53.464 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:53.465 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:53.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:53.465 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:53.465 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:53.466 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:53.466 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:53.466 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:53.466 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:53.467 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:53.467 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:53.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:53.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:53.467 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:53.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.468 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:53.468 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:53.468 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:53.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:53.469 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:53Z" 2023-10-17T19:30:53.469 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:53.470 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:53Z" } 2023-10-17T19:30:53.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:53.471 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:53.471 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:53.472 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:30:58.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:58.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:30:58.474 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:30:58.535 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:30:58.535 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:30:58.586 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:30:58.586 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:30:58.587 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:30:58.587 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:30:58.587 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:30:58.587 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:30:58.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.588 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:30:58.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:58.588 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:30:58.588 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:58.588 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:58.589 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:30:58.589 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:58.589 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:58.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:58.590 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:30:58.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:58.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:58.590 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:30:58.592 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:58.592 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:30:58.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:58.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:58.593 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:30:58.593 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:58.593 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:58.593 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:30:58.594 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:58.594 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:30:58.594 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:58.594 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:30:58.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:58.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:30:58.595 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:30:58.595 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:30:58.596 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:30:58.596 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:30:58.596 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:30:58.596 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:30:58.596 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:30:58.597 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:30:58.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:58.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:30:58.597 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:30:58.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.598 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:30:58.598 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:30:58.598 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:30:58.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:30:58.598 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:30:58Z" 2023-10-17T19:30:58.599 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:30:58.600 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:30:58Z" } 2023-10-17T19:30:58.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:30:58.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:30:58.601 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:30:58.601 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:03.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:03.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:03.604 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:03.666 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:03.667 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:03.720 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:03.720 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:03.720 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:03.721 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:03.721 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:03.721 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:03.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.722 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:03.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:03.722 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:03.722 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:03.722 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:03.723 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:03.723 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:03.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:03.723 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:03.724 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:03.724 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:03.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:03.724 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:03.725 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:03.725 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:03.725 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:03.725 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:03.726 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:03.726 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:03.726 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:03.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:03.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:03.727 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:03.727 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:03.727 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:03.728 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:03.728 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:03.728 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:03.728 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:03.729 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:03.729 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:03.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:03.730 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:03.730 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:03.730 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:03.730 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:31:03.731 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:03.731 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:03.731 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:03.731 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:31:03.731 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:03.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:03.732 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:03.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.732 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:03.732 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:03.733 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:03.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:03.733 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:03Z" 2023-10-17T19:31:03.733 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:03.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:03Z" } 2023-10-17T19:31:03.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:31:03.736 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:03.736 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:31:03.736 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:08.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:08.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:08.739 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:08.800 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:08.851 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:08.852 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:08.852 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:08.852 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:08.852 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:08.853 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:08.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.853 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:08.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:08.854 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:08.854 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:08.854 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:08.854 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:08.855 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:08.855 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:08.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:08.855 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:08.856 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:08.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:08.856 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:08.857 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:08.857 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:08.857 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:08.857 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:08.858 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:08.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:08.858 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:08.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:08.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:08.859 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:08.859 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:08.859 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:08.860 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:08.860 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:08.860 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:08.860 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:08.860 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:08.861 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:08.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:08.861 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:08.862 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:08.862 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:08.862 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:31:08.862 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:08.863 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:08.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:08.863 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:31:08.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:08.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:08.864 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:08.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.864 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:08.865 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:08.865 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:08.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:08.865 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:08Z" 2023-10-17T19:31:08.866 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:08.867 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:08Z" } 2023-10-17T19:31:08.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:31:08.868 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:08.868 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:31:08.869 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:13.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:13.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:13.871 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:13.933 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:13.934 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:13.984 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:13.985 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:13.985 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:13.985 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:13.986 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:13.986 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:13.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.986 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:13.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:13.987 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:13.987 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:13.987 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:13.987 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:13.988 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:13.988 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:13.988 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:13.988 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:13.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:13.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:13.989 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:13.989 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:13.990 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:13.990 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:13.990 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:13.990 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:13.990 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:13.991 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:13.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:13.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:13.991 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:13.992 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:13.992 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:13.992 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:13.992 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:13.993 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:13.993 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:13.993 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:13.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:13.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:31:13.996 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:13.996 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:13.996 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:13.996 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:31:13.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:13.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:13.997 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:13.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.998 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:13.998 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:13.998 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:13.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:13.999 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:13Z" 2023-10-17T19:31:13.999 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:14.000 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:13Z" } 2023-10-17T19:31:14.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:31:14.001 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:14.002 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:31:14.002 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:19.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:19.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:19.004 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:19.116 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:19.117 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:19.117 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:19.117 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:19.118 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:19.118 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:19.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.118 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:19.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:19.119 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:19.119 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:19.119 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:19.120 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:19.120 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:19.120 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:19.120 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:19.122 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:19.122 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:19.122 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:19.123 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:19.123 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:19.123 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:19.123 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:19.124 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:19.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:19.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:19.124 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:19.125 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:19.125 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:19.125 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:19.125 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:19.126 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:19.126 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:19.126 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:19.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:19.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:19.127 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:19.127 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:19.127 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:19.128 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:31:19.128 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:19.128 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:19.128 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:19.128 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:31:19.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:19.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:19.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:19.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.129 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:19.130 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:19.130 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:19.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:19.130 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:19Z" 2023-10-17T19:31:19.131 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:19.132 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:19Z" } 2023-10-17T19:31:19.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:31:19.133 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:19.133 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:31:19.133 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:24.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:24.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:24.136 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:24.197 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:24.197 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:24.248 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:24.248 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:24.249 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:24.249 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:24.249 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:24.249 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:24.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:24.250 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:24.251 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:24.251 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:24.251 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:24.251 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:24.252 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:24.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:24.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:24.252 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:24.253 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:24.253 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:24.253 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:24.254 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:24.254 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:24.254 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:24.254 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:24.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:24.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:24.255 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:24.256 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:24.256 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:24.256 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:24.256 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:24.256 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:24.257 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:24.257 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:24.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:24.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:24.258 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:24.258 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:24.258 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:24.258 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046436", 2023-10-17T19:31:24.259 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:24.259 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:24.259 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:24.259 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.350261Z" 2023-10-17T19:31:24.259 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:24.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:24.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:24.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.260 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:24.261 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:24.261 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:24.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:24.261 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:24Z" 2023-10-17T19:31:24.261 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:24.262 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046436", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:27:51.350261Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:24Z" } 2023-10-17T19:31:24.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046436', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.350261Z'}}] 2023-10-17T19:31:24.263 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:24.264 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046436'} 2023-10-17T19:31:24.264 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000046716 target=00000000000000000000:00000000000000046436 2023-10-17T19:31:29.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.266 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:29.328 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:29.329 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:29.380 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:29.380 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:29.380 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:29.381 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:29.381 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:29.381 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:29.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.381 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:29.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.382 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:29.382 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.382 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:29.383 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:29.383 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.383 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:29.383 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.383 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:29.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.384 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:29.384 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.385 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:29.385 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:29.385 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.385 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:29.386 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.386 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:29.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.387 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:29.387 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.387 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:29.387 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:29.388 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.388 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:29.388 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.388 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:29.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.389 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:29.389 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.390 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:29.390 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:29.390 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.390 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:29.390 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.391 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:29.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.391 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:29.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.392 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:29.392 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:29.392 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:29.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.393 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:29Z" 2023-10-17T19:31:29.393 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:29.394 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:29Z" } 2023-10-17T19:31:29.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:29.395 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:29.395 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:29.396 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:31:29.396 DEBUG:boto:path=/hbmuog-68/ 2023-10-17T19:31:29.397 DEBUG:boto:auth_path=/hbmuog-68/ 2023-10-17T19:31:29.397 DEBUG:boto:Method: HEAD 2023-10-17T19:31:29.397 DEBUG:boto:Path: /hbmuog-68/ 2023-10-17T19:31:29.398 DEBUG:boto:Data: 2023-10-17T19:31:29.398 DEBUG:boto:Headers: {} 2023-10-17T19:31:29.399 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:31:29.399 DEBUG:boto:Port: 8001 2023-10-17T19:31:29.399 DEBUG:boto:Params: {} 2023-10-17T19:31:29.400 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:31:29.401 DEBUG:boto:Token: None 2023-10-17T19:31:29.401 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:29 GMT /hbmuog-68/ 2023-10-17T19:31:29.402 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:mMpe4arLfeXhDHqeN15AUDoM+Lo= 2023-10-17T19:31:29.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mMpe4arLfeXhDHqeN15AUDoM+Lo=', 'Content-Length': '0'} 2023-10-17T19:31:29.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006545d9838480daa2-00652ee111-22c0-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:29.408 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:31:29.408 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:31:29.409 DEBUG:boto:Method: HEAD 2023-10-17T19:31:29.409 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:31:29.410 DEBUG:boto:Data: 2023-10-17T19:31:29.410 DEBUG:boto:Headers: {} 2023-10-17T19:31:29.410 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:31:29.411 DEBUG:boto:Port: 8001 2023-10-17T19:31:29.411 DEBUG:boto:Params: {} 2023-10-17T19:31:29.412 DEBUG:boto:Token: None 2023-10-17T19:31:29.412 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:29 GMT /hbmuog-68/obj3 2023-10-17T19:31:29.413 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QPLxWiyIBRNW9Vjx3vxjoJT1zlc= 2023-10-17T19:31:29.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QPLxWiyIBRNW9Vjx3vxjoJT1zlc=', 'Content-Length': '0'} 2023-10-17T19:31:29.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eb18e049affab85-00652ee111-22c0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:31:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:29.416 DEBUG:boto:path=/hbmuog-68/obj3 2023-10-17T19:31:29.417 DEBUG:boto:auth_path=/hbmuog-68/obj3 2023-10-17T19:31:29.417 DEBUG:boto:Method: GET 2023-10-17T19:31:29.418 DEBUG:boto:Path: /hbmuog-68/obj3 2023-10-17T19:31:29.418 DEBUG:boto:Data: 2023-10-17T19:31:29.418 DEBUG:boto:Headers: {} 2023-10-17T19:31:29.419 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:31:29.419 DEBUG:boto:Port: 8001 2023-10-17T19:31:29.420 DEBUG:boto:Params: {} 2023-10-17T19:31:29.420 DEBUG:boto:Token: None 2023-10-17T19:31:29.421 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:31:29 GMT /hbmuog-68/obj3 2023-10-17T19:31:29.421 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2CAQ3/60JZeUaOYKH6XJd0hRi0Y= 2023-10-17T19:31:29.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2CAQ3/60JZeUaOYKH6XJd0hRi0Y=', 'Content-Length': '0'} 2023-10-17T19:31:29.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:29:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-68:4c84373e-d451-49bb-a8c9-c37d5e5e2160.8303.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077d8213cbd7d95ce-00652ee111-22c0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:31:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:29.424 DEBUG:boto:path=/hbmuog-69/ 2023-10-17T19:31:29.425 DEBUG:boto:auth_path=/hbmuog-69/ 2023-10-17T19:31:29.425 DEBUG:boto:Method: HEAD 2023-10-17T19:31:29.425 DEBUG:boto:Path: /hbmuog-69/ 2023-10-17T19:31:29.426 DEBUG:boto:Data: 2023-10-17T19:31:29.426 DEBUG:boto:Headers: {} 2023-10-17T19:31:29.427 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:31:29.427 DEBUG:boto:Port: 8001 2023-10-17T19:31:29.427 DEBUG:boto:Params: {} 2023-10-17T19:31:29.428 DEBUG:boto:Token: None 2023-10-17T19:31:29.428 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:29 GMT /hbmuog-69/ 2023-10-17T19:31:29.429 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ow54zHex6Wb630oSN9Ejwe5DYXM= 2023-10-17T19:31:29.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ow54zHex6Wb630oSN9Ejwe5DYXM=', 'Content-Length': '0'} 2023-10-17T19:31:29.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088b109c71b300046-00652ee111-22c0-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:29.432 DEBUG:boto:path=/hbmuog-69/obj3 2023-10-17T19:31:29.433 DEBUG:boto:auth_path=/hbmuog-69/obj3 2023-10-17T19:31:29.433 DEBUG:boto:Method: HEAD 2023-10-17T19:31:29.433 DEBUG:boto:Path: /hbmuog-69/obj3 2023-10-17T19:31:29.434 DEBUG:boto:Data: 2023-10-17T19:31:29.434 DEBUG:boto:Headers: {} 2023-10-17T19:31:29.435 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:31:29.435 DEBUG:boto:Port: 8001 2023-10-17T19:31:29.435 DEBUG:boto:Params: {} 2023-10-17T19:31:29.436 DEBUG:boto:Token: None 2023-10-17T19:31:29.436 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:29 GMT /hbmuog-69/obj3 2023-10-17T19:31:29.437 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bxDDz8Nfh9HZdmP0h9VKR/hL4qs= 2023-10-17T19:31:29.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:29 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bxDDz8Nfh9HZdmP0h9VKR/hL4qs=', 'Content-Length': '0'} 2023-10-17T19:31:29.439 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bab11e9a19deb4d9-00652ee111-22c0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:31:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:29.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:29.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:29.440 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hbmuog-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:29.493 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:29.494 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:29.545 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:31:29.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:29.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:29.546 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:29.594 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:29.595 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:29.645 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:31:29.646 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-10-17T19:31:29.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.649 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:29.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:29.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:31:29.746 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:31:29.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.747 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571089.530063_1066.1", 2023-10-17T19:31:29.747 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:29.530063Z" 2023-10-17T19:31:29.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.748 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:29.748 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:31:29.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.749 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:29.749 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:31:29.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.750 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:29.750 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:31:29.750 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.750 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:31:29.751 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:29.752 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:31:29.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:29.753 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:29.805 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:29.805 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:29.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:29.834+0000 7fe41df9b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:29.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:29.835+0000 7fe41df9b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:29.849 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697570965.601721_1052.1", 2023-10-17T19:31:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:31:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.601721Z", 2023-10-17T19:31:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:31:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:31:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:29.863 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:29.864 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:29.864 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:29.864 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:29.864 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:29.865 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:29.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:29.865 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:31:29.865 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:29.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:29.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:29.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.867 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:29.867 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:31:29.867 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:31:29.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:29.868 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:29Z" 2023-10-17T19:31:29.868 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:29.870 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697570965.601721_1052.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:29:25.601721Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:31:29Z" } 2023-10-17T19:31:29.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570965.601721_1052.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.601721Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:31:29.871 DEBUG:rgw_multi.tests:log_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:29.871 DEBUG:rgw_multi.tests:sync_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:29.871 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571089.530063_1066.1 target=1_1697570965.601721_1052.1 2023-10-17T19:31:34.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:34.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:34.874 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:34.934 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:34.935 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:34.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:34.966+0000 7f4f2324f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:34.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:34.967+0000 7f4f2324f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:34.983 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:34.984 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:34.984 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:34.984 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:34.985 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:34.985 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:34.986 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:34.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:34.987 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:34.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:34.987 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:34.988 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:34.988 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:34.988 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571089.530063_1066.1", 2023-10-17T19:31:34.989 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:34.989 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:31:34.990 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:34.990 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:29.530063Z", 2023-10-17T19:31:34.990 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:34.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:34.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:34.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:34.992 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:34.992 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:34.992 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:34.992 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:34.993 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:34.993 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:34.993 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:34.993 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:31:34.993 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:34.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:34.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:34.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:34.994 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:34.995 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:34.995 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:34.995 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:34.995 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:34.995 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:34.996 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:34.996 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:31:34.996 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:34.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:34.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:34.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:34.997 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:34.997 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:34.997 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:34.998 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:34.998 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:34.998 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:34.999 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:34.999 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:31:34.999 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:34.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.000 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:35.001 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:31:35.001 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:31:35.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.001 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:34Z" 2023-10-17T19:31:35.002 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:35.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571089.530063_1066.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:31:29.530063Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:31:34Z" } 2023-10-17T19:31:35.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571089.530063_1066.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:29.530063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:31:35.004 DEBUG:rgw_multi.tests:log_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:35.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:35.005 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:31:35.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.006 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.054 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.101 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[]} 2023-10-17T19:31:35.102 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-70 2023-10-17T19:31:35.102 DEBUG:boto:path=/hbmuog-70/ 2023-10-17T19:31:35.103 DEBUG:boto:auth_path=/hbmuog-70/ 2023-10-17T19:31:35.103 DEBUG:boto:Method: PUT 2023-10-17T19:31:35.104 DEBUG:boto:Path: /hbmuog-70/ 2023-10-17T19:31:35.104 DEBUG:boto:Data: 2023-10-17T19:31:35.104 DEBUG:boto:Headers: {} 2023-10-17T19:31:35.105 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:35.105 DEBUG:boto:Port: 8000 2023-10-17T19:31:35.106 DEBUG:boto:Params: {} 2023-10-17T19:31:35.106 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:31:35.107 DEBUG:boto:Token: None 2023-10-17T19:31:35.107 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:31:35 GMT /hbmuog-70/ 2023-10-17T19:31:35.108 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:aWKS2MzDbTSzS8Xyx8sHlgV4LIk= 2023-10-17T19:31:35.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aWKS2MzDbTSzS8Xyx8sHlgV4LIk=', 'Content-Length': '0'} 2023-10-17T19:31:35.124 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095e7bf9a6c0bd777-00652ee117-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:35.125 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-71 2023-10-17T19:31:35.126 DEBUG:boto:path=/hbmuog-71/ 2023-10-17T19:31:35.127 DEBUG:boto:auth_path=/hbmuog-71/ 2023-10-17T19:31:35.128 DEBUG:boto:Method: PUT 2023-10-17T19:31:35.129 DEBUG:boto:Path: /hbmuog-71/ 2023-10-17T19:31:35.129 DEBUG:boto:Data: 2023-10-17T19:31:35.130 DEBUG:boto:Headers: {} 2023-10-17T19:31:35.131 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:35.132 DEBUG:boto:Port: 8000 2023-10-17T19:31:35.132 DEBUG:boto:Params: {} 2023-10-17T19:31:35.133 DEBUG:boto:Token: None 2023-10-17T19:31:35.134 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:31:35 GMT /hbmuog-71/ 2023-10-17T19:31:35.134 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PgMxva4TIYONlT9cjAlq0tP3rsA= 2023-10-17T19:31:35.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PgMxva4TIYONlT9cjAlq0tP3rsA=', 'Content-Length': '0'} 2023-10-17T19:31:35.149 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021b275ed8fef4503-00652ee117-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:35.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:35.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:35.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.233 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:31:35.247 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:31:35.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.248 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:31:35.248 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:31:35.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.248 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:35.249 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:31:35.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.249 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:35.249 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:31:35.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.250 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:35.250 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:31:35.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.251 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:31:35.252 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:35.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:31:35.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:35.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:35.254 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:35.309 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:35.344+0000 7f98e631c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:35.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:35.345+0000 7f98e631c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:35.359 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:35.360 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:35.360 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:35.360 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:35.360 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:35.361 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:35.361 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.361 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:35.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.362 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:35.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:35.362 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:35.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:31:35.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:35.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:31:35.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:35.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:31:35.363 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:35.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:35.364 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:35.365 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:35.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:35.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:35.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:35.366 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:31:35.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.367 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:35.367 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:35.367 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:35.367 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:35.367 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:35.368 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:35.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:35.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:31:35.368 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.369 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:35.369 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:35.369 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:35.370 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:35.370 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:35.370 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:35.370 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:35.370 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:31:35.371 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.372 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:35.372 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:31:35.372 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:31:35.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.373 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:35Z" 2023-10-17T19:31:35.373 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:35.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571095.147990_1074.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:31:35.147990Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:31:35Z" } 2023-10-17T19:31:35.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:31:35.375 DEBUG:rgw_multi.tests:log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:35.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:35.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:31:35.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.377 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.427 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.427 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.474 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:31:35.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.476 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.531 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.531 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.577 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:31:35.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-70', '--dest-bucket=hbmuog-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-70', '--dest-bucket=hbmuog-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.579 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-70 --dest-bucket=hbmuog-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.635 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.636 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.686 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:31:35.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.687 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.787 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:31:35.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.788 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:35.837 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:35.837 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:35.891 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:35.891 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:35.892 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 18, 2023-10-17T19:31:35.892 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:31:35.892 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:31:35.892 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:31:35.892 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:35.893 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:31:35.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.893 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:31:35.893 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:31:35.893 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:31:35.894 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:31:35.894 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.894 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:31:35.894 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.895 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:31:35.895 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:31:35.895 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:31:35.895 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:31:35.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.896 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:31:35.896 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:31:35.896 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.896 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:31:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:31:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:31:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:31:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:31:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:31:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:31:35.898 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:31:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:31:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:31:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:31:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:31:35.900 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:31:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:31:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:31:35.901 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:31:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:31:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:31:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:31:35.902 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:31:35.903 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:31:35.903 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:31:35.903 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:31:35.903 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:31:35.903 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.904 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.905 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.905 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:31:35.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.906 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:31:35.906 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:31:35.907 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:31:35.907 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:31:35.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.908 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.909 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:31:35.909 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:31:35.910 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:31:35.910 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:31:35.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.911 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:31:35.911 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:31:35.912 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:31:35.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.913 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:31:35.913 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:31:35.913 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:31:35.914 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:31:35.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.915 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:31:35.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.915 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:31:35.915 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:31:35.916 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-70", 2023-10-17T19:31:35.916 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:31:35.916 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:31:35.916 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.917 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:31:35.917 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-71", 2023-10-17T19:31:35.917 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:31:35.917 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:31:35.918 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.918 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:31:35.918 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:31:35.919 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:31:35.919 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:31:35.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.919 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:31:35.920 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:31:35.920 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:31:35.920 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:31:35.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.921 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.921 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:31:35.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.922 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:31:35.922 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.923 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:31:35.924 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.925 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:31:35.925 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:31:35.925 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:31:35.925 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:31:35.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.926 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:31:35.926 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:31:35.926 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.926 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:31:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:31:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:31:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:31:35.928 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:31:35.928 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:31:35.928 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:31:35.928 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:31:35.928 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:31:35.929 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:31:35.929 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:31:35.929 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.929 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.930 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:31:35.930 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:31:35.931 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:31:35.931 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:31:35.931 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.931 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:31:35.932 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:31:35.932 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:31:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:31:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:31:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:31:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:31:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:31:35.934 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:31:35.934 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:31:35.934 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.935 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.935 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.935 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:31:35.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.936 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:31:35.936 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:31:35.936 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:31:35.937 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:31:35.937 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.937 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.937 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:31:35.938 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:31:35.938 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:31:35.938 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:31:35.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.938 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:31:35.939 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:31:35.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.939 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:31:35.940 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:31:35.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.940 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:31:35.940 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:31:35.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.941 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:31:35.941 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:31:35.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.942 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:31:35.942 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:31:35.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:35.944 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:31:35.944 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:31:35.944 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:35.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.945 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:31:35.946 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:31:35.946 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:31:35.946 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:31:35.946 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:31:35.947 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:31:35.947 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:31:35.947 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:31:35.947 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:31:35.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.948 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:31:35.948 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:31:35.948 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:31:35.948 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:31:35.949 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:31:35.949 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:31:35.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.949 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:31:35.950 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:31:35.950 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:31:35.950 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:31:35.950 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:31:35.951 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:31:35.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.951 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:31:35.951 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:31:35.952 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:31:35.952 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:31:35.952 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:31:35.952 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:31:35.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.953 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:31:35.953 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:31:35.953 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:31:35.954 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:31:35.954 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:31:35.954 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:31:35.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:35.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:35.955 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:31:35.955 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:31:35.955 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:35.955 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:35.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:31:35.957 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:31:36.003 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:31:36.067 DEBUG:boto:path=/hbmuog-70/ 2023-10-17T19:31:36.067 DEBUG:boto:auth_path=/hbmuog-70/ 2023-10-17T19:31:36.068 DEBUG:boto:Method: HEAD 2023-10-17T19:31:36.068 DEBUG:boto:Path: /hbmuog-70/ 2023-10-17T19:31:36.069 DEBUG:boto:Data: 2023-10-17T19:31:36.069 DEBUG:boto:Headers: {} 2023-10-17T19:31:36.069 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:36.070 DEBUG:boto:Port: 8000 2023-10-17T19:31:36.070 DEBUG:boto:Params: {} 2023-10-17T19:31:36.071 DEBUG:boto:Token: None 2023-10-17T19:31:36.071 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:36 GMT /hbmuog-70/ 2023-10-17T19:31:36.072 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA= 2023-10-17T19:31:36.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA=', 'Content-Length': '0'} 2023-10-17T19:31:36.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a8e4616bafec702-00652ee118-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:36.205 DEBUG:boto:path=/hbmuog-70/obj1 2023-10-17T19:31:36.206 DEBUG:boto:auth_path=/hbmuog-70/obj1 2023-10-17T19:31:36.206 DEBUG:boto:Method: PUT 2023-10-17T19:31:36.207 DEBUG:boto:Path: /hbmuog-70/obj1 2023-10-17T19:31:36.207 DEBUG:boto:Data: 2023-10-17T19:31:36.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:31:36.208 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:36.209 DEBUG:boto:Port: 8000 2023-10-17T19:31:36.209 DEBUG:boto:Params: {} 2023-10-17T19:31:36.209 DEBUG:boto:Token: None 2023-10-17T19:31:36.210 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:31:36 GMT /hbmuog-70/obj1 2023-10-17T19:31:36.210 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w7e3hTc98W+YT2PD32+zm4hE5Wc= 2023-10-17T19:31:36.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w7e3hTc98W+YT2PD32+zm4hE5Wc='} 2023-10-17T19:31:36.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ccfa923b795c69d-00652ee118-244a-a1'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:36.243 DEBUG:boto:path=/hbmuog-70/ 2023-10-17T19:31:36.243 DEBUG:boto:auth_path=/hbmuog-70/ 2023-10-17T19:31:36.243 DEBUG:boto:Method: HEAD 2023-10-17T19:31:36.244 DEBUG:boto:Path: /hbmuog-70/ 2023-10-17T19:31:36.244 DEBUG:boto:Data: 2023-10-17T19:31:36.245 DEBUG:boto:Headers: {} 2023-10-17T19:31:36.245 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:36.245 DEBUG:boto:Port: 8000 2023-10-17T19:31:36.246 DEBUG:boto:Params: {} 2023-10-17T19:31:36.246 DEBUG:boto:Token: None 2023-10-17T19:31:36.247 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:31:36 GMT /hbmuog-70/ 2023-10-17T19:31:36.247 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA= 2023-10-17T19:31:36.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA=', 'Content-Length': '0'} 2023-10-17T19:31:36.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbe42ec713e86a16-00652ee118-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:36.253 DEBUG:boto:path=/hbmuog-70/obj2 2023-10-17T19:31:36.253 DEBUG:boto:auth_path=/hbmuog-70/obj2 2023-10-17T19:31:36.253 DEBUG:boto:Method: PUT 2023-10-17T19:31:36.254 DEBUG:boto:Path: /hbmuog-70/obj2 2023-10-17T19:31:36.254 DEBUG:boto:Data: 2023-10-17T19:31:36.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:31:36.255 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:31:36.255 DEBUG:boto:Port: 8000 2023-10-17T19:31:36.256 DEBUG:boto:Params: {} 2023-10-17T19:31:36.256 DEBUG:boto:Token: None 2023-10-17T19:31:36.257 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:31:36 GMT /hbmuog-70/obj2 2023-10-17T19:31:36.257 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Lh9tcKEZHehNqpmxnxth4FFgQO0= 2023-10-17T19:31:36.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lh9tcKEZHehNqpmxnxth4FFgQO0='} 2023-10-17T19:31:36.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f2eaa6d64b5baf9-00652ee118-244a-a1'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:31:36.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.304 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:36.389 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:36.466 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:31:36.501 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:31:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:31:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:31:36.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:31:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:31:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.504 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:31:36.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.505 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:31:36.506 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:36.507 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:31:36.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.508 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:36.586 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:36.587 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:36.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:36.635+0000 7fb4954fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:36.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:31:36.636+0000 7fb4954fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:31:36.657 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:36.658 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:36.658 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:36.658 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:36.658 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:36.659 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:31:36.659 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:36.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.659 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:36.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.660 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:36.660 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.660 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:36.660 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:31:36.660 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.661 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:31:36.661 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.661 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:31:36.661 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:36.661 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.662 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:36.662 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.662 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:36.663 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:31:36.663 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.663 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:36.663 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.664 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:36.665 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.665 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:36.665 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:31:36.665 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.665 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:36.666 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.666 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:31:36.666 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:36.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.667 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:36.667 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.667 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:31:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:31:36.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.669 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:36.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.669 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:36.670 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:31:36.670 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:31:36.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.670 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:31:36.671 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:36.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571095.147990_1074.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:31:35.147990Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:31:36Z" } 2023-10-17T19:31:36.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:31:36.673 DEBUG:rgw_multi.tests:log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:36.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:31:36.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:31:36.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.675 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:36.742 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:31:36.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.804 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:36.804 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:36.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.805 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:31:36.805 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:31:36.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.806 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:36.806 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:36.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.806 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:31:36.807 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:31:36.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:31:36.808 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:36.808 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:31:36.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:36.810 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:36.862 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:36.862 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:36.909 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:36.909 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.910 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:36.911 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:36.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.913 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.914 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:36.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.915 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:36.915 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:36.916 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:36.917 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:36.918 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:36.918 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:36.918 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:36.918 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:36.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:31:36.921 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:36.922 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:36Z" } 2023-10-17T19:31:36.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:36.923 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:36.923 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:36.923 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:31:41.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:41.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:41.925 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:41.986 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:41.986 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:42.038 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:42.038 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:42.038 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:42.039 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:42.039 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:42.039 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:42.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.039 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:42.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:42.040 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:42.040 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:42.040 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:42.041 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:42.041 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:42.041 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:42.041 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:42.041 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:42.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:42.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:42.043 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:42.043 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:42.043 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:42.043 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:42.044 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:42.044 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:42.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:42.044 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:42.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:42.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:42.045 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:42.045 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:42.045 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:42.046 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:42.047 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:42.048 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:42.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:42.050 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:42Z" 2023-10-17T19:31:42.051 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:42.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:42Z" } 2023-10-17T19:31:42.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:42.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:42.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:42.054 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:31:47.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:47.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:47.056 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:47.117 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:47.118 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:47.168 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:47.168 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:47.168 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:47.169 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:47.169 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:47.169 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:47.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.169 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:47.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:47.170 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:47.170 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:47.170 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:47.171 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:47.171 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:47.171 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:47.171 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:47.171 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:47.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:47.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:47.172 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:47.174 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:47.174 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:47.174 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:47.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:47.175 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:47.176 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:47.177 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:47.178 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:47.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:47.180 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:47Z" 2023-10-17T19:31:47.181 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:47.182 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:47Z" } 2023-10-17T19:31:47.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:47.183 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:47.184 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:47.184 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:31:52.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:52.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:52.187 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:52.301 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:52.306 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:52.306 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:52.306 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:52.306 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:52Z" 2023-10-17T19:31:52.317 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:52.318 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:52Z" } 2023-10-17T19:31:52.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:52.319 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:52.320 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:52.320 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:31:57.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:57.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:31:57.322 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:31:57.383 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:31:57.383 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:31:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:31:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:31:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:31:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:31:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:31:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:31:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:31:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:31:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:31:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:31:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:31:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:31:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:31:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:31:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:31:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:31:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:31:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:31:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:31:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:31:57Z" 2023-10-17T19:31:57.450 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:31:57.451 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:31:57Z" } 2023-10-17T19:31:57.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:31:57.452 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:31:57.453 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:31:57.453 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:02.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:02.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:02.456 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:02.516 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:02.517 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:02.567 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:02.568 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:02.569 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:02.572 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:02.573 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:02.573 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:02.573 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:02.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:02.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:02.574 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:02.574 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:02.574 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:02.575 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:02.575 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:02.575 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:02.575 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:02.575 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:02.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:02.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:02.576 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:02.576 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:02.577 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:02.577 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:02.577 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:02.577 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:02.578 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:02.578 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:02.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:02.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:02.578 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:02.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.579 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:02.579 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:02.579 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:02.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:02.580 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:02Z" 2023-10-17T19:32:02.580 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:02.581 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:02Z" } 2023-10-17T19:32:02.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:02.582 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:02.583 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:02.583 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:07.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:07.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:07.585 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:07.646 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:07.646 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:07.697 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:07.697 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:07.697 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:07.698 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:07.698 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:07.698 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:07.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:07.700 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:07.700 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:07.700 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:07.700 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:07.701 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:07.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:07.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:07.701 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:07.702 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:07.702 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:07.702 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:07.702 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:07.703 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:07.703 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:07.703 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:07.703 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:07.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:07.704 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:07.704 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:07.704 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:07.704 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:07.705 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:07.705 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:07.705 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:07.705 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:07.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:07.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:07.706 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:07.707 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:07.707 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:07.707 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:07.707 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:07.707 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:07.708 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:07.708 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:07.708 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:07.708 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:07.709 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:07.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.709 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:07.709 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:07.709 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:07.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:07.710 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:07Z" 2023-10-17T19:32:07.710 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:07.711 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:07Z" } 2023-10-17T19:32:07.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:07.712 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:07.713 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:07.713 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:12.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:12.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:12.715 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:12.774 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:12.774 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:12.824 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:12.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:12.826 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:12.827 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:12.828 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:12.828 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:12.828 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:12.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:12.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:12.829 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:12.829 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:12.829 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:12.829 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:12.830 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:12.830 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:12.830 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:12.830 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:12.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:12.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:12.831 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:12.831 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:12.831 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:12.832 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:12.832 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:12.832 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:12.832 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:12.832 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:12.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:12.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:12.833 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:12.834 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:12.834 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:12.834 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:12.834 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:12.834 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:12.835 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:12.835 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:12.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:12.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:12.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:12.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:12.837 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:12Z" 2023-10-17T19:32:12.837 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:12.838 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:12Z" } 2023-10-17T19:32:12.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:12.839 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:12.839 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:12.840 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:17.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:17.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:17.842 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:17.903 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:17.959 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:17.959 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:17.959 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:17.959 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:17.960 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:17.960 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:17.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.960 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:17.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:17.961 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:17.961 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:17.961 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:17.961 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:17.962 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:17.962 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:17.962 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:17.962 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:17.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:17.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:17.963 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:17.963 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:17.964 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:17.964 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:17.964 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:17.964 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:17.965 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:17.965 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:17.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:17.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:17.966 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:17.966 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:17.966 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:17.966 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:17.966 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:17.967 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:17.967 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:17.967 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:17.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:17.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:17.968 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:17.968 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:17.968 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:17.969 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:17.969 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:17.969 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:17.969 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:17.970 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:17.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:17.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:17.970 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:17.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.971 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:17.971 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:17.971 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:17.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:17.972 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:17Z" 2023-10-17T19:32:17.972 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:17.973 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:17Z" } 2023-10-17T19:32:17.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:17.974 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:17.974 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:17.975 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:22.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:22.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:22.977 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:23.037 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:23.038 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:23.088 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:23.088 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:23.088 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:23.088 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:23.089 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:23.089 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:23.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.089 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:23.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:23.090 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:23.090 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:23.090 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:23.090 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:23.091 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:23.091 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:23.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:23.091 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:23.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:23.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:23.092 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:23.092 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:23.092 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:23.093 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:23.093 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:23.093 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:23.093 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:23.094 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:23.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:23.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:23.094 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:23.095 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:23.095 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:23.095 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:23.095 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:23.095 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:23.096 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:23.096 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:23.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:23.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:23.097 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:23.097 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:23.097 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:23.097 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:23.097 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:23.098 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:23.098 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:23.098 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:23.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:23.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:23.099 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:23.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.099 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:23.099 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:23.099 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:23.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:23.100 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:23Z" 2023-10-17T19:32:23.100 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:23.101 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:23Z" } 2023-10-17T19:32:23.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:23.102 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:23.102 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:23.103 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:28.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:28.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:28.105 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:28.167 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:28.167 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:28.218 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:28.219 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:28.219 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:28.219 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:28.219 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:28.219 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:28.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.220 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:28.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:28.220 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:28.221 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:28.221 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:28.221 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:28.221 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:28.221 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:28.222 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:28.222 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:28.222 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:28.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:28Z" 2023-10-17T19:32:28.232 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:28.233 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:28Z" } 2023-10-17T19:32:28.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:28.234 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:28.235 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:28.236 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:33.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:33.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:33.238 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:33.299 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:33.299 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:33.353 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:33.354 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:33.354 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:33.354 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:33.354 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:33.355 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:33.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.355 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:33.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:33.355 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:33.356 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:33.356 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:33.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:33.356 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:33.356 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:33.357 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:33.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:33.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:33.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:33.358 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:33.358 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:33.358 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:33.358 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:33.358 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:33.359 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:33.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:33.360 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:33.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:33.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:33.361 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:33.361 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:33.361 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:33.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:33.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:33.362 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:33.362 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:33.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:33.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:33.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:33.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:33.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:33.364 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:33.365 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:33.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:33.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:33.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:33.366 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:33.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:33.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:33.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:33.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.367 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:33.367 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:33.367 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:33.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:33.368 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:33Z" 2023-10-17T19:32:33.368 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:33.369 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:33Z" } 2023-10-17T19:32:33.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:33.370 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:33.371 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:33.371 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:38.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:38.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:38.374 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:38.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:38.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:38.488 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:38.489 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:38.489 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:38.489 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:38.489 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:38.490 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:38.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.490 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:38.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:38.491 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:38.491 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:38.491 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:38.491 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:38.491 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:38.493 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:38.493 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:38.493 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:38.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:38.494 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:38.494 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:38.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:38.494 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:38.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:38.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:38.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:38.495 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:38.496 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:38.496 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:38.496 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:38.496 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:38.497 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:38.497 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:38.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:38.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:38.498 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:38.498 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:38.498 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:38.499 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:38.499 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:38.499 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:38.499 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:38.500 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:38.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:38.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:38.501 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:38.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.502 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:38.502 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:38.502 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:38.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:38.503 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:38Z" 2023-10-17T19:32:38.503 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:38.504 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:38Z" } 2023-10-17T19:32:38.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:38.505 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:38.505 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:38.506 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:43.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:43.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:43.509 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:43.569 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:43.570 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:43.620 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:43Z" 2023-10-17T19:32:43.633 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:43.634 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:43Z" } 2023-10-17T19:32:43.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:43.635 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:43.635 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:43.636 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:48.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:48.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:48.638 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:48.699 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:48.699 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:48.749 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:48.750 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:48.750 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:48.750 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:48.751 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:48.751 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:48.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.752 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:48.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:48.752 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:48.752 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:48.753 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:48.753 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:48.753 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:48.753 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:48.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:48.754 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:48.754 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:48.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:48.755 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:48.755 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:48.755 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:48.756 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:48.756 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:48.756 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:48.756 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:48.756 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:48.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:48.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:48.757 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:48.758 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:48.758 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:48.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:48.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:48.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:48.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:48.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:48.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:48.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:48.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:48.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:48.761 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:48.761 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:48.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:48.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:48.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:48.762 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:48.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:48.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:48.763 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:48.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.763 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:48.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:48.764 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:48.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:48.764 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:48Z" 2023-10-17T19:32:48.765 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:48.766 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:48Z" } 2023-10-17T19:32:48.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:48.767 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:48.767 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:48.768 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:53.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:53.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:53.770 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:53.832 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:53.832 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:53.883 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:53.883 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:53.883 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:53.884 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:53.884 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:53.884 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:53.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.885 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:53.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:53.885 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:53.885 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:53.885 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:53.886 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:53.886 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:53.886 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:53.886 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:53.887 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:53.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:53.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:53.887 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:53.888 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:53.888 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:53.888 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:53.888 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:53.888 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:53.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:53.889 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:53.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:53.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:53.890 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:53.890 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:53.890 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:53.890 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:53.891 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:53.891 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:53.891 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:53.891 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:53.891 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:53.893 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:53.893 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:53.893 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:53.893 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:53.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:53.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:53.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:53.894 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:53.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.895 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:53.895 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:53.895 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:53.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:53.895 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:53Z" 2023-10-17T19:32:53.896 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:53.897 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:53Z" } 2023-10-17T19:32:53.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:53.898 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:53.898 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:53.899 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:32:58.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:58.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:32:58.902 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:32:58.963 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:32:58.963 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:32:59.015 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:32:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:32:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:32:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:32:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:32:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:32:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:32:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:32:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:32:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:32:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:32:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:32:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:32:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:32:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:32:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:32:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:32:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:32:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:32:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:32:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:32:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:32:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:32:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:32:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:32:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:32:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:32:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:32:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:32:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:32:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:32:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:32:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:32:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:32:59Z" 2023-10-17T19:32:59.029 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:32:59.030 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:32:59Z" } 2023-10-17T19:32:59.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:32:59.031 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:32:59.031 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:32:59.032 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:04.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:04.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:04.034 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:04.095 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:04.096 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:04.148 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:04.148 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:04.148 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:04.149 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:04.149 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:04.149 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:04.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.150 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:04.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:04.150 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:04.150 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:04.151 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:04.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:04.151 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:04.152 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:04.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:04.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:04.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:04.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:04.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:04.153 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:04.154 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:04.154 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:04.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:04.155 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:04.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:04.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:04.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:04.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:04.156 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:04.157 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:04.157 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:04.157 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:04.158 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:04.158 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:04.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:04.159 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:04.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:04.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:04.160 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:04.161 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:04.161 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:04.162 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:04.162 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:04.162 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:04.163 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:04.163 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:04.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:04.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:04.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:04.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.164 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:04.164 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:04.165 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:04.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:04.165 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:04Z" 2023-10-17T19:33:04.166 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:04.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:04Z" } 2023-10-17T19:33:04.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:04.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:04.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:04.170 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:09.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:09.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:09.172 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:09.233 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:09.234 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:09.284 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:09Z" 2023-10-17T19:33:09.296 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:09.297 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:09Z" } 2023-10-17T19:33:09.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:09.298 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:09.299 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:09.299 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:14.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:14.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:14.302 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:14.365 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:14.417 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:14.418 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:14.418 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:14.418 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:14.419 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:14.419 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:14.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.420 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:14.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:14.420 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:14.421 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:14.421 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:14.421 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:14.421 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:14.422 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:14.422 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:14.422 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:14.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:14.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:14.424 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:14.424 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:14.424 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:14.424 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:14.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:14.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:14.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:14.426 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:14.427 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:14.427 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:14.427 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:14.428 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:14.428 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:14.429 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:14.429 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:14.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:14.430 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:14.431 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:14.431 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:14.431 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:14.432 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:14.432 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:14.432 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:14.433 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:14.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:14.434 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:14.434 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:14.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.435 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:14.435 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:14.436 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:14.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:14.436 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:14Z" 2023-10-17T19:33:14.437 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:14.439 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:14Z" } 2023-10-17T19:33:14.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:14.440 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:14.441 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:14.442 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:19.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:19.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:19.445 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:19.505 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:19.506 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:19.557 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:19.557 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:19.558 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:19.558 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:19.558 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:19.558 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:19.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.559 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:19.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:19.559 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:19.559 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:19.560 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:19.561 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:19.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:19.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:19.561 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:19.562 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:19.562 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:19.562 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:19.562 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:19.562 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:19.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:19.563 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:19.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:19.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:19.564 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:19.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:19.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:19.566 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:19.567 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:19.567 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:19.567 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:19.567 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:19.567 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:19.568 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:19.568 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:19.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:19.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:19.568 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:19.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.569 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:19.569 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:19.569 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:19.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:19.570 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:19Z" 2023-10-17T19:33:19.570 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:19.571 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:19Z" } 2023-10-17T19:33:19.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:19.572 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:19.572 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:19.573 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:24.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:24.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:24.575 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:24.639 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:24.639 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:24.691 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:24.691 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:24.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:24.693 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:24.696 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:24.697 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:24.698 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:24.699 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:24.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:24.702 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:24.704 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:24.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:24.706 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:24Z" 2023-10-17T19:33:24.706 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:24.707 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:24Z" } 2023-10-17T19:33:24.708 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:24.708 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:24.709 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:24.709 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:29.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:29.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:29.711 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:29.772 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:29.773 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:29.824 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:29Z" 2023-10-17T19:33:29.839 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:29.840 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:29Z" } 2023-10-17T19:33:29.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:29.841 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:29.841 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:29.842 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:34.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:34.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:34.844 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:34.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:34.906 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:34.957 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:34.958 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:34.958 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:34.958 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:34.959 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:34.959 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:34.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.959 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:34.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:34.960 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:34.960 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:34.960 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:34.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:34.961 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:34.961 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:34.961 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:34.962 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:34.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:34.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:34.963 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:34.963 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:34.963 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:34.963 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:33:34.964 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:34.964 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:34.964 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:34.964 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:33:34.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:34.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:34.965 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:34.965 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:34.966 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:34.966 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:34.966 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:34.966 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:34.967 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:34.967 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:34.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:34.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:34.968 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:34.968 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:34.968 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:34.968 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:33:34.968 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:34.969 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:34.969 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:34.969 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:33:34.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:34.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:34.970 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:34.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.970 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:34.970 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:34.971 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:34.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:34.971 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:34Z" 2023-10-17T19:33:34.971 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:34.972 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.291154Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046716", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:29:25.758413Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:34Z" } 2023-10-17T19:33:34.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:33:34.973 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:34.974 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:33:34.974 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:33:39.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:39.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:39.977 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:40.039 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.039 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.091 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:40.091 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:40.091 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:40.092 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:40.092 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:40.092 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:40.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.092 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:40.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.093 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:40.093 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.093 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:40.094 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:40.094 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.094 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.094 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.094 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:40.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.095 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:40.096 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.096 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:40.096 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:33:40.096 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.097 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.097 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:33:40.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.098 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:40.098 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.098 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:40.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:40.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.099 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:40.099 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.099 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:40.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.100 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:40.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.101 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:40.101 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:33:40.101 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.101 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:40.102 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.102 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:33:40.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:40.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.104 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:33:40.104 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:40.105 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:40Z" } 2023-10-17T19:33:40.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:33:40.106 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:40.107 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:40.107 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:33:40.107 DEBUG:boto:path=/hbmuog-70/ 2023-10-17T19:33:40.108 DEBUG:boto:auth_path=/hbmuog-70/ 2023-10-17T19:33:40.108 DEBUG:boto:Method: HEAD 2023-10-17T19:33:40.109 DEBUG:boto:Path: /hbmuog-70/ 2023-10-17T19:33:40.109 DEBUG:boto:Data: 2023-10-17T19:33:40.110 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.110 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:33:40.110 DEBUG:boto:Port: 8001 2023-10-17T19:33:40.111 DEBUG:boto:Params: {} 2023-10-17T19:33:40.111 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:33:40.112 DEBUG:boto:Token: None 2023-10-17T19:33:40.113 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-70/ 2023-10-17T19:33:40.113 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:NaUmoEY3n75Er7cJBil4YFezxMU= 2023-10-17T19:33:40.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NaUmoEY3n75Er7cJBil4YFezxMU=', 'Content-Length': '0'} 2023-10-17T19:33:40.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000281b88e0af011ed4-00652ee194-2444-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.120 DEBUG:boto:path=/hbmuog-70/obj1 2023-10-17T19:33:40.120 DEBUG:boto:auth_path=/hbmuog-70/obj1 2023-10-17T19:33:40.121 DEBUG:boto:Method: HEAD 2023-10-17T19:33:40.121 DEBUG:boto:Path: /hbmuog-70/obj1 2023-10-17T19:33:40.122 DEBUG:boto:Data: 2023-10-17T19:33:40.122 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.122 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:33:40.123 DEBUG:boto:Port: 8001 2023-10-17T19:33:40.123 DEBUG:boto:Params: {} 2023-10-17T19:33:40.124 DEBUG:boto:Token: None 2023-10-17T19:33:40.124 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-70/obj1 2023-10-17T19:33:40.125 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pFM25cdRtoDz5hEQDVkUw0CIwbU= 2023-10-17T19:33:40.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFM25cdRtoDz5hEQDVkUw0CIwbU=', 'Content-Length': '0'} 2023-10-17T19:33:40.127 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006d75e4c8782873fa-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.127 DEBUG:boto:path=/hbmuog-70/obj2 2023-10-17T19:33:40.128 DEBUG:boto:auth_path=/hbmuog-70/obj2 2023-10-17T19:33:40.128 DEBUG:boto:Method: HEAD 2023-10-17T19:33:40.128 DEBUG:boto:Path: /hbmuog-70/obj2 2023-10-17T19:33:40.129 DEBUG:boto:Data: 2023-10-17T19:33:40.129 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.130 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:33:40.130 DEBUG:boto:Port: 8001 2023-10-17T19:33:40.130 DEBUG:boto:Params: {} 2023-10-17T19:33:40.131 DEBUG:boto:Token: None 2023-10-17T19:33:40.131 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-70/obj2 2023-10-17T19:33:40.132 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ltV+3VEjEOHGxE4WyKEQObzsbLs= 2023-10-17T19:33:40.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ltV+3VEjEOHGxE4WyKEQObzsbLs=', 'Content-Length': '0'} 2023-10-17T19:33:40.134 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001c6ce76ef1e911e4-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.134 DEBUG:boto:path=/hbmuog-71/ 2023-10-17T19:33:40.135 DEBUG:boto:auth_path=/hbmuog-71/ 2023-10-17T19:33:40.135 DEBUG:boto:Method: HEAD 2023-10-17T19:33:40.135 DEBUG:boto:Path: /hbmuog-71/ 2023-10-17T19:33:40.136 DEBUG:boto:Data: 2023-10-17T19:33:40.136 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.137 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:33:40.137 DEBUG:boto:Port: 8001 2023-10-17T19:33:40.137 DEBUG:boto:Params: {} 2023-10-17T19:33:40.138 DEBUG:boto:Token: None 2023-10-17T19:33:40.138 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-71/ 2023-10-17T19:33:40.139 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ts7HkmSqkGypTWU0I/bnzs6h3To= 2023-10-17T19:33:40.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts7HkmSqkGypTWU0I/bnzs6h3To=', 'Content-Length': '0'} 2023-10-17T19:33:40.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3430794fe4424b9-00652ee194-2444-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.142 DEBUG:boto:path=/hbmuog-71/obj1 2023-10-17T19:33:40.143 DEBUG:boto:auth_path=/hbmuog-71/obj1 2023-10-17T19:33:40.143 DEBUG:boto:Method: HEAD 2023-10-17T19:33:40.144 DEBUG:boto:Path: /hbmuog-71/obj1 2023-10-17T19:33:40.144 DEBUG:boto:Data: 2023-10-17T19:33:40.144 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.145 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:33:40.145 DEBUG:boto:Port: 8001 2023-10-17T19:33:40.146 DEBUG:boto:Params: {} 2023-10-17T19:33:40.146 DEBUG:boto:Token: None 2023-10-17T19:33:40.147 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-71/obj1 2023-10-17T19:33:40.147 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AbxSEp/FtzkFK6nAaqlV39c1hJY= 2023-10-17T19:33:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AbxSEp/FtzkFK6nAaqlV39c1hJY=', 'Content-Length': '0'} 2023-10-17T19:33:40.149 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fac36d797d86a4f9-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.152 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2023-10-17T19:33:40.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.154 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:40.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.239 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:33:40.252 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:33:40.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.253 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:33:40.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:33:40.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.254 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:40.254 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:33:40.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.255 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:40.255 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:33:40.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.256 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:40.256 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:33:40.256 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.256 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:33:40.257 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:33:40.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.259 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:40.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:40.339+0000 7fb6530f8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:40.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:40.340+0000 7fb6530f8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:40.354 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:40.354 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:40.355 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:40.355 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:40.355 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:40.355 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:33:40.356 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.356 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:40.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.357 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:40.357 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.357 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.357 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:33:40.357 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.358 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:33:40.358 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.358 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:33:40.358 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.359 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:40.359 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.360 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.360 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:40.360 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.360 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.361 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:33:40.361 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.362 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:40.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.362 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:40.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:33:40.363 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:40.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.365 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.365 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:40.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.366 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:33:40.366 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:40.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.367 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:40.367 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:33:40.367 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:33:40.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.368 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:33:40.368 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:40.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571095.147990_1074.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:31:35.147990Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:33:40Z" } 2023-10-17T19:33:40.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:33:40.370 DEBUG:rgw_multi.tests:log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.371 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:33:40.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.372 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:40.419 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.419 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.466 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:33:40.466 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-72 2023-10-17T19:33:40.467 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:33:40.467 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:33:40.468 DEBUG:boto:Method: PUT 2023-10-17T19:33:40.468 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:33:40.469 DEBUG:boto:Data: 2023-10-17T19:33:40.469 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.469 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:40.470 DEBUG:boto:Port: 8000 2023-10-17T19:33:40.470 DEBUG:boto:Params: {} 2023-10-17T19:33:40.471 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:33:40.471 DEBUG:boto:Token: None 2023-10-17T19:33:40.472 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-72/ 2023-10-17T19:33:40.472 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zjYPRiv3a1qq9D495ekAPt4ok2E= 2023-10-17T19:33:40.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zjYPRiv3a1qq9D495ekAPt4ok2E=', 'Content-Length': '0'} 2023-10-17T19:33:40.491 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b7937fd78b24c33-00652ee194-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.491 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-73 2023-10-17T19:33:40.492 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:33:40.492 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:33:40.493 DEBUG:boto:Method: PUT 2023-10-17T19:33:40.493 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:33:40.494 DEBUG:boto:Data: 2023-10-17T19:33:40.494 DEBUG:boto:Headers: {} 2023-10-17T19:33:40.495 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:40.495 DEBUG:boto:Port: 8000 2023-10-17T19:33:40.496 DEBUG:boto:Params: {} 2023-10-17T19:33:40.496 DEBUG:boto:Token: None 2023-10-17T19:33:40.497 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:33:40 GMT /hbmuog-73/ 2023-10-17T19:33:40.497 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:4l/29SmfyYiK4Y2Um93KqYXIjtY= 2023-10-17T19:33:40.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4l/29SmfyYiK4Y2Um93KqYXIjtY=', 'Content-Length': '0'} 2023-10-17T19:33:40.513 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5f45ef0853b3a6f-00652ee194-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:40.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.514 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:40.565 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.565 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.600 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:33:40.615 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:33:40.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.616 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:33:40.616 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:33:40.511436Z" 2023-10-17T19:33:40.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.617 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:40.617 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:33:40.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.618 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:40.618 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:33:40.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.618 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:33:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.619 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:33:40.620 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:33:40.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:40.622 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:40.676 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.676 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:40.709+0000 7f0d6358a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:40.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:40.710+0000 7f0d6358a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:40.724 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:40.724 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:40.724 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:40.724 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:40.725 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:40.725 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:33:40.725 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.725 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:40.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.726 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:40.726 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.726 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.727 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:33:40.727 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.727 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:33:40.727 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.727 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:33:40.728 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.728 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.728 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:40.729 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.729 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.729 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:40.729 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.729 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.730 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.730 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:33:40.730 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.730 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.731 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:40.731 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.731 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.732 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:40.732 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.732 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.732 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.732 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:33:40.733 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.733 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:40.733 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:40.734 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:40.734 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:40.734 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:40.734 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:40.735 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:40.735 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:40.735 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:33:40.735 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:40.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:40.736 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:40.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.736 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:40.737 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:33:40.737 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:33:40.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:40.737 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:33:40.737 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:40.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:33:40Z" } 2023-10-17T19:33:40.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:33:40.739 DEBUG:rgw_multi.tests:log_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:40.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:33:40.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.742 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:40.793 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.793 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.840 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:33:40.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.841 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:40.944 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:33:40.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-72', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-72', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:40.945 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-72 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-72 --dest-bucket=hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:41.001 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.001 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.048 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:33:41.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-73', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-73', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.049 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-73 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-73 --dest-bucket=hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:41.105 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.106 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.153 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:33:41.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.155 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:41.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.253 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:33:41.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.254 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:41.304 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.304 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:41.357 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:33:41.357 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 19, 2023-10-17T19:33:41.357 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:33:41.358 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:33:41.358 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:33:41.358 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:33:41.358 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:33:41.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.359 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:33:41.359 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:33:41.359 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:33:41.360 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:33:41.360 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.360 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:33:41.360 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.361 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:33:41.361 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:33:41.361 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.361 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.362 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.362 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:33:41.363 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.363 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:33:41.363 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.363 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:33:41.363 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:33:41.364 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:33:41.364 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:33:41.364 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:33:41.364 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:33:41.365 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:33:41.365 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:33:41.365 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:33:41.365 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:33:41.366 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.367 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:33:41.367 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:33:41.367 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:33:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:33:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:33:41.368 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:33:41.369 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:33:41.369 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:33:41.369 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:33:41.370 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:33:41.370 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:33:41.370 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:33:41.371 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:33:41.371 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.372 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:33:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.373 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:33:41.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:33:41.374 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:33:41.374 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:33:41.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.375 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.376 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:33:41.376 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:33:41.376 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:33:41.376 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:33:41.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.377 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:33:41.377 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:33:41.377 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:33:41.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.378 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:33:41.378 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.378 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.378 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:33:41.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.379 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.379 INFO:teuthology.orchestra.run.smithi049.stdout: "directional": [ 2023-10-17T19:33:41.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.380 INFO:teuthology.orchestra.run.smithi049.stdout: "source_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.380 INFO:teuthology.orchestra.run.smithi049.stdout: "dest_zone": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:33:41.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.381 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:33:41.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.381 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:33:41.382 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.382 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-70", 2023-10-17T19:33:41.382 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.382 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:33:41.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.383 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:33:41.384 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-71", 2023-10-17T19:33:41.384 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.385 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:33:41.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.386 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:33:41.386 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.386 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:33:41.387 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:33:41.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.388 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:33:41.388 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:33:41.389 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:33:41.389 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:33:41.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.391 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-72", 2023-10-17T19:33:41.391 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.391 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-72", 2023-10-17T19:33:41.391 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.392 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:33:41.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.392 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:33:41.393 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-73", 2023-10-17T19:33:41.393 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.393 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:33:41.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.394 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:33:41.394 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.394 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:33:41.395 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:33:41.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.395 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:33:41.396 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:33:41.396 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:33:41.396 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:33:41.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.397 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-73", 2023-10-17T19:33:41.397 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.397 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-73", 2023-10-17T19:33:41.397 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.398 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:33:41.398 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.398 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:33:41.399 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-73", 2023-10-17T19:33:41.399 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.399 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:33:41.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:33:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:33:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:33:41.400 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:33:41.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.401 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:33:41.401 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:33:41.402 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:33:41.402 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:33:41.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.403 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.403 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:33:41.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.403 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.404 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:33:41.404 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.404 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.405 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:33:41.405 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:33:41.406 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:33:41.406 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:33:41.406 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.406 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:33:41.407 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.407 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:33:41.407 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:33:41.407 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:33:41.408 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:33:41.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.408 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:33:41.409 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:33:41.409 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.409 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:33:41.410 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.410 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:33:41.410 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:33:41.411 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:33:41.411 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:33:41.411 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:33:41.411 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:33:41.412 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:33:41.412 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:33:41.412 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:33:41.412 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:33:41.413 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.413 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.414 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:33:41.414 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:33:41.414 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:33:41.415 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:33:41.415 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.415 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:33:41.415 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:33:41.416 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:33:41.416 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:33:41.416 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:33:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:33:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:33:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:33:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:33:41.418 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:33:41.418 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.418 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.419 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.420 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:33:41.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.420 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:33:41.420 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:33:41.421 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:33:41.421 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:33:41.421 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.422 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.422 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:33:41.422 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:33:41.422 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:33:41.423 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:33:41.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.423 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:33:41.423 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:33:41.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.424 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:33:41.424 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:33:41.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.425 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:33:41.425 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:33:41.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.425 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.426 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:33:41.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:33:41.427 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:33:41.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:41.428 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:33:41.428 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:33:41.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.429 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:41.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.429 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:33:41.429 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:33:41.430 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:33:41.430 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:33:41.430 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:33:41.431 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:33:41.431 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:33:41.431 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:33:41.432 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:33:41.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.433 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:33:41.433 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:33:41.434 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:33:41.434 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:33:41.434 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:33:41.435 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:33:41.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.435 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:33:41.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:33:41.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:33:41.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:33:41.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:33:41.437 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:33:41.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.437 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:33:41.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:33:41.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:33:41.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:33:41.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:33:41.439 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:33:41.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.439 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:33:41.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:33:41.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:33:41.442 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:33:41.442 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:33:41.442 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:33:41.443 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:41.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:41.443 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:33:41.443 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:33:41.444 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:41.444 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:41.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:33:41.446 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:33:41.509 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.510 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:33:41.615 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:33:41.615 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:33:41.616 DEBUG:boto:Method: HEAD 2023-10-17T19:33:41.616 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:33:41.617 DEBUG:boto:Data: 2023-10-17T19:33:41.617 DEBUG:boto:Headers: {} 2023-10-17T19:33:41.617 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.618 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.618 DEBUG:boto:Params: {} 2023-10-17T19:33:41.619 DEBUG:boto:Token: None 2023-10-17T19:33:41.619 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-72/ 2023-10-17T19:33:41.620 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c= 2023-10-17T19:33:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c=', 'Content-Length': '0'} 2023-10-17T19:33:41.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a868a8e63cb4523-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.645 DEBUG:boto:path=/hbmuog-72/obj1 2023-10-17T19:33:41.645 DEBUG:boto:auth_path=/hbmuog-72/obj1 2023-10-17T19:33:41.646 DEBUG:boto:Method: PUT 2023-10-17T19:33:41.646 DEBUG:boto:Path: /hbmuog-72/obj1 2023-10-17T19:33:41.647 DEBUG:boto:Data: 2023-10-17T19:33:41.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:33:41.648 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.648 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.648 DEBUG:boto:Params: {} 2023-10-17T19:33:41.649 DEBUG:boto:Token: None 2023-10-17T19:33:41.649 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-72/obj1 2023-10-17T19:33:41.650 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:GFBleF8ayJmh+S6iKG/6cj3KeKg= 2023-10-17T19:33:41.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFBleF8ayJmh+S6iKG/6cj3KeKg='} 2023-10-17T19:33:41.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006011d930674e02a8-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.670 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:33:41.670 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:33:41.671 DEBUG:boto:Method: HEAD 2023-10-17T19:33:41.671 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:33:41.672 DEBUG:boto:Data: 2023-10-17T19:33:41.672 DEBUG:boto:Headers: {} 2023-10-17T19:33:41.673 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.673 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.673 DEBUG:boto:Params: {} 2023-10-17T19:33:41.674 DEBUG:boto:Token: None 2023-10-17T19:33:41.674 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-72/ 2023-10-17T19:33:41.675 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c= 2023-10-17T19:33:41.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c=', 'Content-Length': '0'} 2023-10-17T19:33:41.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c40eb8be3868a5-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.681 DEBUG:boto:path=/hbmuog-72/obj2 2023-10-17T19:33:41.681 DEBUG:boto:auth_path=/hbmuog-72/obj2 2023-10-17T19:33:41.682 DEBUG:boto:Method: PUT 2023-10-17T19:33:41.682 DEBUG:boto:Path: /hbmuog-72/obj2 2023-10-17T19:33:41.683 DEBUG:boto:Data: 2023-10-17T19:33:41.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:33:41.683 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.684 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.684 DEBUG:boto:Params: {} 2023-10-17T19:33:41.685 DEBUG:boto:Token: None 2023-10-17T19:33:41.685 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-72/obj2 2023-10-17T19:33:41.686 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rdM2ZQ1ZQRJK+6PaCON/KnF5v74= 2023-10-17T19:33:41.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rdM2ZQ1ZQRJK+6PaCON/KnF5v74='} 2023-10-17T19:33:41.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000447b56d5e917a70d-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.731 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:33:41.732 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:33:41.732 DEBUG:boto:Method: HEAD 2023-10-17T19:33:41.732 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:33:41.733 DEBUG:boto:Data: 2023-10-17T19:33:41.733 DEBUG:boto:Headers: {} 2023-10-17T19:33:41.733 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.734 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.734 DEBUG:boto:Params: {} 2023-10-17T19:33:41.735 DEBUG:boto:Token: None 2023-10-17T19:33:41.735 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-73/ 2023-10-17T19:33:41.735 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0= 2023-10-17T19:33:41.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0=', 'Content-Length': '0'} 2023-10-17T19:33:41.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000588c87b30211342a-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.745 DEBUG:boto:path=/hbmuog-73/obj3 2023-10-17T19:33:41.745 DEBUG:boto:auth_path=/hbmuog-73/obj3 2023-10-17T19:33:41.746 DEBUG:boto:Method: PUT 2023-10-17T19:33:41.746 DEBUG:boto:Path: /hbmuog-73/obj3 2023-10-17T19:33:41.747 DEBUG:boto:Data: 2023-10-17T19:33:41.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:33:41.748 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.748 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.748 DEBUG:boto:Params: {} 2023-10-17T19:33:41.749 DEBUG:boto:Token: None 2023-10-17T19:33:41.749 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-73/obj3 2023-10-17T19:33:41.750 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:6ErFdoxa/unfXlpBdnhXkgg6i+k= 2023-10-17T19:33:41.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ErFdoxa/unfXlpBdnhXkgg6i+k='} 2023-10-17T19:33:41.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba59ac70c12c7800-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.764 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:33:41.765 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:33:41.765 DEBUG:boto:Method: HEAD 2023-10-17T19:33:41.766 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:33:41.766 DEBUG:boto:Data: 2023-10-17T19:33:41.767 DEBUG:boto:Headers: {} 2023-10-17T19:33:41.767 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.767 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.768 DEBUG:boto:Params: {} 2023-10-17T19:33:41.768 DEBUG:boto:Token: None 2023-10-17T19:33:41.769 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-73/ 2023-10-17T19:33:41.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0= 2023-10-17T19:33:41.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0=', 'Content-Length': '0'} 2023-10-17T19:33:41.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fb0c0fc7bdb5d6b-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.774 DEBUG:boto:path=/hbmuog-73/obj4 2023-10-17T19:33:41.774 DEBUG:boto:auth_path=/hbmuog-73/obj4 2023-10-17T19:33:41.775 DEBUG:boto:Method: PUT 2023-10-17T19:33:41.775 DEBUG:boto:Path: /hbmuog-73/obj4 2023-10-17T19:33:41.776 DEBUG:boto:Data: 2023-10-17T19:33:41.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:33:41.776 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:33:41.777 DEBUG:boto:Port: 8000 2023-10-17T19:33:41.777 DEBUG:boto:Params: {} 2023-10-17T19:33:41.778 DEBUG:boto:Token: None 2023-10-17T19:33:41.778 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:33:41 GMT /hbmuog-73/obj4 2023-10-17T19:33:41.779 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:pugnD5NfWfVGUqiRNX4+bsOxngs= 2023-10-17T19:33:41.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pugnD5NfWfVGUqiRNX4+bsOxngs='} 2023-10-17T19:33:41.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002f931956067d8c4-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:33:41.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:41.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:41.824 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:41.901 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:41.902 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:33:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:33:42.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.004 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:33:42.004 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:33:40.511436Z" 2023-10-17T19:33:42.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.004 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:33:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:33:42.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.006 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:42.006 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:33:42.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.007 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:33:42.008 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:42.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:33:42.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.010 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:42.124+0000 7fb4f1d0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:33:42.126+0000 7fb4f1d0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:33:42.144 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:42.144 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:33:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:42.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.146 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:33:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:33:42.147 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:33:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:42.149 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:33:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:33:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.151 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:33:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:33:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:33:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:33:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.156 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:33:42.156 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:33:42.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.157 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:42.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.157 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:42.157 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:33:42.158 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:33:42.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.158 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:33:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:42.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:33:42Z" } 2023-10-17T19:33:42.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:33:42.160 DEBUG:rgw_multi.tests:log_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:42.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:33:42.161 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:33:42.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.163 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:42.214 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:42.215 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:33:42.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.265 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.266 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:33:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:33:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:33:42.267 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:33:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:33:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:33:42.268 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:33:42.270 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:33:42.270 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:33:42.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:42.272 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:42.325 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:42.326 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:33:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:33:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:33:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:33:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:33:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:42.387 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:42Z" } 2023-10-17T19:33:42.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:33:42.388 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:33:42.388 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:42.389 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:33:47.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:47.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:47.391 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:47.452 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:47.453 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:47.505 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:47.506 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:47.507 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:47.507 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:47.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.507 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:47.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:47.508 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:47.508 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:47.508 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:47.508 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:47.508 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:47.509 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:47.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:47.509 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:47.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:47.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:47.510 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:47.510 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:47.510 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:47.511 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:33:47.511 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:47.511 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:47.511 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:33:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:47.513 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:47.513 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:47.513 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:47.513 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:47.513 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:47.514 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:47.514 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:47.514 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:47.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:47.515 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:47.515 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:47.515 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:47.515 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:33:47.515 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:47.516 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:47.516 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:47.516 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:33:47.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:47.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:47.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:47.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.517 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:47.517 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:47.517 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:47.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:47.518 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:47Z" 2023-10-17T19:33:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:47.519 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:47Z" } 2023-10-17T19:33:47.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:33:47.520 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:33:47.520 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:47.521 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:33:52.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:52.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:52.523 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:52.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:52.584 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:52.634 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:52.635 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:52.635 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:52.635 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:52.635 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:52.636 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:52.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.636 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:52.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:52.636 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:52.637 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:52.637 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:52.637 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:52.637 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:52.638 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:52.638 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:52.638 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:52.638 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:52.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:52.639 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:52.639 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:52.639 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:52.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:33:52.640 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:52.640 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:52.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:52.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:33:52.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:52.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:52.641 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:52.641 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:52.642 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:52.642 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:52.642 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:52.642 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:52.642 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:52.643 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:52.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:52.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:52.643 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:52.644 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:52.644 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:52.644 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:33:52.644 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:52.644 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:52.645 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:52.645 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:33:52.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:52.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:52.645 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:52.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.646 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:52.646 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:52.646 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:52.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:52.647 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:52Z" 2023-10-17T19:33:52.647 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:52.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:52Z" } 2023-10-17T19:33:52.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:33:52.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:33:52.650 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:52.650 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:33:57.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:57.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:33:57.652 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:33:57.714 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:33:57.714 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:33:57.764 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:33:57.764 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:33:57.764 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:33:57.765 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:33:57.765 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:33:57.765 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:33:57.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.766 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:33:57.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:57.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:33:57.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:57.766 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:57.767 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:33:57.767 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:57.767 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:57.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:33:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:33:57.769 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:57.769 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:57.769 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:33:57.769 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:57.770 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:33:57.771 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:57.771 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:33:57.771 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:57.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:57.772 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:33:57.772 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:57.772 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:57.772 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:33:57.773 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:57.773 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:33:57.773 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:57.773 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:33:57.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:57.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:33:57.774 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:33:57.775 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:33:57.775 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:33:57.775 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:33:57.775 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:33:57.775 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:33:57.776 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:33:57.776 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:33:57.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:57.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:33:57.776 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:33:57.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.777 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:33:57.777 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:33:57.777 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:33:57.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:33:57.778 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:33:57Z" 2023-10-17T19:33:57.778 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:33:57.779 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:33:57Z" } 2023-10-17T19:33:57.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:33:57.780 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:33:57.781 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:33:57.781 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:02.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:02.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:02.783 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:02.845 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:02.846 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:02.898 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:02.898 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:02.898 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:02.898 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:02.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.899 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:02.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:02.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:02.900 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:02.900 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:02.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:02.900 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:02.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:02.901 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:02.901 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:02.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:02.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:02.902 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:02.902 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:02.902 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:02.902 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:02.902 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:02.903 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:02.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:02.903 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:02.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:02.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:02.904 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:02.904 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:02.904 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:02.905 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:02.905 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:02.905 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:02.905 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:02.905 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:02.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:02.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:02.906 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:02.907 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:02.907 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:02.907 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:02.907 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:02.907 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:02.908 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:02.908 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:02.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:02.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:02.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:02.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.909 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:02.909 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:02.909 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:02.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:02.910 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:02Z" 2023-10-17T19:34:02.910 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:02.911 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:02Z" } 2023-10-17T19:34:02.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:02.912 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:02.913 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:02.913 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:07.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:07.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:07.916 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:07.977 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:07.978 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:08.029 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:08.030 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:08.030 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:08.030 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:08.031 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:08.031 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:08.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.031 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:08.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:08.032 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:08.032 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:08.032 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:08.032 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:08.033 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:08.033 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:08.033 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:08.033 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:08.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:08.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:08.034 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:08.034 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:08.034 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:08.035 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:08.035 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:08.035 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:08.035 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:08.036 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:08.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:08.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:08.036 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:08.037 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:08.037 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:08.037 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:08.037 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:08.037 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:08.038 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:08.038 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:08.038 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:08.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:08.039 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:08.039 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:08.039 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:08.039 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:08.040 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:08.040 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:08.040 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:08.040 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:08.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:08.041 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:08.042 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:08.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:08.042 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:08Z" 2023-10-17T19:34:08.042 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:08.043 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:08Z" } 2023-10-17T19:34:08.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:08.044 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:08.045 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:08.045 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:13.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:13.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:13.047 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:13.106 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:13.107 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:13.156 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:13.157 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:13.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:13.159 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:13.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:13.161 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:13.162 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:13.163 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:13.164 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:13.167 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:13.168 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:13.169 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:13Z" 2023-10-17T19:34:13.169 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:13.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:13Z" } 2023-10-17T19:34:13.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:13.172 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:13.172 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:13.173 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:18.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:18.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:18.176 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:18.289 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:18.290 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:18.290 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:18.290 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:18.290 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:18.290 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:18.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.291 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:18.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:18.291 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:18.292 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:18.292 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:18.292 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:18.292 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:18.292 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:18.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:18.293 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:18.293 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:18.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:18.294 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:18.294 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:18.294 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:18.294 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:18.294 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:18.295 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:18.295 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:18.295 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:18.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:18.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:18.296 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:18.296 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:18.296 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:18.297 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:18.297 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:18.297 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:18.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:18.297 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:18.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:18.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:18.298 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:18.298 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:18.299 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:18.299 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:18.299 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:18.299 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:18.299 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:18.300 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:18.300 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:18.300 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:18.300 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:18.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.301 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:18.301 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:18.301 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:18.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:18.301 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:18Z" 2023-10-17T19:34:18.302 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:18.303 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:18Z" } 2023-10-17T19:34:18.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:18.304 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:18.304 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:18.305 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:23.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:23.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:23.308 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:23.370 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:23.371 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:23.422 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:23.422 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:23.422 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:23.423 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:23.423 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:23.423 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:23.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.423 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:23.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:23.424 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:23.424 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:23.424 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:23.424 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:23.425 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:23.425 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:23.425 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:23.425 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:23.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:23.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:23.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:23.427 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:23.427 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:23.427 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:23.427 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:23.429 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:23.429 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:23.429 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:23.429 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:23.430 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:23.430 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:23.430 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:23.431 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:23.431 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:23.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:23.432 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:23.432 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:23.432 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:23.432 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:23.432 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:23.433 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:23.433 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:23.433 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:23.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:23.434 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:23.434 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:23.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.434 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:23.434 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:23.435 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:23.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:23.435 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:23Z" 2023-10-17T19:34:23.435 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:23.436 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:23Z" } 2023-10-17T19:34:23.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:23.438 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:23.438 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:23.438 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:28.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:28.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:28.441 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:28.503 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:28.503 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:28.555 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:28.556 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:28.556 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:28.556 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:28.557 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:28.557 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:28.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.557 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:28.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:28.558 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:28.558 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:28.558 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:28.558 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:28.559 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:28.559 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:28.559 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:28.559 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:28.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:28.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:28.560 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:28.561 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:28.561 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:28.561 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:28.561 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:28.561 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:28.562 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:28.562 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:28.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:28.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:28.563 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:28.563 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:28.563 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:28.563 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:28.564 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:28.564 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:28.564 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:28.564 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:28.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:28.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:28.565 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:28.565 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:28.566 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:28.566 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:28.566 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:28.566 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:28.566 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:28.567 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:28.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:28.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:28.567 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:28.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.568 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:28.568 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:28.568 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:28.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:28.568 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:28Z" 2023-10-17T19:34:28.569 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:28.570 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:28Z" } 2023-10-17T19:34:28.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:28.571 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:28.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:28.571 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:33.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:33.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:33.574 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:33.637 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:33.637 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:33.687 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:33.696 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:33.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:33.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:33.697 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:33.697 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:33.697 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:33.698 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:33.698 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:33.698 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:33.699 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:33.699 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:33.699 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:33.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:33.700 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:33.700 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:33.700 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:33.700 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:33.701 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:33.701 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:33.701 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:33.701 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:33.701 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:33.702 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:33.702 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:33.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.702 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:33.703 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:33.703 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:33.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:33.703 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:33Z" 2023-10-17T19:34:33.703 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:33.705 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:33Z" } 2023-10-17T19:34:33.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:33.706 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:33.706 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:33.707 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:38.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:38.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:38.710 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:38.772 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:38.772 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:38.823 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:38.823 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:38.823 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:38.824 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:38.824 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:38.824 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:38.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.825 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:38.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:38.825 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:38.825 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:38.825 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:38.826 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:38.826 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:38.826 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:38.826 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:38.827 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:38.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:38.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:38.827 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:38.828 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:38.828 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:38.828 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:38.828 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:38.828 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:38.829 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:38.829 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:38.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:38.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:38.830 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:38.830 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:38.830 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:38.830 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:38.831 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:38.831 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:38.831 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:38.831 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:38.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:38.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:38.832 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:38.832 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:38.832 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:38.833 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:38.833 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:38.833 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:38.833 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:38.834 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:38.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:38.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:38.834 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:38.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.835 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:38.835 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:38.835 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:38.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:38.835 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:38Z" 2023-10-17T19:34:38.836 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:38.837 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:38Z" } 2023-10-17T19:34:38.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:38.839 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:38.839 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:38.839 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:43.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:43.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:43.843 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:43.903 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:43.904 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:43.955 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:43Z" 2023-10-17T19:34:43.969 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:43.970 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:43Z" } 2023-10-17T19:34:43.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:43.971 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:43.971 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:43.972 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:48.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:48.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:48.974 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:49.035 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:49.035 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:49.086 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:49Z" 2023-10-17T19:34:49.101 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:49.103 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:49Z" } 2023-10-17T19:34:49.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:49.104 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:49.104 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:49.105 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:54.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:54.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:54.108 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:54.169 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:54.169 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:54.221 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:54.221 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:54.222 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:54.222 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:54.222 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:54.223 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:54.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.223 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:54.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:54.224 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:54.224 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:54.224 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:54.224 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:54.225 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:54.225 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:54.225 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:54.225 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:54.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:54.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:54.226 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:54.226 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:54.227 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:54.227 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:54.227 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:54.227 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:54.227 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:54.228 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:54.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:54.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:54.230 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:54.230 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:54.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:54.230 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:54.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:54.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:54.231 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:54.231 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:54.231 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:54.232 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:54.232 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:54.232 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:54.232 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:54.232 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:54.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:54.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:54.233 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:54.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.234 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:54.234 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:54.234 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:54.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:54.234 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:54Z" 2023-10-17T19:34:54.235 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:54.236 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:54Z" } 2023-10-17T19:34:54.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:54.237 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:54.237 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:54.239 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:34:59.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:59.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:34:59.242 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:34:59.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:34:59.303 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:34:59.360 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:34:59.360 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:34:59.360 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:34:59.361 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:34:59.361 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:34:59.361 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:34:59.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.362 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:34:59.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:59.362 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:34:59.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:59.362 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:59.363 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:34:59.363 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:59.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:59.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:59.364 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:34:59.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:59.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:59.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:34:59.365 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:59.365 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:59.365 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:34:59.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:59.366 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:34:59.366 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:59.366 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:34:59.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:59.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:34:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:59.367 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:59.368 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:34:59.368 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:59.368 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:34:59.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:59.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:34:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:34:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:34:59.369 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:34:59.370 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:34:59.370 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:34:59.370 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:34:59.370 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:34:59.371 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:34:59.371 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:34:59.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:59.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:34:59.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:34:59.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.372 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:34:59.372 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:34:59.372 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:34:59.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:34:59.373 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:34:59Z" 2023-10-17T19:34:59.373 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:34:59.374 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:34:59Z" } 2023-10-17T19:34:59.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:34:59.375 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:34:59.375 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:34:59.376 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:04.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:04.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:04.379 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:04.442 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:04.442 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:04.495 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:04.495 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:04.495 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:04.495 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:04.496 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:04.496 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:04.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.496 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:04.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:04.497 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:04.497 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:04.497 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:04.497 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:04.497 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:04.498 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:04.498 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:04.498 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:04.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:04.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:04.499 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:04.499 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:04.499 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:04.500 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:04.500 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:04.500 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:04.500 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:04.500 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:04.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:04.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:04.501 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:04.501 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:04.502 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:04.502 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:04.502 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:04.502 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:04.503 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:04.504 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:04.504 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:04.504 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:04.504 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:04.504 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:04.505 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:04.505 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:04.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:04.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:04.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:04.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.506 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:04.506 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:04.506 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:04.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:04.507 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:04Z" 2023-10-17T19:35:04.507 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:04.508 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:04Z" } 2023-10-17T19:35:04.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:04.509 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:04.509 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:04.509 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:09.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:09.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:09.512 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:09.574 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:09.574 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:09.625 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:09Z" 2023-10-17T19:35:09.638 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:09.639 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:09Z" } 2023-10-17T19:35:09.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:09.640 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:09.640 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:09.641 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:14.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:14.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:14.643 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:14.704 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:14.705 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:14.757 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:14.757 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:14.757 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:14.758 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:14.758 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:14.758 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:14.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.759 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:14.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:14.759 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:14.759 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:14.759 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:14.760 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:14.760 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:14.760 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:14.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:14.761 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:14.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:14.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:14.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:14.763 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:14.763 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:14.763 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:14.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:14.764 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:14.764 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:14.764 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:14.765 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:14.765 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:14.765 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:14.765 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:14.765 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:14.766 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:14.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:14.766 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:14.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:14.767 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:14.767 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:14.767 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:14.767 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:14.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:14.768 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:14.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:14.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:14.768 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:14.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.769 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:14.769 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:14.769 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:14.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:14.770 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:14Z" 2023-10-17T19:35:14.770 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:14.771 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:14Z" } 2023-10-17T19:35:14.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:14.772 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:14.772 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:14.773 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:19.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:19.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:19.775 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:19.838 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:19.839 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:19.889 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:19.890 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:19.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:19.892 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:19.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:19.894 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:19.895 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:19.896 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:19.897 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:19.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:19.901 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:19.902 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:19Z" 2023-10-17T19:35:19.902 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:19.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:19Z" } 2023-10-17T19:35:19.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:19.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:19.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:19.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:24.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:24.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:24.907 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:24.967 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:24.968 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:25.018 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:25Z" 2023-10-17T19:35:25.031 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:25.032 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:25Z" } 2023-10-17T19:35:25.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:25.033 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:25.033 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:25.034 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:30.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:30.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:30.036 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:30.097 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:30.097 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:30.149 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:30.149 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:30.149 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:30.149 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:30.150 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:30.150 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:30.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.150 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:30.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:30.151 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:30.151 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:30.151 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:30.151 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:30.152 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:30.152 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:30.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:30.152 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:30.153 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:30.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:30.153 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:30.153 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:30.154 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:30.154 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:30.154 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:30.154 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:30.155 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:30.155 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:30.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:30.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:30.156 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:30.156 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:30.156 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:30.156 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:30.156 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:30.157 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:30.157 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:30.157 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:30.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:30.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:30.158 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:30.158 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:30.159 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:30.159 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:30.159 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:30.159 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:30.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:30.160 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:30.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:30.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:30.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:30.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.161 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:30.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:30.161 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:30.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:30.162 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:30Z" 2023-10-17T19:35:30.162 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:30.163 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:30Z" } 2023-10-17T19:35:30.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:30.164 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:30.165 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:30.165 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:35.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:35.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:35.168 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:35.229 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:35.229 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:35.280 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:35.283 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:35.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:35.285 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:35.286 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:35.287 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:35.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:35.289 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:35.290 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:35.292 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:35.293 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:35Z" 2023-10-17T19:35:35.293 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:35.294 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:35Z" } 2023-10-17T19:35:35.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:35.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:35.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:35.296 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:40.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:40.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:40.298 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:40.357 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:40.357 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:40.409 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:40.409 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:40.409 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:40.410 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:40.410 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:40.410 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:40.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.410 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:40.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:40.411 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:40.411 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:40.411 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:40.412 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:40.412 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:40.412 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:40.412 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:40.412 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:40.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:40.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:40.413 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:40.413 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:40.414 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:40.414 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:35:40.414 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:40.414 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:40.415 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:40.415 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:35:40.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:40.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:40.416 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:40.416 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:40.416 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:40.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:35:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:35:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:40.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:40.418 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:40.418 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:40.418 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:40.419 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:35:40.419 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:40.419 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:40.419 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:40.419 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:35:40.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:40.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:40.420 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:40.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.421 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:40.421 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:40.421 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:40.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:40.421 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:40Z" 2023-10-17T19:35:40.422 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:40.423 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040982", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:31:36.272525Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028515", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:18:37.300757Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046856", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:31:36.237304Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:40Z" } 2023-10-17T19:35:40.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:35:40.424 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:40.424 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:35:40.424 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:35:45.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:45.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:45.427 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:45.489 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:45.489 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:45.540 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:45.540 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:45.540 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:45.541 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:45.541 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:45.541 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:35:45.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.542 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:45.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:45.542 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:45.542 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:45.542 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:45.543 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:35:45.543 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:45.543 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:45.543 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:45.544 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:35:45.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:45.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:45.545 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:45.545 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:45.545 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:45.545 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:35:45.545 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:45.546 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:45.546 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:45.547 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:35:45.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:45.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:45.548 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:45.548 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:45.549 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:45.549 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:35:45.549 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:45.549 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:35:45.549 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:45.550 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:35:45.550 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:45.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:45.551 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:45.551 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:45.551 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:35:45.551 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:35:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:35:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:35:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:45.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:45.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:45.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.553 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:45.553 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:35:45.554 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:35:45.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:45.554 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:45Z" 2023-10-17T19:35:45.554 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:45.555 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041122", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:33:41.702225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028656", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:33:41.789887Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047136", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:33:41.762296Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:35:45Z" } 2023-10-17T19:35:45.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:35:45.556 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:45.557 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:35:45.557 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:35:45.558 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:35:45.558 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:35:45.559 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.559 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:35:45.559 DEBUG:boto:Data: 2023-10-17T19:35:45.560 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.560 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.561 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.561 DEBUG:boto:Params: {} 2023-10-17T19:35:45.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:35:45.562 DEBUG:boto:Token: None 2023-10-17T19:35:45.563 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-72/ 2023-10-17T19:35:45.563 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ip6nsxiiJDLCGbuUHA5SJ0ZzK2g= 2023-10-17T19:35:45.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ip6nsxiiJDLCGbuUHA5SJ0ZzK2g=', 'Content-Length': '0'} 2023-10-17T19:35:45.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074d14d95df35ef6d-00652ee211-2584-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.571 DEBUG:boto:path=/hbmuog-72/obj1 2023-10-17T19:35:45.571 DEBUG:boto:auth_path=/hbmuog-72/obj1 2023-10-17T19:35:45.572 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.572 DEBUG:boto:Path: /hbmuog-72/obj1 2023-10-17T19:35:45.573 DEBUG:boto:Data: 2023-10-17T19:35:45.573 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.573 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.574 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.574 DEBUG:boto:Params: {} 2023-10-17T19:35:45.575 DEBUG:boto:Token: None 2023-10-17T19:35:45.575 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-72/obj1 2023-10-17T19:35:45.576 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jIjVm0hSvIehTVnXQIO/Urc9R/w= 2023-10-17T19:35:45.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jIjVm0hSvIehTVnXQIO/Urc9R/w=', 'Content-Length': '0'} 2023-10-17T19:35:45.578 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001e4099544d22f09b-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.578 DEBUG:boto:path=/hbmuog-72/obj2 2023-10-17T19:35:45.579 DEBUG:boto:auth_path=/hbmuog-72/obj2 2023-10-17T19:35:45.579 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.580 DEBUG:boto:Path: /hbmuog-72/obj2 2023-10-17T19:35:45.580 DEBUG:boto:Data: 2023-10-17T19:35:45.581 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.581 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.581 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.582 DEBUG:boto:Params: {} 2023-10-17T19:35:45.582 DEBUG:boto:Token: None 2023-10-17T19:35:45.583 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-72/obj2 2023-10-17T19:35:45.583 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:tTcgFpxliZWHOWCvHCbj4Xkl2vY= 2023-10-17T19:35:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTcgFpxliZWHOWCvHCbj4Xkl2vY=', 'Content-Length': '0'} 2023-10-17T19:35:45.585 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000db4d77fd08ce9791-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.586 DEBUG:boto:path=/hbmuog-72/obj3 2023-10-17T19:35:45.586 DEBUG:boto:auth_path=/hbmuog-72/obj3 2023-10-17T19:35:45.587 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.587 DEBUG:boto:Path: /hbmuog-72/obj3 2023-10-17T19:35:45.587 DEBUG:boto:Data: 2023-10-17T19:35:45.588 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.588 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.589 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.589 DEBUG:boto:Params: {} 2023-10-17T19:35:45.590 DEBUG:boto:Token: None 2023-10-17T19:35:45.590 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-72/obj3 2023-10-17T19:35:45.590 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:58NyIyKPQj8bC3TEwGXEdS86nhQ= 2023-10-17T19:35:45.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:58NyIyKPQj8bC3TEwGXEdS86nhQ=', 'Content-Length': '0'} 2023-10-17T19:35:45.592 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000089d2d711bb027878-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.593 DEBUG:boto:path=/hbmuog-72/obj4 2023-10-17T19:35:45.593 DEBUG:boto:auth_path=/hbmuog-72/obj4 2023-10-17T19:35:45.594 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.594 DEBUG:boto:Path: /hbmuog-72/obj4 2023-10-17T19:35:45.595 DEBUG:boto:Data: 2023-10-17T19:35:45.595 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.595 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.596 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.596 DEBUG:boto:Params: {} 2023-10-17T19:35:45.597 DEBUG:boto:Token: None 2023-10-17T19:35:45.597 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-72/obj4 2023-10-17T19:35:45.598 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:bBSGuPf8+IczGZyg83BJpnN7RmY= 2023-10-17T19:35:45.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bBSGuPf8+IczGZyg83BJpnN7RmY=', 'Content-Length': '0'} 2023-10-17T19:35:45.600 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006e6fa38c0cf7a3fd-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.600 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:35:45.601 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:35:45.601 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.602 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:35:45.602 DEBUG:boto:Data: 2023-10-17T19:35:45.602 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.603 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.603 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.604 DEBUG:boto:Params: {} 2023-10-17T19:35:45.604 DEBUG:boto:Token: None 2023-10-17T19:35:45.604 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/ 2023-10-17T19:35:45.605 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Vl5/cdJYoKxC/bABPYFwYzPC7wA= 2023-10-17T19:35:45.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vl5/cdJYoKxC/bABPYFwYzPC7wA=', 'Content-Length': '0'} 2023-10-17T19:35:45.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ed1bdb17cb36d8a-00652ee211-2584-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.608 DEBUG:boto:path=/hbmuog-73/obj1 2023-10-17T19:35:45.609 DEBUG:boto:auth_path=/hbmuog-73/obj1 2023-10-17T19:35:45.609 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.610 DEBUG:boto:Path: /hbmuog-73/obj1 2023-10-17T19:35:45.610 DEBUG:boto:Data: 2023-10-17T19:35:45.610 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.611 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.612 DEBUG:boto:Params: {} 2023-10-17T19:35:45.612 DEBUG:boto:Token: None 2023-10-17T19:35:45.613 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj1 2023-10-17T19:35:45.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+yRyzan9v08UXwnRaAcNzOs2BLU= 2023-10-17T19:35:45.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+yRyzan9v08UXwnRaAcNzOs2BLU=', 'Content-Length': '0'} 2023-10-17T19:35:45.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d728c3808dd6b1d4-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.616 DEBUG:boto:path=/hbmuog-73/obj1 2023-10-17T19:35:45.617 DEBUG:boto:auth_path=/hbmuog-73/obj1 2023-10-17T19:35:45.617 DEBUG:boto:Method: GET 2023-10-17T19:35:45.617 DEBUG:boto:Path: /hbmuog-73/obj1 2023-10-17T19:35:45.618 DEBUG:boto:Data: 2023-10-17T19:35:45.618 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.619 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.619 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.620 DEBUG:boto:Params: {} 2023-10-17T19:35:45.620 DEBUG:boto:Token: None 2023-10-17T19:35:45.620 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj1 2023-10-17T19:35:45.621 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:w5FCAF4HmaijGJ+ile4fnLdFbVk= 2023-10-17T19:35:45.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w5FCAF4HmaijGJ+ile4fnLdFbVk=', 'Content-Length': '0'} 2023-10-17T19:35:45.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006427ceaa52446a1d-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.624 DEBUG:boto:path=/hbmuog-73/obj2 2023-10-17T19:35:45.624 DEBUG:boto:auth_path=/hbmuog-73/obj2 2023-10-17T19:35:45.625 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.625 DEBUG:boto:Path: /hbmuog-73/obj2 2023-10-17T19:35:45.626 DEBUG:boto:Data: 2023-10-17T19:35:45.626 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.626 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.627 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.627 DEBUG:boto:Params: {} 2023-10-17T19:35:45.628 DEBUG:boto:Token: None 2023-10-17T19:35:45.628 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj2 2023-10-17T19:35:45.629 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:dt3SS4h+Uns9Lfoey4D7kNl2jsM= 2023-10-17T19:35:45.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dt3SS4h+Uns9Lfoey4D7kNl2jsM=', 'Content-Length': '0'} 2023-10-17T19:35:45.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b66ae5256e2b35a4-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.631 DEBUG:boto:path=/hbmuog-73/obj2 2023-10-17T19:35:45.632 DEBUG:boto:auth_path=/hbmuog-73/obj2 2023-10-17T19:35:45.632 DEBUG:boto:Method: GET 2023-10-17T19:35:45.633 DEBUG:boto:Path: /hbmuog-73/obj2 2023-10-17T19:35:45.633 DEBUG:boto:Data: 2023-10-17T19:35:45.633 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.634 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.634 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.635 DEBUG:boto:Params: {} 2023-10-17T19:35:45.635 DEBUG:boto:Token: None 2023-10-17T19:35:45.636 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj2 2023-10-17T19:35:45.636 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:x+I+NmnXE+W+iNLZ+M+S+NtpWdk= 2023-10-17T19:35:45.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x+I+NmnXE+W+iNLZ+M+S+NtpWdk=', 'Content-Length': '0'} 2023-10-17T19:35:45.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e24da96fa82ac2fe-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.639 DEBUG:boto:path=/hbmuog-73/obj3 2023-10-17T19:35:45.640 DEBUG:boto:auth_path=/hbmuog-73/obj3 2023-10-17T19:35:45.640 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.641 DEBUG:boto:Path: /hbmuog-73/obj3 2023-10-17T19:35:45.641 DEBUG:boto:Data: 2023-10-17T19:35:45.641 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.642 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.643 DEBUG:boto:Params: {} 2023-10-17T19:35:45.643 DEBUG:boto:Token: None 2023-10-17T19:35:45.643 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj3 2023-10-17T19:35:45.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XxrAEprI9GxuidcMkGsdQSycNXg= 2023-10-17T19:35:45.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XxrAEprI9GxuidcMkGsdQSycNXg=', 'Content-Length': '0'} 2023-10-17T19:35:45.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090786a7fe150a8bb-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.647 DEBUG:boto:path=/hbmuog-73/obj3 2023-10-17T19:35:45.647 DEBUG:boto:auth_path=/hbmuog-73/obj3 2023-10-17T19:35:45.648 DEBUG:boto:Method: GET 2023-10-17T19:35:45.648 DEBUG:boto:Path: /hbmuog-73/obj3 2023-10-17T19:35:45.649 DEBUG:boto:Data: 2023-10-17T19:35:45.649 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.649 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.650 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.650 DEBUG:boto:Params: {} 2023-10-17T19:35:45.651 DEBUG:boto:Token: None 2023-10-17T19:35:45.651 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj3 2023-10-17T19:35:45.652 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:id591CZLAv/wGK8iaRIW7fizvR4= 2023-10-17T19:35:45.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:id591CZLAv/wGK8iaRIW7fizvR4=', 'Content-Length': '0'} 2023-10-17T19:35:45.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041b5197caa655d7b-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.694 DEBUG:boto:path=/hbmuog-73/obj4 2023-10-17T19:35:45.695 DEBUG:boto:auth_path=/hbmuog-73/obj4 2023-10-17T19:35:45.695 DEBUG:boto:Method: HEAD 2023-10-17T19:35:45.696 DEBUG:boto:Path: /hbmuog-73/obj4 2023-10-17T19:35:45.696 DEBUG:boto:Data: 2023-10-17T19:35:45.696 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.697 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.697 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.698 DEBUG:boto:Params: {} 2023-10-17T19:35:45.698 DEBUG:boto:Token: None 2023-10-17T19:35:45.699 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj4 2023-10-17T19:35:45.699 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zqag8nKSfFKKZ0sJRoDp1nQG66o= 2023-10-17T19:35:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zqag8nKSfFKKZ0sJRoDp1nQG66o=', 'Content-Length': '0'} 2023-10-17T19:35:45.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de004abbd257428c-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.702 DEBUG:boto:path=/hbmuog-73/obj4 2023-10-17T19:35:45.703 DEBUG:boto:auth_path=/hbmuog-73/obj4 2023-10-17T19:35:45.703 DEBUG:boto:Method: GET 2023-10-17T19:35:45.704 DEBUG:boto:Path: /hbmuog-73/obj4 2023-10-17T19:35:45.704 DEBUG:boto:Data: 2023-10-17T19:35:45.704 DEBUG:boto:Headers: {} 2023-10-17T19:35:45.705 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:35:45.705 DEBUG:boto:Port: 8001 2023-10-17T19:35:45.705 DEBUG:boto:Params: {} 2023-10-17T19:35:45.706 DEBUG:boto:Token: None 2023-10-17T19:35:45.706 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:35:45 GMT /hbmuog-73/obj4 2023-10-17T19:35:45.707 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Ips92TLOxWlT8La1nITTs7R7Zi8= 2023-10-17T19:35:45.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ips92TLOxWlT8La1nITTs7R7Zi8=', 'Content-Length': '0'} 2023-10-17T19:35:45.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047b7bf52420593dc-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:35:45.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.711 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:45.814 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:35:45.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.815 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:45.867 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:45.867 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:45.914 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:35:45.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:45.916 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:45.971 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:45.972 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.018 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:35:46.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.020 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.075 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.075 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.123 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:35:46.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.125 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.180 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.180 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.228 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:35:46.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.229 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.282 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.282 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.330 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:35:46.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.331 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.382 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.382 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.435 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:46.436 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:35:46.436 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 20, 2023-10-17T19:35:46.436 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:35:46.437 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:35:46.437 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:35:46.437 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:35:46.437 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:35:46.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.438 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:35:46.438 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:35:46.438 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:35:46.438 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:35:46.439 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.439 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:35:46.439 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.439 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:35:46.439 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:35:46.440 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.440 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:35:46.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.440 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.440 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:35:46.441 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.441 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:35:46.441 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.441 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:35:46.441 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:35:46.442 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:35:46.442 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:35:46.442 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:35:46.442 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:35:46.442 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:35:46.443 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:35:46.443 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:35:46.443 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:35:46.443 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.444 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.444 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:35:46.444 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:35:46.445 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.445 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:35:46.445 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.445 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:35:46.446 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:35:46.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:35:46.446 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:35:46.446 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:35:46.446 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:35:46.447 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:35:46.447 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:35:46.447 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:35:46.447 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:35:46.447 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.448 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.448 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.448 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.448 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:35:46.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.449 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:35:46.449 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:35:46.449 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:35:46.449 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:35:46.450 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.450 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.450 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:35:46.451 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:35:46.451 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:35:46.451 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:35:46.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.451 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:35:46.452 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:35:46.452 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:35:46.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.452 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:35:46.453 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:35:46.453 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.453 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:35:46.453 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.454 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.454 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:35:46.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.455 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:35:46.455 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:35:46.455 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-70", 2023-10-17T19:35:46.455 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:35:46.455 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.456 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:35:46.456 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.456 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:35:46.457 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-71", 2023-10-17T19:35:46.457 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:35:46.457 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.457 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:35:46.457 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.458 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:35:46.458 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:35:46.458 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:35:46.459 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:35:46.460 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:35:46.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.460 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.460 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "allowed" 2023-10-17T19:35:46.461 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.461 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.461 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:35:46.461 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.462 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.462 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:35:46.462 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:35:46.463 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:35:46.463 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:35:46.463 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.463 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:35:46.463 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.464 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:35:46.464 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:35:46.464 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:35:46.464 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:35:46.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.465 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:35:46.465 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:35:46.465 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.465 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:35:46.465 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.466 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:35:46.466 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:35:46.466 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:35:46.466 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:35:46.467 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:35:46.467 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:35:46.467 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:35:46.467 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:35:46.467 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:35:46.468 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:35:46.468 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.468 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.469 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:35:46.469 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:35:46.469 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:35:46.469 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:35:46.470 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.470 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:35:46.470 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:35:46.470 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:35:46.470 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:35:46.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:35:46.471 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:35:46.471 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:35:46.471 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:35:46.471 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:35:46.472 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:35:46.472 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.472 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.472 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.473 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.473 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:35:46.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.473 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:35:46.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:35:46.474 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:35:46.474 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:35:46.474 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:35:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:35:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:35:46.475 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:35:46.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.476 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:35:46.476 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:35:46.476 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:35:46.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.478 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.479 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:35:46.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.480 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:35:46.480 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:35:46.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:46.480 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:35:46.481 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:35:46.481 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.481 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:46.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.481 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:35:46.482 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:35:46.482 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:35:46.482 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:35:46.482 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:35:46.482 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:35:46.483 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:35:46.483 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:35:46.483 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:35:46.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.484 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:35:46.484 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:35:46.484 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:35:46.484 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:35:46.484 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:35:46.485 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:35:46.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.485 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:35:46.485 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:35:46.485 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:35:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:35:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:35:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:35:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:35:46.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:35:46.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:35:46.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:35:46.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:35:46.487 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:35:46.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.488 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:35:46.488 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:35:46.488 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:35:46.488 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:35:46.489 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:35:46.489 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:35:46.489 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:46.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:46.489 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:35:46.490 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:35:46.490 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:46.490 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:46.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.492 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.549 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.549 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.636 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:35:46.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.637 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.832 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:35:46.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.833 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:46.914 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:46.914 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:46.993 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:35:46.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:46.995 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-72 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-72 --source-bucket=hbmuog-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:47.071 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.072 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.146 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:35:47.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.148 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-73 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-72 --source-bucket=hbmuog-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.264 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:35:47.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.265 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:47.320 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.320 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.372 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:35:47.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:35:47.374 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:35:47.421 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.421 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.467 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:35:47.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:47.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:47.468 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:47.517 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.517 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.549 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:35:47.563 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:35:47.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.564 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:35:47.564 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:35:47.356986Z" 2023-10-17T19:35:47.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.565 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:35:47.565 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:35:47.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.566 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:35:47.566 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:35:47.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.567 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:35:47.567 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:35:47.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.568 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:35:47.569 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:47.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:35:47.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:47.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:47.571 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:47.653 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:47.650+0000 7f946d6f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:47.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:47.651+0000 7f946d6f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:47.665 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:47.666 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:35:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.667 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:47.668 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:35:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:35:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:47.669 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:35:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.670 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:35:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:47.671 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:47.672 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:35:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:47.674 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:35:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:47.675 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:35:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:47.677 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:47.677 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:47.677 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:47.677 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:35:47.678 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:47.678 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:47.678 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:47.678 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:35:47.678 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:47.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:47.679 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:47.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.679 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:47.680 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:35:47.680 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:35:47.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:47.681 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:47Z" 2023-10-17T19:35:47.681 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:47.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:35:47Z" } 2023-10-17T19:35:47.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:35:47.683 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:47.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:47.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:35:52.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:52.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:52.687 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:52.746 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:52.746 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:52.780 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:52.778+0000 7f042d799780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:52.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:52.779+0000 7f042d799780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:52.795 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:35:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:35:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:35:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:35:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:35:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:35:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:35:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:35:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:35:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:35:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:35:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:35:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:52Z" 2023-10-17T19:35:52.811 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:52.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:35:52Z" } 2023-10-17T19:35:52.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:35:52.813 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:52.814 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:52.814 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:35:57.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:57.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:35:57.817 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:35:57.876 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:35:57.876 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:35:57.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:57.912+0000 7f68ce7cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:57.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:35:57.913+0000 7f68ce7cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:35:57.928 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:35:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:35:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:35:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:35:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:35:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:35:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:35:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:35:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:35:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:35:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:35:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:35:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:35:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:35:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:35:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:35:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:35:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:35:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:35:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:35:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:35:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:35:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:35:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:35:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:35:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:35:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:35:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:35:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:35:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:35:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:35:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:35:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:35:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:35:57Z" 2023-10-17T19:35:57.943 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:35:57.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:35:57Z" } 2023-10-17T19:35:57.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:35:57.945 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:57.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:35:57.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:02.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:02.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:02.949 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:03.043+0000 7fad12982780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:03.044+0000 7fad12982780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:03.059 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:03.060 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:03.060 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:03.060 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:03.060 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:03.061 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:03.061 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:03.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.061 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:03.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:03.062 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:03.062 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:03.062 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:03.062 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:03.062 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:03.063 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:03.063 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:03.063 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:03.063 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:03.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:03.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:03.064 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:03.064 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:03.064 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:03.065 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:03.065 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:03.065 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:03.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:03.066 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:03.066 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:03.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:03.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:03.067 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:03.067 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:03.067 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:03.068 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:03.068 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:03.068 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:03.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:03.068 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:03.069 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:03.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:03.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:03.070 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:03.070 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:03.070 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:03.070 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:03.070 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:03.071 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:03.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:03.071 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:03.071 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:03.071 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:03.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:03.072 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:03.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.072 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:03.073 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:03.073 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:03.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:03.073 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:03Z" 2023-10-17T19:36:03.073 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:03.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:03Z" } 2023-10-17T19:36:03.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:03.076 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:03.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:03.076 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:08.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:08.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:08.079 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:08.137 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:08.137 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:08.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:08.171+0000 7f336948e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:08.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:08.172+0000 7f336948e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:08.187 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:08.187 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:08.188 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:08.188 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:08.188 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:08.188 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:08.188 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:08.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.189 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:08.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:08.189 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:08.190 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:08.190 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:08.190 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:08.190 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:08.190 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:08.191 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:08.191 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:08.191 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:08.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:08.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:08.192 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:08.192 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:08.192 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:08.193 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:08.193 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:08.193 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:08.193 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:08.193 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:08.194 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:08.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:08.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:08.195 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:08.195 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:08.195 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:08.195 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:08.195 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:08.196 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:08.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:08.196 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:08.196 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:08.196 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:08.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:08.197 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:08.197 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:08.197 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:08.198 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:08.198 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:08.198 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:08.198 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:08.199 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:08.199 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:08.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:08.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:08.199 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:08.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.200 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:08.200 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:08.200 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:08.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:08.201 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:08Z" 2023-10-17T19:36:08.201 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:08.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:08Z" } 2023-10-17T19:36:08.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:08.203 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:08.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:08.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:13.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:13.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:13.206 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:13.265 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:13.266 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:13.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:13.297+0000 7f75e86a6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:13.300 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:13.299+0000 7f75e86a6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:13.315 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:13.315 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.316 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:13.317 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:13.318 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:13.319 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:13.320 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:13.321 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:13.322 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:13.323 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:13.324 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:13.325 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:13.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:13.326 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:13.326 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:13.326 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:13.327 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:13.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:13.328 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:13Z" 2023-10-17T19:36:13.328 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:13.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:13Z" } 2023-10-17T19:36:13.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:13.330 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:13.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:13.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:18.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:18.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:18.334 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:18.394 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:18.394 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:18.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:18.430+0000 7f8271558780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:18.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:18.431+0000 7f8271558780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:18.447 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:18.448 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:18.449 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:18.449 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:18.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.449 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:18.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:18.450 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:18.450 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:18.450 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:18.450 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:18.451 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:18.451 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:18.451 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:18.451 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:18.452 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:18.452 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:18.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:18.453 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:18.453 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:18.453 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:18.453 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:18.453 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:18.454 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:18.454 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:18.454 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:18.454 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:18.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:18.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:18.455 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:18.455 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:18.456 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:18.456 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:18.456 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:18.456 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:18.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:18.457 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:18.458 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:18.458 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:18.458 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:18.458 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:18.459 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:18.459 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:18.459 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:18.459 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:18.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:18.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:18.460 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:18.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.460 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:18.460 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:18.461 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:18.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:18.461 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:18Z" 2023-10-17T19:36:18.461 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:18.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:18Z" } 2023-10-17T19:36:18.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:18.463 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:18.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:18.464 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:23.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:23.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:23.466 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:23.525 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:23.526 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:23.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:23.558+0000 7f6f9bc5d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:23.561 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:23.559+0000 7f6f9bc5d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:23.575 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.578 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:23.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:23.578 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:23.579 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:23.579 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:23.579 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:23.580 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:23.580 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:23.580 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:23.580 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:23.581 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:23.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:23.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:23.582 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:23.583 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:23.583 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:23.584 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:23.584 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:23.584 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:23.585 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:23.585 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:23.586 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:23.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:23.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:23.587 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:23.587 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:23.588 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:23.588 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:23.589 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:23.589 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:23.590 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:23.590 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:23.590 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:23.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:23.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:23.591 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:23.591 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:23.592 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:23.592 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:23.592 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:23.592 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:23.592 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:23.593 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:23.593 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:23.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:23.593 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:23.594 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:23.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.594 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:23.594 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:23.595 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:23.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:23.595 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:23Z" 2023-10-17T19:36:23.595 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:23.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:23Z" } 2023-10-17T19:36:23.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:23.598 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:23.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:23.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:28.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:28.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:28.601 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:28.662 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:28.662 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:28.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:28.695+0000 7fee93cef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:28.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:28.696+0000 7fee93cef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:28.714 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:28.714 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:28.715 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:28.715 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:28.715 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:28.715 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:28.715 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:28.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.716 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:28.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:28.716 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:28.717 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:28.717 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:28.717 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:28.717 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:28.717 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:28.718 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:28.718 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:28.718 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:28.718 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:28.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:28.719 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:28.719 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:28.719 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:28.720 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:28.720 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:28.720 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:28.720 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:28.720 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:28.721 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:28.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:28.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:28.721 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:28.722 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:28.722 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:28.722 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:28.722 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:28.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:28.723 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:28.723 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:28.723 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:28.723 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:28.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:28.724 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:28.724 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:28.724 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:28.725 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:28.725 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:28.725 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:28.725 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:28.725 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:28.726 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:28.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:28.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:28.726 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:28.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.727 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:28.727 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:28.727 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:28.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:28.728 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:28Z" 2023-10-17T19:36:28.728 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:28.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:28Z" } 2023-10-17T19:36:28.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:28.730 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:28.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:28.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:33.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:33.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:33.733 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:33.791 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:33.792 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:33.825+0000 7f570b288780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:33.826+0000 7f570b288780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:33.842 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:33.842 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:33.843 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:33.843 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:33.843 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:33.843 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:33.844 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:33.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.844 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:33.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:33.845 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:33.845 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:33.845 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:33.845 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:33.845 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:33.846 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:33.846 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:33.846 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:33.846 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:33.847 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:33.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:33.847 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:33.848 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:33.848 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:33.848 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:33.848 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:33.848 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:33.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:33.849 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:33.849 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:33.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:33.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:33.850 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:33.850 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:33.850 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:33.851 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:33.851 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:33.851 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:33.851 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:33.851 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:33.852 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:33.852 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:33.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:33.853 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:33.853 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:33.853 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:33.853 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:33.853 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:33.854 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:33.854 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:33.854 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:33.854 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:33.855 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:33.855 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:33.855 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:33.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.855 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:33.856 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:33.856 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:33.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:33.856 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:33Z" 2023-10-17T19:36:33.857 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:33.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:33Z" } 2023-10-17T19:36:33.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:33.859 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:33.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:33.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:38.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:38.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:38.863 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:38.922 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:38.922 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:38.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:38.955+0000 7fa517b62780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:38.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:38.956+0000 7fa517b62780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:38.971 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:38.971 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:38.972 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:38.972 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:38.972 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:38.972 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:38.973 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:38.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.973 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:38.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:38.973 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:38.974 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:38.974 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:38.974 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:38.974 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:38.974 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:38.975 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:38.975 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:38.975 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:38.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:38.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:38.976 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:38.976 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:38.976 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:38.977 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:38.977 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:38.977 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:38.977 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:38.977 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:38.978 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:38.978 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:38.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:38.978 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:38.979 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:38.979 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:38.979 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:38.979 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:38.980 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:38.980 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:38.980 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:38.980 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:38.981 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:38.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:38.981 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:38.981 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:38.982 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:38.982 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:38.982 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:38.982 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:38.983 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:38.983 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:38.983 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:38.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:38.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:38.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:38.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.984 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:38.984 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:38.984 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:38.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:38.985 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:38Z" 2023-10-17T19:36:38.985 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:38.986 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:38Z" } 2023-10-17T19:36:38.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:38.987 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:38.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:38.988 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:43.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:43.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:43.991 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:44.051 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:44.051 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:44.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:44.084+0000 7f3f7fc63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:44.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:44.086+0000 7f3f7fc63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:44.101 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:44.101 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:44.103 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:44.104 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:44.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:44.106 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:44.107 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:44.109 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:44.110 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:44.110 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:44.110 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:44.110 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:44.111 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:44.111 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:44.111 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:44.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:44.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:44.112 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:44.112 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:44.113 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:44.113 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:44.113 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:44.113 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:44.113 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:44.114 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:44.114 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:44.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:44.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:44.115 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:44.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.115 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:44.115 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:44.115 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:44.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:44.116 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:44Z" 2023-10-17T19:36:44.116 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:44.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:44Z" } 2023-10-17T19:36:44.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:44.118 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:44.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:44.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:49.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:49.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:49.121 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:49.180 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:49.181 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:49.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:49.213+0000 7f404cd2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:49.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:49.214+0000 7f404cd2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:49.230 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:49.230 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:49.230 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:49.231 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:49.231 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:49.231 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:49.231 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:49.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.232 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:49.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:49.232 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:49.232 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:49.232 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:49.233 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:49.233 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:49.233 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:49.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:49.234 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:49.234 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:49.234 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:49.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:49.235 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:49.235 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:49.235 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:49.235 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:49.235 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:49.236 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:49.236 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:49.236 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:49.236 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:49.236 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:49.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:49.237 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:49.237 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:49.238 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:49.238 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:49.238 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:49.238 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:49.238 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:49.239 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:49.239 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:49.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:49.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:49.240 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:49.240 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:49.240 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:49.240 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:49.240 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:49.242 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:49.242 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:49.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.242 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:49.243 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:49.243 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:49.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:49.243 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:49Z" 2023-10-17T19:36:49.243 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:49.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:49Z" } 2023-10-17T19:36:49.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:49.245 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:49.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:49.246 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:54.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:54.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:54.249 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:54.309 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:54.309 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:54.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:54.341+0000 7f3788d7a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:54.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:54.342+0000 7f3788d7a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:54.358 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:54.359 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:54.359 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:54.359 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:54.359 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:54.360 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:54.360 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:54.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.360 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:54.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:54.361 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:54.361 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:54.361 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:54.361 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:54.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:54.362 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:54.362 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:54.362 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:54.363 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:54.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:54.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:54.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:54.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:54.364 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:54.364 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:54.364 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:54.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:54.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:54.365 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:54.365 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:54.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:54.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:54.366 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:54.366 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:54.367 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:54.367 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:54.367 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:54.367 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:54.367 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:54.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:54.368 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:54.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:54.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:54.369 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:54.369 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:54.369 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:54.369 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:54.370 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:54.370 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:54.370 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:54.370 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:54.370 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:54.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:54.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:54.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:54.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.371 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:54.372 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:54.372 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:54.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:54.372 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:54Z" 2023-10-17T19:36:54.372 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:54.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:54Z" } 2023-10-17T19:36:54.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:54.375 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:54.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:54.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:36:59.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:59.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:36:59.378 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:36:59.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:36:59.435 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:36:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:59.467+0000 7fb807934780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:59.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:36:59.468+0000 7fb807934780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:36:59.484 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:36:59.484 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:36:59.484 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:36:59.485 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:36:59.485 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:36:59.485 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:36:59.485 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:59.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.486 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:36:59.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:59.486 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:36:59.486 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:59.487 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:59.487 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:36:59.487 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:59.488 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:36:59.488 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:59.488 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:36:59.488 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:59.489 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:59.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:59.489 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:36:59.489 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:59.490 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:59.490 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:36:59.490 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:59.490 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:59.491 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:59.491 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:36:59.491 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:59.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:59.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:59.492 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:36:59.492 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:59.493 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:59.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:36:59.493 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:59.494 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:59.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:59.495 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:36:59.495 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:59.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:59.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:36:59.496 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:36:59.496 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:36:59.496 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:36:59.496 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:36:59.497 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:36:59.497 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:36:59.497 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:36:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:36:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:36:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:36:59.498 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:36:59.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.499 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:36:59.499 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:36:59.499 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:36:59.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:36:59.500 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:36:59Z" 2023-10-17T19:36:59.500 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:36:59.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:36:59Z" } 2023-10-17T19:36:59.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:36:59.502 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:59.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:36:59.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:04.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:04.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:04.506 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:04.565 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:04.566 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:04.598+0000 7f3c0ca16780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:04.599+0000 7f3c0ca16780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:04.615 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:04.615 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:04.616 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:04.616 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:04.617 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:04.617 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:04.617 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:04.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.618 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:04.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:04.619 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:04.619 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:04.620 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:04.620 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:04.620 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:04.621 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:04.621 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:04.622 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:04.622 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:04.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:04.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:04.623 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:04.623 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:04.624 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:04.624 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:04.624 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:04.624 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:04.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:04.625 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:04.625 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:04.625 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:04.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:04.626 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:04.627 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:04.627 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:04.627 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:04.628 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:04.628 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:04.628 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:04.628 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:04.629 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:04.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:04.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:04.629 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:04.630 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:04.630 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:04.630 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:04.630 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:04.630 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:04.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:04.632 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:04.632 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:04.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.632 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:04.632 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:04.633 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:04.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:04.633 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:04Z" 2023-10-17T19:37:04.633 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:04.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:04Z" } 2023-10-17T19:37:04.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:04.635 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:04.636 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:04.636 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:09.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:09.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:09.638 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:09.697 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:09.698 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:09.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:09.731+0000 7fd481b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:09.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:09.732+0000 7fd481b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:09.747 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:09Z" 2023-10-17T19:37:09.760 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:09.761 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:09Z" } 2023-10-17T19:37:09.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:09.762 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:09.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:09.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:14.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:14.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:14.765 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:14.824 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:14.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:14.857+0000 7fcbaac12780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:14.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:14.858+0000 7fcbaac12780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:14.873 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:14.874 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:14.874 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:14.874 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:14.874 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:14.875 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:14.875 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:14.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.875 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:14.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:14.876 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:14.876 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:14.876 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:14.876 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:14.877 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:14.877 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:14.877 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:14.877 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:14.877 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:14.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:14.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:14.878 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:14.878 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:14.879 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:14.879 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:14.879 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:14.879 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:14.880 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:14.880 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:14.880 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:14.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:14.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:14.881 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:14.881 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:14.881 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:14.881 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:14.886 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:14.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.886 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:14.886 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:14.886 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:14.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:14.887 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:14Z" 2023-10-17T19:37:14.887 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:14.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:14Z" } 2023-10-17T19:37:14.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:14.889 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:14.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:14.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:19.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:19.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:19.892 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:19.952 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:19.952 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:19.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:19.987+0000 7fb1ef045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:19.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:19.988+0000 7fb1ef045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:20.006 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:20.007 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:20.007 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:20.007 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:20.007 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:20.008 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:20.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.008 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:20.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:20.008 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:20.009 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:20.009 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:20.009 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:20.009 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:20.009 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:20.010 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:20.010 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:20.010 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:20.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:20.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:20.011 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:20.011 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:20.011 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:20.012 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:20.012 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:20.012 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:20.012 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:20.012 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:20.013 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:20.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:20.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:20.013 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:20.014 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:20.014 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:20.014 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:20.014 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:20.015 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:20.015 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:20.015 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:20.015 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:20.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:20.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:20.016 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:20.016 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:20.017 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:20.017 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:20.017 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:20.017 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:20.017 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:20.018 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:20.018 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:20.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:20.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:20.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:20.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.019 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:20.019 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:20.019 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:20.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:20.020 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:19Z" 2023-10-17T19:37:20.020 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:20.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:19Z" } 2023-10-17T19:37:20.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:20.022 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:20.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:20.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:25.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:25.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:25.025 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:25.084 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:25.084 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:25.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:25.116+0000 7f7e0ee70780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:25.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:25.117+0000 7f7e0ee70780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:25.133 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:25.133 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:25.133 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:25.133 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:25.134 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:25.134 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:25.134 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:25.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.135 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:25.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:25.135 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:25.135 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:25.135 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:25.136 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:25.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:25.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:25.137 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:25.137 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:25.138 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:25.138 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:25.138 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:25.138 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:25.138 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:25.139 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:25.139 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:25.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:25.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:25.140 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:25.140 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:25.140 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:25.140 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:25.140 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:25.141 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:25.141 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:25.141 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:25.141 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:25.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:25.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:25.142 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:25.142 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:25.143 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:25.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:25.143 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:25.143 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:25.143 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:25.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:25.144 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:25.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:25.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:25.144 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:25.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.145 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:25.145 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:25.145 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:25.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:25.146 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:25Z" 2023-10-17T19:37:25.146 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:25.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:25Z" } 2023-10-17T19:37:25.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:25.148 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:25.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:25.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:30.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:30.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:30.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:30.210 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:30.211 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:30.243+0000 7f00595a7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:30.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:30.244+0000 7f00595a7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:30.263 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:30.263 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:30.264 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:30.264 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:30.264 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:30.264 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:30.265 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:30.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.265 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:30.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:30.265 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:30.266 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:30.266 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:30.266 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:30.266 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:30.266 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:30.267 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:30.267 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:30.267 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:30.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:30.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:30.268 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:30.268 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:30.268 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:30.269 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:30.269 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:30.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:30.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:30.269 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:30.270 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:30.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:30.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:30.271 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:30.271 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:30.271 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:30.271 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:30.271 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:30.272 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:30.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:30.272 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:30.272 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:30.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:30.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:30.273 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:30.273 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:30.274 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:30.274 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:30.274 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:30.274 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:30.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:30.275 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:30.275 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:30.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:30.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:30.276 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:30.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.276 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:30.277 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:30.277 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:30.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:30.278 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:30Z" 2023-10-17T19:37:30.278 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:30.280 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:30Z" } 2023-10-17T19:37:30.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:30.281 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:30.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:30.282 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:35.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:35.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:35.285 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:35.344 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:35.345 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:35.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:35.378+0000 7ff6a2751780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:35.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:35.379+0000 7ff6a2751780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:35.395 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:35.395 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:35.396 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:35.396 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:35.396 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:35.396 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:35.396 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:35.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.397 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:35.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:35.397 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:35.398 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:35.398 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:35.398 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:35.398 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:35.398 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:35.399 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:35.399 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:35.399 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:35.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:35.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:35.400 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:35.400 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:35.400 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:35.401 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:35.401 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:35.401 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:35.401 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:35.403 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:35.403 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:35.403 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:35.403 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:35.403 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:35.404 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:35.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:35.404 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:35.404 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:35.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:35.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:35.405 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:35.406 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:35.406 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:35.406 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:35.406 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:35.406 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:35.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:35.407 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:35.407 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:35.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:35.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:35.408 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:35.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.408 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:35.408 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:35.409 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:35.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:35.409 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:35Z" 2023-10-17T19:37:35.409 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:35.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:35Z" } 2023-10-17T19:37:35.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:35.411 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:35.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:35.412 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:40.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:40.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:40.415 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:40.473 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:40.474 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:40.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:40.506+0000 7fe827040780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:40.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:40.507+0000 7fe827040780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:40.523 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:40Z" 2023-10-17T19:37:40.536 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:40.537 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:40Z" } 2023-10-17T19:37:40.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:40.538 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:40.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:40.539 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:45.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:45.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:45.541 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:45.600 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:45.600 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:45.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:45.635+0000 7f3705cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:45.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:45.636+0000 7f3705cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:45.655 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:45.655 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:45.655 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:45.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.657 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:45.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:45.657 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:45.657 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:45.658 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:45.658 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:37:45.658 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:45.658 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:45.659 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:45.659 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:37:45.659 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:45.659 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:45.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:45.660 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:45.661 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:45.661 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:45.661 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:45.661 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:45.661 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:45.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:45.662 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:45.662 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:45.662 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:45.663 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:45.663 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:45.663 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:45.663 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:45.663 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:45.664 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:45.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:45.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:45.665 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:45.665 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:45.665 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:45.665 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:45.665 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:45.666 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:45.666 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:45.666 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:45.666 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:45.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:45.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:45.667 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:45.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.667 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:45.668 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:45.668 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:45.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:45.668 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:45Z" 2023-10-17T19:37:45.669 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:45.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571220.511436_1095.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:33:40.511436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:45Z" } 2023-10-17T19:37:45.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:45.671 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:45.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:45.671 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:37:50.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:50.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:50.674 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:50.732 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:50.733 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:50.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:50.765+0000 7f3df1b3c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:50.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:50.766+0000 7f3df1b3c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:50.781 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:37:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:37:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:50Z" 2023-10-17T19:37:50.795 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:50.796 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571347.356986_1134.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:35:47.356986Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:50Z" } 2023-10-17T19:37:50.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571347.356986_1134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:35:47.356986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:50.797 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:50.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:50.798 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:37:50.798 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:37:50.799 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:37:50.799 DEBUG:boto:Method: HEAD 2023-10-17T19:37:50.800 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:37:50.800 DEBUG:boto:Data: 2023-10-17T19:37:50.800 DEBUG:boto:Headers: {} 2023-10-17T19:37:50.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.801 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.802 DEBUG:boto:Params: {} 2023-10-17T19:37:50.802 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:37:50.803 DEBUG:boto:Token: None 2023-10-17T19:37:50.804 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-72/ 2023-10-17T19:37:50.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4= 2023-10-17T19:37:50.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4=', 'Content-Length': '0'} 2023-10-17T19:37:50.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049e7a501e6670afb-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.811 DEBUG:boto:path=/hbmuog-72/obj5 2023-10-17T19:37:50.812 DEBUG:boto:auth_path=/hbmuog-72/obj5 2023-10-17T19:37:50.812 DEBUG:boto:Method: PUT 2023-10-17T19:37:50.813 DEBUG:boto:Path: /hbmuog-72/obj5 2023-10-17T19:37:50.813 DEBUG:boto:Data: 2023-10-17T19:37:50.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:37:50.814 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.814 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.815 DEBUG:boto:Params: {} 2023-10-17T19:37:50.815 DEBUG:boto:Token: None 2023-10-17T19:37:50.816 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-72/obj5 2023-10-17T19:37:50.816 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7OqWnv54bEkp627888Zl/NwqrwU= 2023-10-17T19:37:50.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7OqWnv54bEkp627888Zl/NwqrwU='} 2023-10-17T19:37:50.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdbb6cc640d9d4af-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.825 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:37:50.825 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:37:50.826 DEBUG:boto:Method: HEAD 2023-10-17T19:37:50.826 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:37:50.827 DEBUG:boto:Data: 2023-10-17T19:37:50.827 DEBUG:boto:Headers: {} 2023-10-17T19:37:50.828 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.828 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.828 DEBUG:boto:Params: {} 2023-10-17T19:37:50.829 DEBUG:boto:Token: None 2023-10-17T19:37:50.829 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-72/ 2023-10-17T19:37:50.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4= 2023-10-17T19:37:50.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4=', 'Content-Length': '0'} 2023-10-17T19:37:50.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc58951d65ec951f-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.833 DEBUG:boto:path=/hbmuog-72/obj6 2023-10-17T19:37:50.834 DEBUG:boto:auth_path=/hbmuog-72/obj6 2023-10-17T19:37:50.834 DEBUG:boto:Method: PUT 2023-10-17T19:37:50.835 DEBUG:boto:Path: /hbmuog-72/obj6 2023-10-17T19:37:50.835 DEBUG:boto:Data: 2023-10-17T19:37:50.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:37:50.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.836 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.837 DEBUG:boto:Params: {} 2023-10-17T19:37:50.837 DEBUG:boto:Token: None 2023-10-17T19:37:50.838 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-72/obj6 2023-10-17T19:37:50.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8ML7fPhP+dinNpfymqRgUlwsqHM= 2023-10-17T19:37:50.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8ML7fPhP+dinNpfymqRgUlwsqHM='} 2023-10-17T19:37:50.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d732bc6b2b18e6cd-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.846 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:37:50.846 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:37:50.847 DEBUG:boto:Method: HEAD 2023-10-17T19:37:50.847 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:37:50.847 DEBUG:boto:Data: 2023-10-17T19:37:50.848 DEBUG:boto:Headers: {} 2023-10-17T19:37:50.848 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.849 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.849 DEBUG:boto:Params: {} 2023-10-17T19:37:50.849 DEBUG:boto:Token: None 2023-10-17T19:37:50.850 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-73/ 2023-10-17T19:37:50.850 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus= 2023-10-17T19:37:50.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus=', 'Content-Length': '0'} 2023-10-17T19:37:50.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000473f9a6624662329-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.854 DEBUG:boto:path=/hbmuog-73/obj7 2023-10-17T19:37:50.855 DEBUG:boto:auth_path=/hbmuog-73/obj7 2023-10-17T19:37:50.855 DEBUG:boto:Method: PUT 2023-10-17T19:37:50.855 DEBUG:boto:Path: /hbmuog-73/obj7 2023-10-17T19:37:50.856 DEBUG:boto:Data: 2023-10-17T19:37:50.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:37:50.857 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.857 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.858 DEBUG:boto:Params: {} 2023-10-17T19:37:50.858 DEBUG:boto:Token: None 2023-10-17T19:37:50.858 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-73/obj7 2023-10-17T19:37:50.859 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:AhqfJx1kOSRz5E5/FCd+GWPw5jw= 2023-10-17T19:37:50.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AhqfJx1kOSRz5E5/FCd+GWPw5jw='} 2023-10-17T19:37:50.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1c6a3acc9303eac-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.907 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:37:50.908 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:37:50.908 DEBUG:boto:Method: HEAD 2023-10-17T19:37:50.908 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:37:50.909 DEBUG:boto:Data: 2023-10-17T19:37:50.909 DEBUG:boto:Headers: {} 2023-10-17T19:37:50.910 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.910 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.910 DEBUG:boto:Params: {} 2023-10-17T19:37:50.911 DEBUG:boto:Token: None 2023-10-17T19:37:50.911 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-73/ 2023-10-17T19:37:50.912 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus= 2023-10-17T19:37:50.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus=', 'Content-Length': '0'} 2023-10-17T19:37:50.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f35240d96d1b2d6-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.915 DEBUG:boto:path=/hbmuog-73/obj8 2023-10-17T19:37:50.916 DEBUG:boto:auth_path=/hbmuog-73/obj8 2023-10-17T19:37:50.916 DEBUG:boto:Method: PUT 2023-10-17T19:37:50.917 DEBUG:boto:Path: /hbmuog-73/obj8 2023-10-17T19:37:50.917 DEBUG:boto:Data: 2023-10-17T19:37:50.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:37:50.918 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:37:50.918 DEBUG:boto:Port: 8000 2023-10-17T19:37:50.919 DEBUG:boto:Params: {} 2023-10-17T19:37:50.919 DEBUG:boto:Token: None 2023-10-17T19:37:50.920 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:37:50 GMT /hbmuog-73/obj8 2023-10-17T19:37:50.920 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ehU4YxZhiyDwVu+Kpc6MQSUrLm8= 2023-10-17T19:37:50.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ehU4YxZhiyDwVu+Kpc6MQSUrLm8='} 2023-10-17T19:37:50.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005799405f3e3eda8e-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:37:50.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:50.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:50.929 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:50.981 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:50.982 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:51.015 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:37:51.028 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:37:51.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.029 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:37:51.029 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:35:47.356986Z" 2023-10-17T19:37:51.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.030 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:51.030 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:37:51.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.031 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:51.031 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:37:51.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.032 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:51.032 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:37:51.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.032 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:37:51.033 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:51.034 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:37:51.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.035 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:51.085 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:51.086 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:51.116+0000 7fc9c1f6c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:37:51.117+0000 7fc9c1f6c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:37:51.130 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:51.131 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:51.131 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:51.131 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:51.131 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:51.132 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:37:51.132 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:51.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.132 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:51.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.133 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:51.133 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.133 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:51.133 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:37:51.134 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.134 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:37:51.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.134 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:37:51.134 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:51.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.137 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:51.137 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.137 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:51.138 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:37:51.138 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.138 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:51.138 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.139 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:37:51.139 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:51.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.140 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:51.140 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.140 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:51.141 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:37:51.141 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.141 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:51.141 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.141 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:37:51.142 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:51.142 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.143 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:51.143 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.143 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:37:51.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:37:51.144 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.144 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:51.144 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:37:51.145 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:37:51.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:51.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.146 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:51.146 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:37:51.146 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:37:51.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.147 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:37:51.147 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:51.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571347.356986_1134.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:35:47.356986Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:37:51Z" } 2023-10-17T19:37:51.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571347.356986_1134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:35:47.356986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:37:51.150 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:51.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:37:51.151 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:37:51.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.152 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:51.202 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:51.203 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:51.252 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:37:51.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.253 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:37:51.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:37:51.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.253 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:37:51.254 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:37:51.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.254 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:37:51.254 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:37:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:37:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:37:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.256 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:37:51.257 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:37:51.257 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:37:51.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:51.258 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:51.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:51.310 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:51.359 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:51.360 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:51.360 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:51.360 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:51.360 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:51.361 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:37:51.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.361 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:51.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.362 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:51.362 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.362 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:51.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:37:51.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.363 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:51.363 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.363 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:37:51.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.364 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:51.364 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.364 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:51.364 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:37:51.365 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.365 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:51.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.365 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:37:51.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.366 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:51.366 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:37:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:37:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:37:51.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:51.370 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:51.370 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:51.371 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:51.371 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:37:51.371 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:51.371 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:37:51.371 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:51.372 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:37:51.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:51.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:51.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.373 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:51.373 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:37:51.373 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:37:51.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:51.374 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:37:51.374 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:51.375 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041122", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:33:41.702225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028656", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:33:41.789887Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047136", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:33:41.762296Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:37:51Z" } 2023-10-17T19:37:51.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:37:51.376 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:37:51.377 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:37:51.377 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:37:56.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:56.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:37:56.379 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:37:56.441 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:37:56.441 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:37:56.491 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:37:56.492 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:37:56.492 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:37:56.492 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:37:56.492 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:37:56.492 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:37:56.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.493 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:37:56.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:56.493 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:37:56.493 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:56.494 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:56.494 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:37:56.494 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:56.494 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:56.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:37:56.496 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:56.496 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:56.496 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:37:56.496 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:56.496 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:37:56.497 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:56.497 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:37:56.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:56.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:56.498 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:37:56.498 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:56.498 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:56.498 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:37:56.499 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:56.499 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:37:56.499 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:56.499 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:37:56.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:56.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:37:56.500 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:37:56.500 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:37:56.500 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:37:56.501 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:37:56.501 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:37:56.501 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:37:56.501 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:37:56.501 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:37:56.502 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:56.502 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:37:56.502 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:37:56.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.503 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:37:56.503 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:37:56.503 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:37:56.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:37:56.503 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:37:56Z" 2023-10-17T19:37:56.504 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:37:56.504 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041122", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:33:41.702225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028656", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:33:41.789887Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047136", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:33:41.762296Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:37:56Z" } 2023-10-17T19:37:56.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:37:56.505 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:37:56.506 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:37:56.506 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:38:01.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:01.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:01.509 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:01.570 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:01.571 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:01.621 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:01.621 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:01.622 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:01.622 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:01.622 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:01.622 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:01.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.623 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:01.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:01.623 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:01.624 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:01.624 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:01.624 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:01.624 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:01.625 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:01.625 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:01.625 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:01.625 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:01.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:01.626 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:01.626 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:01.626 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:38:01.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:01.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:01.628 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:01.628 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:01.628 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:01.629 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:38:01.629 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:01.629 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:01.629 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:01.630 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:38:01.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:01.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:01.630 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:38:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:01.632 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:01.632 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:01.632 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:38:01.632 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:01.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:01.633 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:01.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.633 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:01.633 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:01.634 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:01.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:01.634 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:01Z" 2023-10-17T19:38:01.634 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:01.635 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041122", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:33:41.702225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028656", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:33:41.789887Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047136", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:33:41.762296Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:01Z" } 2023-10-17T19:38:01.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:38:01.636 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:01.637 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:38:01.637 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:38:06.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:06.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:06.639 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:06.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:06.700 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:06.751 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:06.751 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:06.751 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:06.752 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:06.752 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:06.752 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:06.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.753 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:06.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:06.753 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:06.753 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:06.753 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:06.754 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:06.754 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:06.754 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:06.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:06.754 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:06.755 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:06.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:06.755 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:06.756 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:06.756 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:06.756 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:38:06.756 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:06.756 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:06.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:06.757 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:38:06.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:06.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:06.758 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:06.758 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:06.758 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:06.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:38:06.759 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:06.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:06.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:06.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:38:06.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:06.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:06.760 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:06.760 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:06.760 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:06.761 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:38:06.761 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:06.761 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:06.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:06.761 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:38:06.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:06.762 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:06.762 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:06.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:06Z" 2023-10-17T19:38:06.763 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:06.764 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041122", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:33:41.702225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028656", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:33:41.789887Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047136", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:33:41.762296Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:06Z" } 2023-10-17T19:38:06.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:38:06.765 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:06.766 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:38:06.766 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:38:11.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:11.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:11.769 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:11.830 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:11.831 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:11.882 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:11.883 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:11.883 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:11.883 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:11.883 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:11.883 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:11.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.884 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:11.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:11.884 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:11.884 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:11.885 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:11.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:11.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:11.886 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:11.887 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:11.888 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:11.888 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:11.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:11.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:11.889 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:11.889 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:11.889 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:11.889 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:11.889 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:11.890 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:11.890 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:11.890 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:11.890 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:11.891 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:11.892 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:11.892 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:11.892 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:11.892 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:11.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:11.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:11.893 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:11.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.893 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:11.893 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:11.894 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:11.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:11.894 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:11Z" 2023-10-17T19:38:11.894 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:11.895 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:11Z" } 2023-10-17T19:38:11.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:11.896 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:11.896 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:11.897 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:38:11.897 DEBUG:boto:path=/hbmuog-73/ 2023-10-17T19:38:11.897 DEBUG:boto:auth_path=/hbmuog-73/ 2023-10-17T19:38:11.898 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.898 DEBUG:boto:Path: /hbmuog-73/ 2023-10-17T19:38:11.899 DEBUG:boto:Data: 2023-10-17T19:38:11.899 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.899 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.900 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.900 DEBUG:boto:Params: {} 2023-10-17T19:38:11.901 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:38:11.901 DEBUG:boto:Token: None 2023-10-17T19:38:11.902 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-73/ 2023-10-17T19:38:11.902 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:8ITrzw6hRCYX6qrZ3fgkI7v04hw= 2023-10-17T19:38:11.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8ITrzw6hRCYX6qrZ3fgkI7v04hw=', 'Content-Length': '0'} 2023-10-17T19:38:11.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000730d59fc16e9b519-00652ee2a3-26af-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.908 DEBUG:boto:path=/hbmuog-73/obj5 2023-10-17T19:38:11.908 DEBUG:boto:auth_path=/hbmuog-73/obj5 2023-10-17T19:38:11.909 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.909 DEBUG:boto:Path: /hbmuog-73/obj5 2023-10-17T19:38:11.910 DEBUG:boto:Data: 2023-10-17T19:38:11.910 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.911 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.911 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.911 DEBUG:boto:Params: {} 2023-10-17T19:38:11.912 DEBUG:boto:Token: None 2023-10-17T19:38:11.912 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-73/obj5 2023-10-17T19:38:11.913 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rL75/CMAIClm4NCoIUQOlxUNRo0= 2023-10-17T19:38:11.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rL75/CMAIClm4NCoIUQOlxUNRo0=', 'Content-Length': '0'} 2023-10-17T19:38:11.914 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000061c21bc59653c41b-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.915 DEBUG:boto:path=/hbmuog-73/obj6 2023-10-17T19:38:11.915 DEBUG:boto:auth_path=/hbmuog-73/obj6 2023-10-17T19:38:11.916 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.916 DEBUG:boto:Path: /hbmuog-73/obj6 2023-10-17T19:38:11.916 DEBUG:boto:Data: 2023-10-17T19:38:11.917 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.917 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.918 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.918 DEBUG:boto:Params: {} 2023-10-17T19:38:11.918 DEBUG:boto:Token: None 2023-10-17T19:38:11.919 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-73/obj6 2023-10-17T19:38:11.919 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:O7EAb2VQdIC3hENsBrXIkW4U4ds= 2023-10-17T19:38:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O7EAb2VQdIC3hENsBrXIkW4U4ds=', 'Content-Length': '0'} 2023-10-17T19:38:11.921 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000054f1f0f7dc3fed6a-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.922 DEBUG:boto:path=/hbmuog-73/obj7 2023-10-17T19:38:11.922 DEBUG:boto:auth_path=/hbmuog-73/obj7 2023-10-17T19:38:11.923 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.923 DEBUG:boto:Path: /hbmuog-73/obj7 2023-10-17T19:38:11.923 DEBUG:boto:Data: 2023-10-17T19:38:11.924 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.924 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.925 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.925 DEBUG:boto:Params: {} 2023-10-17T19:38:11.925 DEBUG:boto:Token: None 2023-10-17T19:38:11.926 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-73/obj7 2023-10-17T19:38:11.926 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:W96P5RLmBjXt9E6RMdoILdE70zE= 2023-10-17T19:38:11.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W96P5RLmBjXt9E6RMdoILdE70zE=', 'Content-Length': '0'} 2023-10-17T19:38:11.928 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009e65f13d73eca24b-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.929 DEBUG:boto:path=/hbmuog-73/obj8 2023-10-17T19:38:11.929 DEBUG:boto:auth_path=/hbmuog-73/obj8 2023-10-17T19:38:11.930 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.930 DEBUG:boto:Path: /hbmuog-73/obj8 2023-10-17T19:38:11.930 DEBUG:boto:Data: 2023-10-17T19:38:11.931 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.931 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.931 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.932 DEBUG:boto:Params: {} 2023-10-17T19:38:11.932 DEBUG:boto:Token: None 2023-10-17T19:38:11.933 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-73/obj8 2023-10-17T19:38:11.933 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:jxH0uFqbn1s6RVb3JcqCNGRTa/k= 2023-10-17T19:38:11.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jxH0uFqbn1s6RVb3JcqCNGRTa/k=', 'Content-Length': '0'} 2023-10-17T19:38:11.935 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c46153a391f231af-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.935 DEBUG:boto:path=/hbmuog-72/ 2023-10-17T19:38:11.936 DEBUG:boto:auth_path=/hbmuog-72/ 2023-10-17T19:38:11.936 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.936 DEBUG:boto:Path: /hbmuog-72/ 2023-10-17T19:38:11.937 DEBUG:boto:Data: 2023-10-17T19:38:11.937 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.937 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.938 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.938 DEBUG:boto:Params: {} 2023-10-17T19:38:11.939 DEBUG:boto:Token: None 2023-10-17T19:38:11.939 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-72/ 2023-10-17T19:38:11.939 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:2Vw+LP9XXnf2/qkKXNmXHHWQDiQ= 2023-10-17T19:38:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2Vw+LP9XXnf2/qkKXNmXHHWQDiQ=', 'Content-Length': '0'} 2023-10-17T19:38:11.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a0db4938b5f101a-00652ee2a3-26af-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.943 DEBUG:boto:path=/hbmuog-72/obj7 2023-10-17T19:38:11.943 DEBUG:boto:auth_path=/hbmuog-72/obj7 2023-10-17T19:38:11.943 DEBUG:boto:Method: HEAD 2023-10-17T19:38:11.944 DEBUG:boto:Path: /hbmuog-72/obj7 2023-10-17T19:38:11.944 DEBUG:boto:Data: 2023-10-17T19:38:11.944 DEBUG:boto:Headers: {} 2023-10-17T19:38:11.945 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:38:11.945 DEBUG:boto:Port: 8001 2023-10-17T19:38:11.946 DEBUG:boto:Params: {} 2023-10-17T19:38:11.946 DEBUG:boto:Token: None 2023-10-17T19:38:11.946 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:11 GMT /hbmuog-72/obj7 2023-10-17T19:38:11.947 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sEhiNP/kGPp4rHQp2KNhw9hApJ4= 2023-10-17T19:38:11.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sEhiNP/kGPp4rHQp2KNhw9hApJ4=', 'Content-Length': '0'} 2023-10-17T19:38:11.949 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000166e34046079af75-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:11.953 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-10-17T19:38:11.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:11.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:11.955 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:12.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:12.007 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:12.038 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:38:12.052 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:38:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:38:12.052 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:35:47.356986Z" 2023-10-17T19:38:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:38:12.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:38:12.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:38:12.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.055 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:38:12.056 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:38:12.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.058 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:12.108 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:12.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:12.137+0000 7f0720d54780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:12.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:12.138+0000 7f0720d54780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:12.153 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:12.153 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:12.153 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:12.153 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:12.153 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:12.154 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:12.154 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.154 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:38:12.155 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.156 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:38:12.156 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.156 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:38:12.156 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.157 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:12.157 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.157 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.158 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:12.158 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.158 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.158 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.158 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:38:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:12.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.164 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:12.164 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:38:12.164 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:38:12.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.165 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:12Z" 2023-10-17T19:38:12.165 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:12.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571347.356986_1134.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:35:47.356986Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:38:12Z" } 2023-10-17T19:38:12.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571347.356986_1134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:35:47.356986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:38:12.167 DEBUG:rgw_multi.tests:log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:38:12.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:12.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:12.169 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:12.215 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:12.215 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:12.261 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:12.261 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-74 2023-10-17T19:38:12.262 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:38:12.262 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:38:12.262 DEBUG:boto:Method: PUT 2023-10-17T19:38:12.263 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:38:12.263 DEBUG:boto:Data: 2023-10-17T19:38:12.263 DEBUG:boto:Headers: {} 2023-10-17T19:38:12.264 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:12.264 DEBUG:boto:Port: 8000 2023-10-17T19:38:12.265 DEBUG:boto:Params: {} 2023-10-17T19:38:12.265 DEBUG:boto:Token: None 2023-10-17T19:38:12.266 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:38:12 GMT /hbmuog-74/ 2023-10-17T19:38:12.266 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:kgO/SrXLVA4SIaqS0Tnd848ATTw= 2023-10-17T19:38:12.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kgO/SrXLVA4SIaqS0Tnd848ATTw=', 'Content-Length': '0'} 2023-10-17T19:38:12.283 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000084e8174aaf042ee7-00652ee2a4-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:12.283 INFO:rgw_multi.tests:create bucket zone=a1 name=hbmuog-75 2023-10-17T19:38:12.284 DEBUG:boto:path=/hbmuog-75/ 2023-10-17T19:38:12.284 DEBUG:boto:auth_path=/hbmuog-75/ 2023-10-17T19:38:12.284 DEBUG:boto:Method: PUT 2023-10-17T19:38:12.285 DEBUG:boto:Path: /hbmuog-75/ 2023-10-17T19:38:12.285 DEBUG:boto:Data: 2023-10-17T19:38:12.285 DEBUG:boto:Headers: {} 2023-10-17T19:38:12.286 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:12.286 DEBUG:boto:Port: 8000 2023-10-17T19:38:12.287 DEBUG:boto:Params: {} 2023-10-17T19:38:12.287 DEBUG:boto:Token: None 2023-10-17T19:38:12.287 DEBUG:boto:StringToSign: PUT Tue, 17 Oct 2023 19:38:12 GMT /hbmuog-75/ 2023-10-17T19:38:12.288 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Q6uxi8B5JTyENa0W1/xeEEyHUj8= 2023-10-17T19:38:12.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:12 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Q6uxi8B5JTyENa0W1/xeEEyHUj8=', 'Content-Length': '0'} 2023-10-17T19:38:12.303 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006bc2305d84e38a4d-00652ee2a4-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:12.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.304 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:12.355 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:12.356 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:12.389 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:38:12.406 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:38:12.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.407 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:38:12.407 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:38:12.301256Z" 2023-10-17T19:38:12.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.408 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:12.408 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:38:12.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:12.409 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:38:12.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.410 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:38:12.411 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.411 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:38:12.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:12.412 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:12.463 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:12.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:12.493+0000 7f3575ae5780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:12.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:12.494+0000 7f3575ae5780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:12.509 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:12.509 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:12.510 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:12.510 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:12.510 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:12.510 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:12.511 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.511 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:12.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.511 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.295551_1141.1", 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:38:12.512 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.513 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.295551Z", 2023-10-17T19:38:12.513 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.514 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:12.514 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.514 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.514 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:12.514 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.516 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:12.516 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.516 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.516 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:12.517 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.517 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.517 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.517 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:38:12.517 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:12.518 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:12.518 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:12.519 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:38:12.520 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:12.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:12.520 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:12.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.521 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:12.521 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:38:12.521 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:38:12.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:12.521 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:12Z" 2023-10-17T19:38:12.522 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:12.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.295551_1141.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.295551Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:38:12Z" } 2023-10-17T19:38:12.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.295551_1141.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.295551Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:38:12.523 DEBUG:rgw_multi.tests:log_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.295551_1141.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:12.524 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571492.301256_1144.1 target=1_1697571492.295551_1141.1 2023-10-17T19:38:17.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:17.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:17.527 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:17.585 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:17.585 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:17.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:17.617+0000 7f472331f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:17.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:17.618+0000 7f472331f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:17.634 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:17.634 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:17.634 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:17.635 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:17.635 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:17.635 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:17.635 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:17.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.636 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:17.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:17.636 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:17.636 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:17.637 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:17.637 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:38:17.637 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:17.637 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:38:17.637 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:17.638 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:38:17.638 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:17.638 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:17.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:17.639 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:17.639 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:17.639 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:17.639 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:17.639 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:17.640 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:17.640 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:17.640 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:38:17.640 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:17.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:17.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:17.641 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:17.641 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:17.641 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:17.642 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:17.642 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:17.642 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:17.642 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:17.643 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:38:17.643 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:17.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:17.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:17.644 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:17.644 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:17.644 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:17.644 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:17.644 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:17.645 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:17.645 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:17.645 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:38:17.645 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:17.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:17.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:17.646 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:17.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.646 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:17.646 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:38:17.647 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:38:17.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:17.647 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:17Z" 2023-10-17T19:38:17.647 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:17.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:38:17Z" } 2023-10-17T19:38:17.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:38:17.649 DEBUG:rgw_multi.tests:log_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:17.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:17.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:38:17.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.651 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:17.704 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:17.750 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:17.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.752 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:17.806 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:17.807 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:17.853 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:17.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.855 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-74 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-74 --dest-bucket=hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:17.959 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:17.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:17.961 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-75 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-74 --dest-bucket=hbmuog-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:18.016 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.016 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.062 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:18.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-74', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.064 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-74 --dest-bucket=hbmuog-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:18.120 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.120 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.167 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-74","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:38:18.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.168 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:18.219 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.219 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.271 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-74","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:38:18.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.273 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:18.322 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.323 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.378 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:18.378 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:18.378 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 21, 2023-10-17T19:38:18.379 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:38:18.379 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:38:18.379 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:38:18.379 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:18.380 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:38:18.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.380 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:38:18.380 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:38:18.381 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:38:18.381 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:38:18.381 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.381 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:38:18.382 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.382 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:38:18.382 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:38:18.382 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.382 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.383 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.383 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:38:18.383 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.384 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:38:18.384 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.384 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:38:18.384 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:38:18.384 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:38:18.385 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:38:18.385 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:38:18.385 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:38:18.385 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:38:18.385 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:38:18.386 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:38:18.386 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:38:18.386 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.387 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:38:18.387 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:38:18.387 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.388 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:38:18.388 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.388 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:38:18.388 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:38:18.388 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:38:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:38:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:38:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:38:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:38:18.389 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:38:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:38:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:38:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.390 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:38:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.391 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:38:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:38:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:38:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:38:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:38:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:38:18.393 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:38:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:38:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:38:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:38:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:38:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:38:18.395 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:38:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.397 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:38:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:38:18.398 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:38:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:38:18.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:38:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-75", 2023-10-17T19:38:18.400 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:38:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.401 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:38:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:38:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:38:18.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.403 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:38:18.403 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:38:18.403 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:38:18.404 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:38:18.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.405 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-74", 2023-10-17T19:38:18.405 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.405 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:38:18.405 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.406 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:38:18.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.406 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:38:18.407 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:38:18.407 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.407 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:38:18.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.408 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:38:18.408 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.408 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:38:18.408 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:38:18.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.409 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:38:18.409 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:38:18.410 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:38:18.410 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:38:18.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.411 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-75", 2023-10-17T19:38:18.411 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.411 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:38:18.411 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.412 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:38:18.412 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.412 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:38:18.412 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-75", 2023-10-17T19:38:18.413 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.413 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:38:18.413 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.414 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:38:18.414 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:38:18.414 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:38:18.414 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:38:18.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.415 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:38:18.415 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:38:18.415 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:38:18.416 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:38:18.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.416 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.416 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "enabled" 2023-10-17T19:38:18.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.417 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:38:18.418 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.418 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.418 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:38:18.419 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:38:18.419 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:38:18.419 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:38:18.419 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.419 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:38:18.420 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.420 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:38:18.420 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:38:18.420 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:38:18.420 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:38:18.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.421 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:38:18.421 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:38:18.421 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.421 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:38:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:38:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:38:18.422 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:38:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:38:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:38:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:38:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:38:18.423 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:38:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:38:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:38:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:38:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:38:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:38:18.425 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:38:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:38:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:38:18.426 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:38:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:38:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:38:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:38:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:38:18.427 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:38:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:38:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:38:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:38:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.429 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:38:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:38:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:38:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:38:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:38:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:38:18.431 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:38:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:38:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:38:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:38:18.432 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:38:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:38:18.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.434 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:38:18.434 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:38:18.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.434 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.435 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:38:18.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.435 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:38:18.435 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:38:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:38:18.436 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:38:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:38:18.437 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:38:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:38:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:38:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:38:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:38:18.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:38:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:38:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:38:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.439 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:38:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:38:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:38:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:38:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:38:18.440 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:38:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:38:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:38:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:38:18.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:38:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:38:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:38:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.442 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:38:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:38:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:38:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:38:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:38:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:38:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:38:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:38:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:38:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:38:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:38:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:38:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.446 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:38:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:38:18.447 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:18.447 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:18.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:38:18.449 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:38:18.499 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.499 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.561 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-74","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:38:18.561 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:38:18.562 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:38:18.562 DEBUG:boto:Method: HEAD 2023-10-17T19:38:18.563 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:38:18.563 DEBUG:boto:Data: 2023-10-17T19:38:18.564 DEBUG:boto:Headers: {} 2023-10-17T19:38:18.564 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:18.564 DEBUG:boto:Port: 8000 2023-10-17T19:38:18.565 DEBUG:boto:Params: {} 2023-10-17T19:38:18.565 DEBUG:boto:Token: None 2023-10-17T19:38:18.566 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:18 GMT /hbmuog-74/ 2023-10-17T19:38:18.566 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f8qqH3KHvezSjZ/yHozHQcleUWg= 2023-10-17T19:38:18.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f8qqH3KHvezSjZ/yHozHQcleUWg=', 'Content-Length': '0'} 2023-10-17T19:38:18.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004316c5000b653d69-00652ee2aa-2861-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:18.609 DEBUG:boto:path=/hbmuog-74/obj1 2023-10-17T19:38:18.609 DEBUG:boto:auth_path=/hbmuog-74/obj1 2023-10-17T19:38:18.609 DEBUG:boto:Method: PUT 2023-10-17T19:38:18.610 DEBUG:boto:Path: /hbmuog-74/obj1 2023-10-17T19:38:18.610 DEBUG:boto:Data: 2023-10-17T19:38:18.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:38:18.611 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:18.612 DEBUG:boto:Port: 8000 2023-10-17T19:38:18.612 DEBUG:boto:Params: {} 2023-10-17T19:38:18.612 DEBUG:boto:Token: None 2023-10-17T19:38:18.613 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:38:18 GMT /hbmuog-74/obj1 2023-10-17T19:38:18.613 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:YNKcgM4BgOQdhEqBlqJbJD6o1+c= 2023-10-17T19:38:18.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:38:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YNKcgM4BgOQdhEqBlqJbJD6o1+c='} 2023-10-17T19:38:18.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f296d93d50d1c74-00652ee2aa-2861-a1'), ('Date', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:18.635 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:38:18.636 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:38:18.636 DEBUG:boto:Method: HEAD 2023-10-17T19:38:18.637 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:38:18.637 DEBUG:boto:Data: 2023-10-17T19:38:18.637 DEBUG:boto:Headers: {} 2023-10-17T19:38:18.638 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:18.638 DEBUG:boto:Port: 8000 2023-10-17T19:38:18.639 DEBUG:boto:Params: {} 2023-10-17T19:38:18.639 DEBUG:boto:Token: None 2023-10-17T19:38:18.640 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:38:18 GMT /hbmuog-74/ 2023-10-17T19:38:18.640 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:f8qqH3KHvezSjZ/yHozHQcleUWg= 2023-10-17T19:38:18.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f8qqH3KHvezSjZ/yHozHQcleUWg=', 'Content-Length': '0'} 2023-10-17T19:38:18.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f826813eef4869e2-00652ee2aa-2861-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:18.644 DEBUG:boto:path=/hbmuog-74/obj2 2023-10-17T19:38:18.644 DEBUG:boto:auth_path=/hbmuog-74/obj2 2023-10-17T19:38:18.644 DEBUG:boto:Method: PUT 2023-10-17T19:38:18.645 DEBUG:boto:Path: /hbmuog-74/obj2 2023-10-17T19:38:18.645 DEBUG:boto:Data: 2023-10-17T19:38:18.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:38:18.646 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:38:18.647 DEBUG:boto:Port: 8000 2023-10-17T19:38:18.647 DEBUG:boto:Params: {} 2023-10-17T19:38:18.647 DEBUG:boto:Token: None 2023-10-17T19:38:18.648 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:38:18 GMT /hbmuog-74/obj2 2023-10-17T19:38:18.648 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:MTEnNYijmD9QJ6tPPcvmOz99P0E= 2023-10-17T19:38:18.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:38:18 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MTEnNYijmD9QJ6tPPcvmOz99P0E='} 2023-10-17T19:38:18.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4685cb36d9d1a91-00652ee2aa-2861-a1'), ('Date', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:38:18.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:18.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:18.696 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:18.769 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.769 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:18.862 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:38:18.897 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:38:18.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.897 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:38:18.898 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:38:12.301256Z" 2023-10-17T19:38:18.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:18.899 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:38:18.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:18.900 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:38:18.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:18.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:38:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:18.901 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:38:18.902 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:18.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:38:18.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:18.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:18.904 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:18.983 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:18.984 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:19.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:19.046+0000 7efee5f04780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:19.050 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:38:19.048+0000 7efee5f04780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:38:19.076 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:19.076 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:19.077 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:19.077 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:19.077 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:19.077 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:38:19.078 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:19.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.078 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:19.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.079 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:19.079 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.079 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:19.079 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:38:19.079 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.080 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:38:19.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.080 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:38:19.080 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:19.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.081 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:19.081 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.082 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:19.082 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:38:19.082 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.082 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:19.082 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.083 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:38:19.083 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:19.083 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.084 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:19.084 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.084 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:19.084 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:38:19.084 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.085 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:19.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.085 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:38:19.085 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:19.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.086 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:19.086 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.087 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:38:19.087 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:38:19.087 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.087 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:19.087 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.088 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:38:19.088 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:38:19.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.088 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:19.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.089 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:19.089 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:38:19.089 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:38:19.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.090 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:19Z" 2023-10-17T19:38:19.090 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:19.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:38:19Z" } 2023-10-17T19:38:19.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:38:19.092 DEBUG:rgw_multi.tests:log_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:19.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:38:19.093 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:38:19.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:19.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:19.094 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:19.164 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:19.165 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:19.239 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:38:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.239 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.240 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:38:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:38:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:19.241 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:38:19.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:38:19.243 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.243 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:38:19.244 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:19.244 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:38:19.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:19.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:19.246 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:19.305 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:19.305 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:19.353 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:19.353 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:19.353 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:19.354 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:19.354 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:19.354 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:19.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.355 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:19.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.355 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:19.355 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:19.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.357 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:19.358 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.358 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:19.358 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:19.358 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.359 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:19.359 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.359 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:19.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.360 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:19.360 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.361 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:19.362 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.362 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:19.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:19.363 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:19.363 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:19.363 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:19.363 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:19.364 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:19.364 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:19.364 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:19.364 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:19.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:19.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:19.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.366 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:19.366 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:19.366 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:19.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:19.367 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:19Z" 2023-10-17T19:38:19.367 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:19.368 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:19Z" } 2023-10-17T19:38:19.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:19.369 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:19.370 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:19.370 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:24.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:24.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:24.373 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:24.437 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:24.438 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:24.488 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:24Z" 2023-10-17T19:38:24.500 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:24.501 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:24Z" } 2023-10-17T19:38:24.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:24.502 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:24.503 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:24.503 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:29.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:29.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:29.506 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:29.566 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:29.567 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:29.617 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:29.618 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:29.618 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:29.618 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:29.618 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:29.618 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:29.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.619 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:29.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:29.619 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:29.620 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:29.620 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:29.620 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:29.620 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:29.620 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:29.621 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:29.621 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:29.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:29.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:29.622 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:29.622 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:29.622 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:29.622 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:29.623 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:29.623 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:29.623 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:29.625 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:29.625 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:29.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:29.626 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:29.626 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:29.626 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:29.627 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:29.627 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:29.627 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:29.627 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:29.627 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:29.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:29.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:29.628 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:29.629 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:29.629 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:29.629 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:29.629 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:29.629 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:29.630 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:29.630 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:29.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:29.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:29.631 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:29.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.631 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:29.631 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:29.631 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:29.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:29.632 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:29Z" 2023-10-17T19:38:29.632 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:29.633 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:29Z" } 2023-10-17T19:38:29.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:29.634 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:29.635 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:29.635 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:34.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:34.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:34.638 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:34.698 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:34.699 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:34.749 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:34.749 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:34.749 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:34.749 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:34.750 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:34.750 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:34.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.750 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:34.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:34.751 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:34.751 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:34.751 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:34.751 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:34.752 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:34.752 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:34.752 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:34.752 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:34.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:34.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:34.753 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:34.753 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:34.753 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:34.754 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:34.754 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:34.754 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:34.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:34.755 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:34.755 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:34.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:34.755 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:34.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:34.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:34.759 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:34.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:34.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:34.760 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:34.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.760 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:34.760 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:34.760 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:34.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:34.761 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:34Z" 2023-10-17T19:38:34.761 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:34.762 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:34Z" } 2023-10-17T19:38:34.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:34.764 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:34.764 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:34.764 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:39.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:39.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:39.767 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:39.829 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:39.830 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:39.880 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:39.881 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:39.881 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:39.882 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:39.882 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:39.882 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:39.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.883 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:39.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:39.884 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:39.884 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:39.885 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:39.885 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:39.885 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:39.886 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:39.886 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:39.887 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:39.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:39.889 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:39.889 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:39.890 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:39.890 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:39.890 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:39.891 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:39.891 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:39.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:39.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:39.893 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:39.893 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:39.894 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:39.894 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:39.894 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:39.897 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:39.897 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:39.898 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:39.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:39.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:39.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:39.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:39.900 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:39.900 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:39.901 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:39.901 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:39.901 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:39.902 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:39.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:39.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:39.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:39.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.903 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:39.904 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:39.904 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:39.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:39.905 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:39Z" 2023-10-17T19:38:39.905 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:39.907 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:39Z" } 2023-10-17T19:38:39.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:39.909 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:39.909 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:39.910 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:44.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:44.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:44.913 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:44.975 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:44.975 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:45.026 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:45Z" 2023-10-17T19:38:45.039 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:45.040 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:45Z" } 2023-10-17T19:38:45.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:45.041 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:45.042 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:45.042 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:50.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:50.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:50.045 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:50.106 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:50.107 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:50.157 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:50.158 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:50.158 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:50.158 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:50.159 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:50.159 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:50.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.159 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:50.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:50.160 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:50.160 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:50.160 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:50.160 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:50.161 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:50.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:50.161 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:50.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:50.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:50.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:50.162 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:50.163 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:50.163 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:50.163 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:50.163 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:50.164 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:50.164 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:50.164 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:50.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:50.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:50.165 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:50.165 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:50.165 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:50.166 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:50.166 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:50.166 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:50.167 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:50.167 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:50.167 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:50.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:50.168 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:50.168 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:50.168 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:50.168 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:50.169 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:50.169 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:50.169 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:50.169 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:50.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:50.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:50.170 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:50.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.171 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:50.171 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:50.171 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:50.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:50.172 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:50Z" 2023-10-17T19:38:50.172 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:50.173 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:50Z" } 2023-10-17T19:38:50.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:50.174 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:50.174 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:50.175 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:38:55.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:55.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:38:55.177 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:38:55.237 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:38:55.238 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:38:55.288 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:38:55.288 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:38:55.289 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:38:55.289 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:38:55.289 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:38:55.289 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:38:55Z" 2023-10-17T19:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:38:55.303 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:38:55Z" } 2023-10-17T19:38:55.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:38:55.304 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:38:55.304 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:38:55.304 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:00.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:00.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:00.307 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:00.422 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:00Z" 2023-10-17T19:39:00.435 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:00.436 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:00Z" } 2023-10-17T19:39:00.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:00.437 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:00.437 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:00.437 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:05.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:05.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:05.439 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:05.501 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:05.501 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:05.553 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:05Z" 2023-10-17T19:39:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:05.568 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:05Z" } 2023-10-17T19:39:05.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:05.569 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:05.570 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:05.570 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:10.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:10.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:10.572 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:10.634 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:10.635 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:10.686 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:10.686 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.687 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:10.688 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:10.689 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:10.690 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:10.691 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:10.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:10.693 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:10.694 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:10.695 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:10.696 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.697 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:10.698 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:10Z" 2023-10-17T19:39:10.698 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:10.700 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:10Z" } 2023-10-17T19:39:10.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:10.701 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:10.702 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:10.702 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:15.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:15.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:15.706 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:15.818 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:15.820 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:15.820 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:15.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.820 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:15.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:15.821 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:15.821 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:15.821 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:15.821 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:15.821 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:15.822 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:15.822 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:15.822 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:15.822 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:15.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:15.823 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:15.823 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:15.823 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:15.824 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:15.824 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:15.824 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:15.824 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:15.824 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:15.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:15.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:15.825 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:15.825 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:15.826 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:15.826 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:15.826 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:15.826 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:15.827 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:15.827 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:15.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:15.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:15.828 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:15.828 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:15.828 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:15.828 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:15.828 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:15.829 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:15.829 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:15.829 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:15.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:15.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:15.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:15.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.830 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:15.830 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:15.830 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:15.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:15.831 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:15Z" 2023-10-17T19:39:15.831 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:15.832 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:15Z" } 2023-10-17T19:39:15.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:15.833 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:15.834 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:15.834 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:20.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:20.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:20.836 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:20.897 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:20.897 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:20.949 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:20Z" 2023-10-17T19:39:20.964 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:20.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:20Z" } 2023-10-17T19:39:20.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:20.966 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:20.966 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:20.966 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:25.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:25.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:25.969 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:26.031 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:26.031 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:26.083 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:26.084 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:26.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:26.086 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:26.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:26.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:26.088 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:26.088 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:26.089 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:26.089 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:26.089 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:26.089 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:26.089 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:26.090 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:26.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:26.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:26.091 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:26.091 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:26.091 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:26.091 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:26.092 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:26.092 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:26.092 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:26.092 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:26.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:26.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:26.094 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:26.094 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:26.095 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:26.095 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:26.095 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:26.096 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:26.096 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:26.096 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:26.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:26.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:26.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:26.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.098 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:26.098 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:26.098 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:26.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:26.099 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:26Z" 2023-10-17T19:39:26.099 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:26.100 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:26Z" } 2023-10-17T19:39:26.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:26.101 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:26.102 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:26.102 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:31.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:31.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:31.104 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:31.165 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:31.166 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:31.217 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:31.217 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:31.218 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:31.218 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:31.218 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:31.218 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:31.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.219 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:31.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:31.219 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:31.219 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:31.220 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:31.220 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:31.220 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:31.220 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:31.220 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:31.221 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:31.221 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:31.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:31.223 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:31.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:31.223 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:31.223 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:31.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:31.224 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:31.225 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:31.226 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:31.227 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:31.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.229 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:31.229 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:31.229 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:31.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:31.230 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:31Z" 2023-10-17T19:39:31.230 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:31.231 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:31Z" } 2023-10-17T19:39:31.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:31.232 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:31.232 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:31.233 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:36.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:36.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:36.236 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:36.296 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:36.296 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:36.347 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:36.348 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:36.348 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:36.348 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:36.348 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:36.349 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:36.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.349 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:36.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:36.350 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:36.350 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:36.350 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:36.350 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:36.351 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:36.351 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:36.351 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:36.351 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:36.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:36.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:36.352 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:36.352 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:36.353 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:36.353 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:36.353 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:36.354 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:36.354 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:36.354 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:36.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:36.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:36.356 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:36.356 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:36.356 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:36.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:36.357 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:36.357 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:36.357 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:36.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:36.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:36.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:36.358 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:36.359 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:36.359 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:36.359 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:36.359 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:36.359 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:36.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:36.360 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:36.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:36.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:36.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:36.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.362 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:36.362 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:36.362 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:36.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:36.363 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:36Z" 2023-10-17T19:39:36.363 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:36.365 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:36Z" } 2023-10-17T19:39:36.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:36.367 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:36.367 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:36.367 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:41.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:41.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:41.370 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:41.432 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:41.483 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:41.483 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:41.484 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:41.484 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:41.484 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:41.484 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:41.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.485 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:41.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:41.485 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:41.485 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:41.486 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:41.486 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:41.486 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:41.486 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:41.487 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:41.487 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:41.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:41.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:41.488 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:41.488 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:41.488 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:41.488 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:41.488 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:41.489 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:41.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:41.489 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:41.489 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:41.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:41.490 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:41.490 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:41.490 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:41.490 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:41.491 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:41.491 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:41.491 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:41.491 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:41.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:41.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:41.492 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:41.492 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:41.492 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:41.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:41.493 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:41.493 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:41.493 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:41.493 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:41.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:41.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:41.494 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:41.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.494 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:41.495 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:41.495 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:41.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:41.495 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:41Z" 2023-10-17T19:39:41.496 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:41.497 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:41Z" } 2023-10-17T19:39:41.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:41.498 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:41.498 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:41.498 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:46.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:46.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:46.501 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:46.561 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:46.562 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:46.614 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:39:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:39:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:39:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:39:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:46Z" 2023-10-17T19:39:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:46.630 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041262", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:37:50.844371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047416", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:37:50.864535Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:46Z" } 2023-10-17T19:39:46.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:39:46.631 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:46.631 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:39:46.632 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041402 target=00000000000000000000:00000000000000041262, shard=3 master=00000000000000000000:00000000000000047556 target=00000000000000000000:00000000000000047416 2023-10-17T19:39:51.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:51.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:51.634 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:51.695 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:51.696 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:51.748 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:51.748 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:51.748 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:51.749 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:51.749 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:51.749 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:39:51.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.749 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:51.750 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:39:51.751 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:51.751 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:51.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:51.751 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:39:51.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:51.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:51.752 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:51.752 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:51.752 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:39:51.753 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:51.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:51.754 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:51.754 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:51.755 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:51.755 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:39:51.755 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:51.755 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:39:51.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:51.756 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:39:51.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:51.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:51.756 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:51.757 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:51.757 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:39:51.757 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:39:51.757 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:51.757 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:39:51.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:51.758 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:39:51.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:51.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:51.759 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:51.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.759 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:51.759 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:39:51.759 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:39:51.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:51.760 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:51Z" 2023-10-17T19:39:51.760 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:51.761 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047556", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:38:18.632571Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:39:51Z" } 2023-10-17T19:39:51.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047556', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.632571Z'}}] 2023-10-17T19:39:51.762 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:51.762 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:39:51.763 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:39:51.763 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:39:51.764 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:39:51.764 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.765 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:39:51.765 DEBUG:boto:Data: 2023-10-17T19:39:51.766 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.766 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.766 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.767 DEBUG:boto:Params: {} 2023-10-17T19:39:51.768 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:39:51.768 DEBUG:boto:Token: None 2023-10-17T19:39:51.769 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-74/ 2023-10-17T19:39:51.769 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:zGB2WSBlCAVwcCZSwCeV3hgCM2E= 2023-10-17T19:39:51.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zGB2WSBlCAVwcCZSwCeV3hgCM2E=', 'Content-Length': '0'} 2023-10-17T19:39:51.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f614b5e1bb1e920-00652ee307-286d-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.777 DEBUG:boto:path=/hbmuog-74/obj1 2023-10-17T19:39:51.777 DEBUG:boto:auth_path=/hbmuog-74/obj1 2023-10-17T19:39:51.777 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.778 DEBUG:boto:Path: /hbmuog-74/obj1 2023-10-17T19:39:51.778 DEBUG:boto:Data: 2023-10-17T19:39:51.779 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.779 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.780 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.780 DEBUG:boto:Params: {} 2023-10-17T19:39:51.780 DEBUG:boto:Token: None 2023-10-17T19:39:51.781 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-74/obj1 2023-10-17T19:39:51.781 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:RVnx5O74DIBTYjyou033fsUjhO4= 2023-10-17T19:39:51.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RVnx5O74DIBTYjyou033fsUjhO4=', 'Content-Length': '0'} 2023-10-17T19:39:51.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c385423c356d3262-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.785 DEBUG:boto:path=/hbmuog-74/obj1 2023-10-17T19:39:51.785 DEBUG:boto:auth_path=/hbmuog-74/obj1 2023-10-17T19:39:51.786 DEBUG:boto:Method: GET 2023-10-17T19:39:51.786 DEBUG:boto:Path: /hbmuog-74/obj1 2023-10-17T19:39:51.787 DEBUG:boto:Data: 2023-10-17T19:39:51.787 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.788 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.788 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.788 DEBUG:boto:Params: {} 2023-10-17T19:39:51.789 DEBUG:boto:Token: None 2023-10-17T19:39:51.789 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-74/obj1 2023-10-17T19:39:51.790 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:50KCrHCEqkI9QFnCKYhwS8NsygU= 2023-10-17T19:39:51.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:50KCrHCEqkI9QFnCKYhwS8NsygU=', 'Content-Length': '0'} 2023-10-17T19:39:51.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c433eec74a59147-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.793 DEBUG:boto:path=/hbmuog-74/obj2 2023-10-17T19:39:51.794 DEBUG:boto:auth_path=/hbmuog-74/obj2 2023-10-17T19:39:51.794 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.795 DEBUG:boto:Path: /hbmuog-74/obj2 2023-10-17T19:39:51.795 DEBUG:boto:Data: 2023-10-17T19:39:51.796 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.796 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.796 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.797 DEBUG:boto:Params: {} 2023-10-17T19:39:51.797 DEBUG:boto:Token: None 2023-10-17T19:39:51.798 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-74/obj2 2023-10-17T19:39:51.798 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HHkMK29zGVemONp0C2g1qOXDNKs= 2023-10-17T19:39:51.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HHkMK29zGVemONp0C2g1qOXDNKs=', 'Content-Length': '0'} 2023-10-17T19:39:51.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed64e92e4281b6ef-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.801 DEBUG:boto:path=/hbmuog-74/obj2 2023-10-17T19:39:51.802 DEBUG:boto:auth_path=/hbmuog-74/obj2 2023-10-17T19:39:51.802 DEBUG:boto:Method: GET 2023-10-17T19:39:51.803 DEBUG:boto:Path: /hbmuog-74/obj2 2023-10-17T19:39:51.803 DEBUG:boto:Data: 2023-10-17T19:39:51.803 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.804 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.804 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.805 DEBUG:boto:Params: {} 2023-10-17T19:39:51.805 DEBUG:boto:Token: None 2023-10-17T19:39:51.806 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-74/obj2 2023-10-17T19:39:51.806 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZBzOnOmy8f03qtcBQa6Po+JPZi8= 2023-10-17T19:39:51.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZBzOnOmy8f03qtcBQa6Po+JPZi8=', 'Content-Length': '0'} 2023-10-17T19:39:51.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bada633443e75787-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.809 DEBUG:boto:path=/hbmuog-75/ 2023-10-17T19:39:51.810 DEBUG:boto:auth_path=/hbmuog-75/ 2023-10-17T19:39:51.810 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.810 DEBUG:boto:Path: /hbmuog-75/ 2023-10-17T19:39:51.811 DEBUG:boto:Data: 2023-10-17T19:39:51.811 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.812 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.812 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.813 DEBUG:boto:Params: {} 2023-10-17T19:39:51.813 DEBUG:boto:Token: None 2023-10-17T19:39:51.814 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-75/ 2023-10-17T19:39:51.814 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:+6ASgvxi0KFc32T0Bc1+RoZ04lo= 2023-10-17T19:39:51.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+6ASgvxi0KFc32T0Bc1+RoZ04lo=', 'Content-Length': '0'} 2023-10-17T19:39:51.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e02d31696d5cbb6e-00652ee307-286d-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.818 DEBUG:boto:path=/hbmuog-75/obj1 2023-10-17T19:39:51.818 DEBUG:boto:auth_path=/hbmuog-75/obj1 2023-10-17T19:39:51.819 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.819 DEBUG:boto:Path: /hbmuog-75/obj1 2023-10-17T19:39:51.820 DEBUG:boto:Data: 2023-10-17T19:39:51.820 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.820 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.821 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.821 DEBUG:boto:Params: {} 2023-10-17T19:39:51.822 DEBUG:boto:Token: None 2023-10-17T19:39:51.822 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-75/obj1 2023-10-17T19:39:51.822 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:wpxoVKBlD2oLnqvGp6IbWNTXtPo= 2023-10-17T19:39:51.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wpxoVKBlD2oLnqvGp6IbWNTXtPo=', 'Content-Length': '0'} 2023-10-17T19:39:51.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cebab54eccc33a60-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.825 DEBUG:boto:path=/hbmuog-75/obj1 2023-10-17T19:39:51.826 DEBUG:boto:auth_path=/hbmuog-75/obj1 2023-10-17T19:39:51.827 DEBUG:boto:Method: GET 2023-10-17T19:39:51.827 DEBUG:boto:Path: /hbmuog-75/obj1 2023-10-17T19:39:51.827 DEBUG:boto:Data: 2023-10-17T19:39:51.828 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.828 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.829 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.829 DEBUG:boto:Params: {} 2023-10-17T19:39:51.829 DEBUG:boto:Token: None 2023-10-17T19:39:51.830 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-75/obj1 2023-10-17T19:39:51.830 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:HV1HgdnDo74bvQDUDQJ80+K+OSg= 2023-10-17T19:39:51.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HV1HgdnDo74bvQDUDQJ80+K+OSg=', 'Content-Length': '0'} 2023-10-17T19:39:51.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b7ffa15eff7ea37-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.833 DEBUG:boto:path=/hbmuog-75/obj2 2023-10-17T19:39:51.834 DEBUG:boto:auth_path=/hbmuog-75/obj2 2023-10-17T19:39:51.834 DEBUG:boto:Method: HEAD 2023-10-17T19:39:51.835 DEBUG:boto:Path: /hbmuog-75/obj2 2023-10-17T19:39:51.835 DEBUG:boto:Data: 2023-10-17T19:39:51.835 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.836 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.836 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.837 DEBUG:boto:Params: {} 2023-10-17T19:39:51.837 DEBUG:boto:Token: None 2023-10-17T19:39:51.838 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-75/obj2 2023-10-17T19:39:51.838 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:DIczV0cDR+eEH3TNulParKatKsI= 2023-10-17T19:39:51.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DIczV0cDR+eEH3TNulParKatKsI=', 'Content-Length': '0'} 2023-10-17T19:39:51.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f99cd6fbe95ceb04-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.841 DEBUG:boto:path=/hbmuog-75/obj2 2023-10-17T19:39:51.841 DEBUG:boto:auth_path=/hbmuog-75/obj2 2023-10-17T19:39:51.842 DEBUG:boto:Method: GET 2023-10-17T19:39:51.842 DEBUG:boto:Path: /hbmuog-75/obj2 2023-10-17T19:39:51.843 DEBUG:boto:Data: 2023-10-17T19:39:51.843 DEBUG:boto:Headers: {} 2023-10-17T19:39:51.843 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:39:51.844 DEBUG:boto:Port: 8001 2023-10-17T19:39:51.844 DEBUG:boto:Params: {} 2023-10-17T19:39:51.845 DEBUG:boto:Token: None 2023-10-17T19:39:51.845 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:39:51 GMT /hbmuog-75/obj2 2023-10-17T19:39:51.846 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:V2DQY+Y7GTJCF0icwNr8xpKErdg= 2023-10-17T19:39:51.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:39:51 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:V2DQY+Y7GTJCF0icwNr8xpKErdg=', 'Content-Length': '0'} 2023-10-17T19:39:51.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063436edb3339ca5d-00652ee307-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:39:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:39:51.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:51.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:51.850 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:51.902 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:51.903 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:51.949 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:39:51.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:51.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:51.951 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.006 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.037 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: cannot find source zone id for name=* 2023-10-17T19:39:52.037 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: cannot find source zone id for name=* 2023-10-17T19:39:52.054 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:39:52.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.055 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.106 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.106 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.153 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:39:52.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.156 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.205 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.262 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:52.263 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:39:52.263 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 22, 2023-10-17T19:39:52.263 INFO:teuthology.orchestra.run.smithi049.stdout: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:39:52.264 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": [], 2023-10-17T19:39:52.264 INFO:teuthology.orchestra.run.smithi049.stdout: "period_map": { 2023-10-17T19:39:52.264 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:39:52.264 INFO:teuthology.orchestra.run.smithi049.stdout: "zonegroups": [ 2023-10-17T19:39:52.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.265 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "a", 2023-10-17T19:39:52.265 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a", 2023-10-17T19:39:52.265 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:39:52.265 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": true, 2023-10-17T19:39:52.266 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.266 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:39:52.266 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.266 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:39:52.266 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:39:52.267 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:39:52.267 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.267 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:39:52.268 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a1", 2023-10-17T19:39:52.268 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.268 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:39:52.268 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.268 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:39:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:39:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:39:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:39:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:39:52.269 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:39:52.270 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:39:52.270 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:39:52.270 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:39:52.270 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:39:52.271 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.271 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.271 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:39:52.272 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "a2", 2023-10-17T19:39:52.272 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.272 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:39:52.272 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.272 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:39:52.273 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:39:52.273 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:39:52.273 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:39:52.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:39:52.273 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:39:52.274 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:39:52.274 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:39:52.274 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:39:52.274 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:39:52.275 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.275 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.275 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:39:52.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.276 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:39:52.276 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:39:52.276 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:39:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:39:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:39:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:39:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:39:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [ 2023-10-17T19:39:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.279 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-group", 2023-10-17T19:39:52.279 INFO:teuthology.orchestra.run.smithi049.stdout: "data_flow": { 2023-10-17T19:39:52.279 INFO:teuthology.orchestra.run.smithi049.stdout: "symmetrical": [ 2023-10-17T19:39:52.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.281 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-flow1", 2023-10-17T19:39:52.282 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.282 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:39:52.282 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:39:52.282 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.283 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pipes": [ 2023-10-17T19:39:52.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.284 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-74", 2023-10-17T19:39:52.284 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.284 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:39:52.284 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.284 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:39:52.285 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.285 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:39:52.285 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:39:52.285 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.286 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:39:52.286 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.286 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:39:52.287 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.287 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:39:52.287 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:39:52.287 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.288 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:39:52.288 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:39:52.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:39:52.288 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:39:52.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.289 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe-hbmuog-75", 2023-10-17T19:39:52.289 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.290 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-74", 2023-10-17T19:39:52.290 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.290 INFO:teuthology.orchestra.run.smithi049.stdout: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:39:52.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.291 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:39:52.291 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "hbmuog-75", 2023-10-17T19:39:52.291 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.291 INFO:teuthology.orchestra.run.smithi049.stdout: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:39:52.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.292 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:39:52.292 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.293 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:39:52.293 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:39:52.293 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.293 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:39:52.294 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:39:52.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:39:52.294 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:39:52.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.295 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "sync-pipe", 2023-10-17T19:39:52.295 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.295 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:39:52.296 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.296 INFO:teuthology.orchestra.run.smithi049.stdout: "*" 2023-10-17T19:39:52.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.296 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": { 2023-10-17T19:39:52.297 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket": "*", 2023-10-17T19:39:52.297 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.297 INFO:teuthology.orchestra.run.smithi049.stdout: "*" 2023-10-17T19:39:52.297 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.298 INFO:teuthology.orchestra.run.smithi049.stdout: "params": { 2023-10-17T19:39:52.298 INFO:teuthology.orchestra.run.smithi049.stdout: "source": { 2023-10-17T19:39:52.298 INFO:teuthology.orchestra.run.smithi049.stdout: "filter": { 2023-10-17T19:39:52.298 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [] 2023-10-17T19:39:52.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.299 INFO:teuthology.orchestra.run.smithi049.stdout: "dest": {}, 2023-10-17T19:39:52.299 INFO:teuthology.orchestra.run.smithi049.stdout: "priority": 0, 2023-10-17T19:39:52.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "system", 2023-10-17T19:39:52.300 INFO:teuthology.orchestra.run.smithi049.stdout: "user": "" 2023-10-17T19:39:52.300 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.300 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.300 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "allowed" 2023-10-17T19:39:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.301 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:39:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "b", 2023-10-17T19:39:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b", 2023-10-17T19:39:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "api_name": "", 2023-10-17T19:39:52.303 INFO:teuthology.orchestra.run.smithi049.stdout: "is_master": false, 2023-10-17T19:39:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:39:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames": [], 2023-10-17T19:39:52.304 INFO:teuthology.orchestra.run.smithi049.stdout: "hostnames_s3website": [], 2023-10-17T19:39:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:39:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "zones": [ 2023-10-17T19:39:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.305 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:39:52.306 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b2", 2023-10-17T19:39:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:39:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.307 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:39:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:39:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:39:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:39:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:39:52.308 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:39:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:39:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:39:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:39:52.309 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:39:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.310 INFO:teuthology.orchestra.run.smithi049.stdout: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:39:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "b1", 2023-10-17T19:39:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "endpoints": [ 2023-10-17T19:39:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:39:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.311 INFO:teuthology.orchestra.run.smithi049.stdout: "log_meta": false, 2023-10-17T19:39:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "log_data": true, 2023-10-17T19:39:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_index_max_shards": 11, 2023-10-17T19:39:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "read_only": false, 2023-10-17T19:39:52.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_type": "", 2023-10-17T19:39:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from_all": true, 2023-10-17T19:39:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_from": [], 2023-10-17T19:39:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_zone": "", 2023-10-17T19:39:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "supported_features": [ 2023-10-17T19:39:52.313 INFO:teuthology.orchestra.run.smithi049.stdout: "compress-encrypted", 2023-10-17T19:39:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.314 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "placement_targets": [ 2023-10-17T19:39:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default-placement", 2023-10-17T19:39:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2023-10-17T19:39:52.315 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_classes": [ 2023-10-17T19:39:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "STANDARD" 2023-10-17T19:39:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.316 INFO:teuthology.orchestra.run.smithi049.stdout: "default_placement": "default-placement", 2023-10-17T19:39:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:39:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_policy": { 2023-10-17T19:39:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "groups": [] 2023-10-17T19:39:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.317 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled_features": [ 2023-10-17T19:39:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: "resharding" 2023-10-17T19:39:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.318 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2023-10-17T19:39:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: "short_zone_ids": [ 2023-10-17T19:39:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:39:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3843991460 2023-10-17T19:39:52.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:39:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1846350003 2023-10-17T19:39:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:39:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 3451171648 2023-10-17T19:39:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:52.321 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:39:52.322 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 1989104912 2023-10-17T19:39:52.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.322 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:52.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.323 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zonegroup": "a", 2023-10-17T19:39:52.323 INFO:teuthology.orchestra.run.smithi049.stdout: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:39:52.323 INFO:teuthology.orchestra.run.smithi049.stdout: "period_config": { 2023-10-17T19:39:52.323 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_quota": { 2023-10-17T19:39:52.324 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:39:52.324 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:39:52.324 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:39:52.324 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:39:52.324 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:39:52.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.325 INFO:teuthology.orchestra.run.smithi049.stdout: "user_quota": { 2023-10-17T19:39:52.325 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false, 2023-10-17T19:39:52.325 INFO:teuthology.orchestra.run.smithi049.stdout: "check_on_raw": false, 2023-10-17T19:39:52.325 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size": -1, 2023-10-17T19:39:52.326 INFO:teuthology.orchestra.run.smithi049.stdout: "max_size_kb": 0, 2023-10-17T19:39:52.326 INFO:teuthology.orchestra.run.smithi049.stdout: "max_objects": -1 2023-10-17T19:39:52.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.326 INFO:teuthology.orchestra.run.smithi049.stdout: "user_ratelimit": { 2023-10-17T19:39:52.326 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:39:52.327 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:39:52.327 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:39:52.327 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:39:52.327 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:39:52.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.328 INFO:teuthology.orchestra.run.smithi049.stdout: "bucket_ratelimit": { 2023-10-17T19:39:52.328 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:39:52.328 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:39:52.328 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:39:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:39:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:39:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: "anonymous_ratelimit": { 2023-10-17T19:39:52.329 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_ops": 0, 2023-10-17T19:39:52.330 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_ops": 0, 2023-10-17T19:39:52.330 INFO:teuthology.orchestra.run.smithi049.stdout: "max_read_bytes": 0, 2023-10-17T19:39:52.330 INFO:teuthology.orchestra.run.smithi049.stdout: "max_write_bytes": 0, 2023-10-17T19:39:52.330 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": false 2023-10-17T19:39:52.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:52.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:52.331 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:39:52.331 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_name": "test-realm", 2023-10-17T19:39:52.331 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:52.331 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:52.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.333 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.385 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.386 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.462 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:39:52.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.463 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.537 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.669 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:39:52.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.670 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.753 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.754 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.830 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:39:52.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-74', '--source-bucket=*', '--dest-bucket=hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-74', '--source-bucket=*', '--dest-bucket=hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.831 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-74 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-74 '--source-bucket=*' --dest-bucket=hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:52.908 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:52.908 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:52.987 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:39:52.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-74', '--source-bucket=*', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-74', '--source-bucket=*', '--dest-bucket=hbmuog-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:52.989 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-75 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-74 '--source-bucket=*' --dest-bucket=hbmuog-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:53.104 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:39:53.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:53.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:53.105 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:53.158 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:53.158 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:53.208 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:39:53.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:53.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:39:53.210 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:39:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:53.257 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:53.302 INFO:teuthology.orchestra.run.smithi049.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-74","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-75","source":{"bucket":"hbmuog-74","zones":["a1"]},"dest":{"bucket":"hbmuog-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:39:53.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:53.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:53.303 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:53.354 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:53.354 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:53.387 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:39:53.401 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:39:53.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.402 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571593.193690_1189.1", 2023-10-17T19:39:53.402 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:39:53.193690Z" 2023-10-17T19:39:53.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.403 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:39:53.403 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:39:53.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.403 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:39:53.404 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:39:53.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.404 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:39:53.404 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:39:53.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.405 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:39:53.406 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:39:53.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:39:53.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:53.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:53.408 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:53.458 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:53.458 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:53.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:39:53.486+0000 7f3b58b75780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:39:53.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:39:53.487+0000 7f3b58b75780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:39:53.502 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:53.502 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:39:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:53.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:53.504 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:39:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:39:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:53.506 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:39:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:53.507 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:39:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.508 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:53.509 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:53.509 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:53.509 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:39:53.509 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:53.509 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:53.510 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:53.510 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:39:53.510 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:53.510 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:53.513 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:53.513 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:53.513 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:53.514 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:39:53.514 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:53.514 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:53.514 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:53.514 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:39:53.515 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:53.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:53.515 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:53.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.516 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:53.516 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:39:53.516 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:39:53.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:53.516 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:53Z" 2023-10-17T19:39:53.517 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:53.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:39:53Z" } 2023-10-17T19:39:53.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:39:53.518 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:39:53.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:39:53.519 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:39:58.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:58.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:39:58.522 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:39:58.581 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:39:58.582 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:39:58.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:39:58.617+0000 7f2808778780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:39:58.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:39:58.618+0000 7f2808778780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:39:58.635 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:39:58.635 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:39:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:39:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:39:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:39:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:39:58.636 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:39:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:39:58.637 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:39:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:39:58.638 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:39:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:39:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:39:58.640 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:39:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:58.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:58.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:58.642 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:39:58.642 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:58.643 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:39:58.644 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:58.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:58.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:39:58.644 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:39:58.645 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:39:58.645 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:39:58.645 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:39:58.645 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:39:58.645 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:39:58.646 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:39:58.646 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:39:58.646 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:39:58.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:58.647 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:39:58.647 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:39:58.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.647 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:39:58.647 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:39:58.648 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:39:58.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:39:58.648 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:39:58Z" 2023-10-17T19:39:58.648 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:39:58.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:39:58Z" } 2023-10-17T19:39:58.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:39:58.650 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:39:58.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:39:58.651 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:03.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:03.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:03.654 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:03.713 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:03.714 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:03.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:03.746+0000 7fa30f836780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:03.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:03.747+0000 7fa30f836780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:03.763 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:03.763 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:03.764 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:03.765 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:03.766 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:03.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:03.768 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:03.769 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:03.771 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:03.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:03.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:03.773 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:03.773 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:03.774 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:03.774 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:03.774 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:03.774 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:03.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:03.775 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:03.775 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:03.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:03.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:03.776 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:03.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.777 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:03.777 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:03.777 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:03.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:03.778 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:03Z" 2023-10-17T19:40:03.778 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:03.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:03Z" } 2023-10-17T19:40:03.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:03.780 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:03.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:03.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:08.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:08.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:08.784 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:08.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:08.880+0000 7fcc9d76f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:08.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:08.881+0000 7fcc9d76f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:08.896 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:08.896 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:08.897 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:08.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:08.899 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:08.901 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:08.902 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:08.902 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:08.902 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:08.902 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:08.903 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:08.903 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:08.903 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:08.903 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:08.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:08.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:08.904 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:08.904 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:08.905 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:08.905 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:08.905 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:08.905 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:08.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:08.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.910 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:08.910 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:08.911 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:08.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:08.911 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:08Z" 2023-10-17T19:40:08.911 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:08.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:08Z" } 2023-10-17T19:40:08.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:08.914 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:08.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:08.915 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:13.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:13.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:13.918 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:14.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:14.010+0000 7f91d13c0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:14.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:14.011+0000 7f91d13c0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:14.026 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:14.026 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:14.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:14.028 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:14.029 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:14.031 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:14.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:14.034 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:14.034 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:14.034 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:14.034 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:14.034 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:14.035 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:14.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:14.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:14.035 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:14.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:14.037 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:14.037 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:14.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:14.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:14.037 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:14Z" 2023-10-17T19:40:14.039 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:14.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:14Z" } 2023-10-17T19:40:14.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:14.041 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:14.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:14.042 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:19.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:19.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:19.044 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:19.103 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:19.104 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:19.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:19.137+0000 7f15b1cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:19.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:19.138+0000 7f15b1cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:19.153 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:19.153 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.154 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:19.155 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:19.156 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:19.157 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:19.158 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:19.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:19.160 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:19.161 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:19.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:19.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:19.162 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:19.162 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:19.163 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:19.164 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:19.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:19.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:19.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:19.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.165 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:19.165 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:19.165 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:19.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:19.165 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:19Z" 2023-10-17T19:40:19.166 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:19.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:19Z" } 2023-10-17T19:40:19.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:19.168 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:19.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:19.168 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:24.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:24.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:24.171 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:24.230 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:24.230 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:24.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:24.264+0000 7fdaea8c2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:24.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:24.265+0000 7fdaea8c2780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:24.281 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:24.281 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:24.281 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:24.282 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:24.283 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:24.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:24.286 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:24.287 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:24.288 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:24.289 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:24.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:24.291 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:24.292 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:24.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:24.294 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:24Z" 2023-10-17T19:40:24.295 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:24.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:24Z" } 2023-10-17T19:40:24.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:24.297 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:24.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:24.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:29.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:29.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:29.300 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:29.360 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:29.361 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:29.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:29.393+0000 7fdbafe44780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:29.396 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:29.394+0000 7fdbafe44780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:29.410 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:29.410 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:29.410 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:29.410 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:29.411 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:29.411 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:29.411 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:29.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.412 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:29.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:29.412 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:29.412 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:29.413 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:29.413 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:29.413 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:29.413 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:29.413 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:29.414 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:29.414 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:29.414 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:29.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:29.415 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:29.415 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:29.415 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:29.416 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:29.416 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:29.416 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:29.416 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:29.416 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:29.417 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:29.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:29.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:29.418 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:29.418 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:29.418 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:29.418 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:29.419 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:29.419 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:29.419 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:29.419 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:29.419 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:29.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:29.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:29.420 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:29.421 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:29.421 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:29.421 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:29.421 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:29.421 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:29.422 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:29.422 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:29.422 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:29.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:29.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:29.423 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:29.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.423 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:29.423 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:29.424 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:29.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:29.424 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:29Z" 2023-10-17T19:40:29.424 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:29.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:29Z" } 2023-10-17T19:40:29.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:29.426 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:29.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:29.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:34.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:34.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:34.429 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:34.486 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:34.486 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:34.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:34.518+0000 7f0b06ba0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:34.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:34.519+0000 7f0b06ba0780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:34.535 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:34.535 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:34.535 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:34.536 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:34.536 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:34.536 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:34.536 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:34.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.537 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:34.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:34.537 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:34.537 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:34.538 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:34.538 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:34.538 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:34.538 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:34.539 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:34.539 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:34.539 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:34.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:34.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:34.540 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:34.540 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:34.540 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:34.540 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:34.541 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:34.541 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:34.541 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:34.541 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:34.542 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:34.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:34.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:34.542 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:34.543 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:34.543 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:34.543 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:34.543 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:34.543 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:34.544 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:34.544 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:34.544 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:34.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:34.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:34.545 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:34.545 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:34.545 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:34.545 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:34.546 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:34.546 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:34.546 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:34.546 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:34.547 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:34.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:34.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:34.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:34.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.548 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:34.548 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:34.548 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:34.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:34.549 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:34Z" 2023-10-17T19:40:34.549 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:34.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:34Z" } 2023-10-17T19:40:34.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:34.551 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:34.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:34.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:39.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:39.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:39.555 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:39.614 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:39.615 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:39.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:39.649+0000 7f5fc7cce780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:39.652 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:39.651+0000 7f5fc7cce780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:39.667 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:39.668 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:39.668 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:39.668 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:39.668 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:39.671 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:39.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:39.673 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:39.674 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:39.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:39.676 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:39.677 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:39.678 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:39.680 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:39.681 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:39.682 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:39.683 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:39.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.685 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:39.686 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:39.687 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:39Z" 2023-10-17T19:40:39.688 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:39.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:39Z" } 2023-10-17T19:40:39.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:39.691 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:39.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:39.692 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:44.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:44.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:44.695 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:44.754 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:44.754 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:44.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:44.788+0000 7f1d2f0ae780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:44.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:44.789+0000 7f1d2f0ae780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:44.804 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:44.805 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:44.805 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:44.805 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:44.806 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:44.806 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:44.806 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:44.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.807 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:44.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:44.807 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:44.807 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:44.808 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:44.808 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:44.808 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:44.808 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:44.808 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:44.809 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:44.809 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:44.809 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:44.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:44.810 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:44.810 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:44.810 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:44.810 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:44.811 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:44.811 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:44.811 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:44.811 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:44.811 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:44.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:44.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:44.812 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:44.813 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:44.813 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:44.813 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:44.813 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:44.814 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:44.814 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:44.814 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:44.814 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:44.814 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:44.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:44.815 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:44.815 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:44.816 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:44.816 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:44.816 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:44.816 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:44.816 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:44.817 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:44.817 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:44.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:44.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:44.817 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:44.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.818 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:44.818 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:44.818 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:44.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:44.819 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:44Z" 2023-10-17T19:40:44.819 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:44.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:44Z" } 2023-10-17T19:40:44.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:44.821 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:44.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:44.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:49.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:49.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:49.824 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:49.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:49.917+0000 7f8f8ae68780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:49.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:49.918+0000 7f8f8ae68780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:49.933 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:49.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:49.933 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:49.933 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:49.934 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:49.934 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:49.934 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:49.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.935 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:49.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:49.935 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:49.935 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:49.935 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:49.936 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:49.936 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:49.936 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:49.936 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:49.937 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:49.937 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:49.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:49.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:49.938 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:49.938 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:49.938 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:49.938 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:49.938 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:49.939 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:49.939 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:49.939 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:49.939 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:49.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:49.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:49.940 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:49.940 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:49.941 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:49.941 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:49.941 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:49.941 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:49.941 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:49.942 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:49.942 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:49.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:49.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:49.943 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:49.943 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:49.943 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:49.943 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:49.944 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:49.944 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:49.944 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:49.944 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:49.944 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:49.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:49.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:49.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:49.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.945 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:49.946 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:49.946 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:49.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:49.946 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:49Z" 2023-10-17T19:40:49.947 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:49.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:49Z" } 2023-10-17T19:40:49.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:49.949 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:49.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:49.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:40:54.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:54.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:40:54.951 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:40:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:40:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:40:55.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:55.043+0000 7f0f88533780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:55.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:40:55.044+0000 7f0f88533780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:40:55.059 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:40:55.059 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:40:55.059 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:40:55.060 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:40:55.060 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:40:55.060 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:40:55.060 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:55.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.061 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:40:55.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:55.061 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:40:55.061 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:55.062 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:55.062 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:40:55.062 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:55.062 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:40:55.062 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:55.063 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:40:55.063 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:55.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:55.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:55.064 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:40:55.064 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:55.064 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:55.064 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:40:55.065 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:55.065 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:55.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:55.065 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:40:55.066 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:55.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:55.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:55.067 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:40:55.067 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:55.067 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:55.068 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:40:55.068 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:55.068 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:55.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:55.069 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:40:55.069 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:55.070 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:55.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:40:55.071 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:40:55.071 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:40:55.072 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:40:55.072 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:40:55.072 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:40:55.073 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:40:55.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:40:55.073 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:40:55.074 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:40:55.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:55.074 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:40:55.074 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:40:55.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.075 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:40:55.075 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:40:55.075 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:40:55.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:40:55.076 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:40:55Z" 2023-10-17T19:40:55.076 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:40:55.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:40:55Z" } 2023-10-17T19:40:55.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:40:55.078 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:55.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:40:55.079 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:00.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:00.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:00.081 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:00.140 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:00.141 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:00.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:00.171+0000 7fe5b2ece780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:00.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:00.172+0000 7fe5b2ece780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:00.189 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:00.189 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:00.189 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:00.189 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:00.190 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:00.190 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:00.190 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:00.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.190 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:00.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:00.191 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:00.191 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:00.191 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:00.192 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:00.192 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:00.192 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:00.192 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:00.192 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:00.193 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:00.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:00.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:00.194 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:00.194 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:00.194 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:00.194 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:00.194 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:00.195 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:00.195 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:00.195 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:00.195 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:00.196 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:00.197 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:00.197 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:00.197 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:00.197 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:00.197 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:00.198 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:00.198 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:00.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:00.198 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:00.199 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:00.199 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:00.199 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:00.199 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:00.199 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:00.200 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:00.200 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:00.200 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:00.200 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:00.201 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:00.201 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:00.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.201 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:00.201 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:00.202 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:00.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:00.202 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:00Z" 2023-10-17T19:41:00.202 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:00.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:00Z" } 2023-10-17T19:41:00.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:00.204 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:00.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:00.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:05.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:05.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:05.212 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:05.273 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:05.273 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:05.307 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:05.305+0000 7f0c18c7f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:05.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:05.306+0000 7f0c18c7f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:05.322 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:05.323 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:05.323 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:05.324 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:05.324 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:05.325 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:05.325 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:05.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.325 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:05.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:05.327 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:05.328 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:05.328 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:05.328 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:05.329 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:05.330 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:05.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:05.331 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:05.331 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:05.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:05.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:05.332 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:05.332 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:05.332 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:05.332 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:05.333 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:05.333 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:05.333 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:05.333 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:05.333 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:05.334 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:05.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:05.334 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:05.335 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:05.335 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:05.335 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:05.335 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:05.335 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:05.337 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:05.337 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:05.337 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:05.337 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:05.337 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:05.339 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:05.339 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:05.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.339 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:05.339 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:05.340 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:05.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:05.340 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:05Z" 2023-10-17T19:41:05.340 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:05.342 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:05Z" } 2023-10-17T19:41:05.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:05.343 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:05.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:05.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:10.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:10.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:10.346 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:10.406 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:10.406 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:10.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:10.439+0000 7faf8910c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:10.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:10.441+0000 7faf8910c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:10.457 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:10Z" 2023-10-17T19:41:10.470 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:10.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:10Z" } 2023-10-17T19:41:10.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:10.472 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:10.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:10.473 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:15.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:15.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:15.475 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:15.534 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:15.535 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:15.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:15.568+0000 7f69a17ab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:15.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:15.569+0000 7f69a17ab780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:15.585 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:15.585 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:15.586 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:15.586 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:15.586 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:15.586 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:15.587 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:15.588 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:15.588 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:15.588 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:15.588 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:15.588 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:15.589 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:15.589 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:15.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:15.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:15.590 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:15.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:15.591 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:15.591 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:15.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:15.592 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:15.593 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:15.593 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:15.593 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:15.593 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:15.593 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:15.594 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:15.595 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:15.595 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:15.595 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:15.595 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:15.595 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.596 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:15Z" 2023-10-17T19:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:15.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:15Z" } 2023-10-17T19:41:15.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:15.599 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:15.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:15.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:20.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:20.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:20.602 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:20.661 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:20.661 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:20.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:20.695+0000 7f761b316780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:20.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:20.696+0000 7f761b316780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:20.711 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:20.712 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:20.716 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:20.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:20.718 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:20.719 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:20.720 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:20.721 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:20.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:20Z" 2023-10-17T19:41:20.723 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:20.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:20Z" } 2023-10-17T19:41:20.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:20.725 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:20.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:20.726 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:25.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:25.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:25.728 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:25.787 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:25.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:25.819+0000 7f92a1059780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:25.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:25.820+0000 7f92a1059780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:25.835 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:25.835 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:25.836 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:25.836 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:25.836 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:25.836 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:25.836 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:25.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.837 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:25.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:25.837 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:25.837 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:25.838 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:25.838 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:25.838 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:25.838 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:25.838 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:25.839 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:25.839 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:25.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:25.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:25.840 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:25.840 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:25.840 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:25.840 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:25.840 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:25.841 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:25.841 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:25.841 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:25.841 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:25.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:25.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:25.842 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:25.842 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:25.842 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:25.843 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:25.844 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:25.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:25.844 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:25.844 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:25.845 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:25.846 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:25.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:25.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:25.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:25.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:25Z" 2023-10-17T19:41:25.848 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:25.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:25Z" } 2023-10-17T19:41:25.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:25.849 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:25.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:25.850 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:30.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:30.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:30.852 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:30.912 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:30.912 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:30.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:30.946+0000 7f7bcb051780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:30.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:30.947+0000 7f7bcb051780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:30.962 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:30.962 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:30.963 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:30.963 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:30.963 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:30.963 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:30.964 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:30.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.964 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:30.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:30.965 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:30.965 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:30.965 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:30.966 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:30.966 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:30.966 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:30.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:30.967 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:30.967 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:30.967 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:30.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:30.968 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:30.968 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:30.969 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:30.969 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:30.969 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:30.969 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:30.970 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:30.970 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:30.970 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:30.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:30.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:30.971 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:30.971 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:30.972 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:30.972 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:30.972 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:30.972 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:30.973 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:30.973 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:30.973 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:30.973 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:30.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:30.974 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:30.975 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:30.975 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:30.975 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:30.975 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:30.976 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:30.976 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:30.976 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:30.976 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:30.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:30.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:30.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:30.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.978 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:30.978 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:30.978 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:30.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:30.979 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:30Z" 2023-10-17T19:41:30.979 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:30.980 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:30Z" } 2023-10-17T19:41:30.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:30.981 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:30.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:30.982 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:35.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:35.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:35.985 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:36.042 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:36.043 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:36.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:36.074+0000 7fe3b5635780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:36.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:36.075+0000 7fe3b5635780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:36.091 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:36.097 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:36.097 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:36.097 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:36.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:36.098 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:36.098 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:36.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:36.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:36.099 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:36.099 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:36.099 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:36.099 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:36.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:36.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:36.101 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:36.101 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:36.102 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:36.102 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:36.102 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:36.102 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:36.102 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:36.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.104 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:36.104 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:36.104 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:36.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:36.105 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:36Z" 2023-10-17T19:41:36.105 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:36.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:36Z" } 2023-10-17T19:41:36.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:36.107 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:36.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:36.108 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:41.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:41.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:41.110 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:41.170 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:41.170 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:41.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:41.203+0000 7ff949524780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:41.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:41.204+0000 7ff949524780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:41.220 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:41.220 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:41.220 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:41.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:41.222 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:41.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:41.225 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:41.226 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:41.227 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:41.228 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:41.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:41.230 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:41Z" 2023-10-17T19:41:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:41.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:41Z" } 2023-10-17T19:41:41.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:41.236 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:41.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:41.236 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:46.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:46.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:46.239 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:46.300 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:46.301 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:46.333+0000 7f63e569b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:46.334+0000 7f63e569b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:46.351 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:46Z" 2023-10-17T19:41:46.365 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:46.366 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:46Z" } 2023-10-17T19:41:46.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:46.367 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:46.367 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:46.368 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:51.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:51.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:51.370 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:51.431 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:51.432 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:51.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:51.464+0000 7f3500289780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:51.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:51.465+0000 7f3500289780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:51.482 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:51.482 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:51.483 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:51.483 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:51.483 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:51.483 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:51.484 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:51.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.484 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:51.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:51.485 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:51.485 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:51.485 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:51.485 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571492.301256_1144.1", 2023-10-17T19:41:51.485 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:51.486 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:51.486 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:51.486 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:12.301256Z", 2023-10-17T19:41:51.486 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:51.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:51.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:51.487 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:51.488 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:51.488 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:51.488 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:51.488 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:51.488 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:51.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:51.489 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:51.489 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:51.489 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:51.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:51.490 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:51.490 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:51.490 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:51.490 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:51.491 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:51.491 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:51.491 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:51.491 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:51.492 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:51.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:51.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:51.493 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:51.493 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:51.493 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:51.493 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:51.493 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:51.494 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:51.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:51.494 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:51.494 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:51.495 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:51.496 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:51.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:51.496 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:51Z" 2023-10-17T19:41:51.496 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:51.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571492.301256_1144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:38:12.301256Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:51Z" } 2023-10-17T19:41:51.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571492.301256_1144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:38:12.301256Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:51.498 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:51.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571492.301256_1144.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:51.499 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697571593.193690_1189.1 target=1_1697571492.301256_1144.1 2023-10-17T19:41:56.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.501 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:56.560 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:56.560 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:56.594+0000 7fa95ddc1780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:56.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:56.595+0000 7fa95ddc1780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:56.609 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:56.610 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:56.610 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:56.610 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:56.610 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:56.611 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:56.611 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.611 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:56.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.612 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:56.612 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.612 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.612 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571593.193690_1189.1", 2023-10-17T19:41:56.613 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.613 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:56.613 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.613 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:39:53.193690Z", 2023-10-17T19:41:56.613 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.614 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:56.615 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.615 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.615 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:56.615 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.615 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.616 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.616 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:56.616 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.617 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:56.617 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.617 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.621 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.621 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:56.621 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:56.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.622 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:56.622 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:56.623 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:56.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.623 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:56Z" 2023-10-17T19:41:56.623 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:56.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571593.193690_1189.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:39:53.193690Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:56Z" } 2023-10-17T19:41:56.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571593.193690_1189.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:39:53.193690Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:56.625 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:56.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:56.626 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:41:56.626 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:41:56.627 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:41:56.627 DEBUG:boto:Method: HEAD 2023-10-17T19:41:56.628 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:41:56.628 DEBUG:boto:Data: 2023-10-17T19:41:56.629 DEBUG:boto:Headers: {} 2023-10-17T19:41:56.629 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:41:56.630 DEBUG:boto:Port: 8000 2023-10-17T19:41:56.630 DEBUG:boto:Params: {} 2023-10-17T19:41:56.631 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:41:56.631 DEBUG:boto:Token: None 2023-10-17T19:41:56.632 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:41:56 GMT /hbmuog-74/ 2023-10-17T19:41:56.632 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nXJUnHAJzQiHKr72RyKz0TGCcaU= 2023-10-17T19:41:56.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:41:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nXJUnHAJzQiHKr72RyKz0TGCcaU=', 'Content-Length': '0'} 2023-10-17T19:41:56.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aff2b7020fd1ae3e-00652ee384-294c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:41:56.639 DEBUG:boto:path=/hbmuog-74/obj3 2023-10-17T19:41:56.640 DEBUG:boto:auth_path=/hbmuog-74/obj3 2023-10-17T19:41:56.640 DEBUG:boto:Method: PUT 2023-10-17T19:41:56.641 DEBUG:boto:Path: /hbmuog-74/obj3 2023-10-17T19:41:56.641 DEBUG:boto:Data: 2023-10-17T19:41:56.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:41:56.642 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:41:56.642 DEBUG:boto:Port: 8000 2023-10-17T19:41:56.643 DEBUG:boto:Params: {} 2023-10-17T19:41:56.643 DEBUG:boto:Token: None 2023-10-17T19:41:56.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:41:56 GMT /hbmuog-74/obj3 2023-10-17T19:41:56.644 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:JmntVqankqSiu0a3NQvvz3kSfOA= 2023-10-17T19:41:56.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:41:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JmntVqankqSiu0a3NQvvz3kSfOA='} 2023-10-17T19:41:56.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006c579df0272a00e-00652ee384-294c-a1'), ('Date', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:41:56.652 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:41:56.652 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:41:56.653 DEBUG:boto:Method: HEAD 2023-10-17T19:41:56.653 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:41:56.654 DEBUG:boto:Data: 2023-10-17T19:41:56.654 DEBUG:boto:Headers: {} 2023-10-17T19:41:56.655 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:41:56.655 DEBUG:boto:Port: 8000 2023-10-17T19:41:56.655 DEBUG:boto:Params: {} 2023-10-17T19:41:56.656 DEBUG:boto:Token: None 2023-10-17T19:41:56.656 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:41:56 GMT /hbmuog-74/ 2023-10-17T19:41:56.657 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:nXJUnHAJzQiHKr72RyKz0TGCcaU= 2023-10-17T19:41:56.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:41:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nXJUnHAJzQiHKr72RyKz0TGCcaU=', 'Content-Length': '0'} 2023-10-17T19:41:56.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002a448a2d4a86ad4-00652ee384-294c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:41:56.660 DEBUG:boto:path=/hbmuog-74/obj4 2023-10-17T19:41:56.660 DEBUG:boto:auth_path=/hbmuog-74/obj4 2023-10-17T19:41:56.661 DEBUG:boto:Method: PUT 2023-10-17T19:41:56.661 DEBUG:boto:Path: /hbmuog-74/obj4 2023-10-17T19:41:56.661 DEBUG:boto:Data: 2023-10-17T19:41:56.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:41:56.662 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:41:56.663 DEBUG:boto:Port: 8000 2023-10-17T19:41:56.663 DEBUG:boto:Params: {} 2023-10-17T19:41:56.664 DEBUG:boto:Token: None 2023-10-17T19:41:56.664 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 17 Oct 2023 19:41:56 GMT /hbmuog-74/obj4 2023-10-17T19:41:56.665 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:E0vd/cdF06b8VR5LZncf8I+MEwU= 2023-10-17T19:41:56.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:41:56 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:E0vd/cdF06b8VR5LZncf8I+MEwU='} 2023-10-17T19:41:56.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000070bd0b49536fa8d-00652ee384-294c-a1'), ('Date', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:41:56.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.712 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:56.763 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:56.763 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:56.795 INFO:teuthology.orchestra.run.smithi049.stderr:No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:41:56.808 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:41:56.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.808 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571593.193690_1189.1", 2023-10-17T19:41:56.809 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:39:53.193690Z" 2023-10-17T19:41:56.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.809 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:56.810 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:41:56.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.810 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:56.811 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "0.000000" 2023-10-17T19:41:56.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.811 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:56.811 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:41:56.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.812 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:41:56.813 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:56.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-17T19:41:56.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.815 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:56.867 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:56.895+0000 7fa4dbdef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:56.897 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-17T19:41:56.896+0000 7fa4dbdef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:41:56.910 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:56.911 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:56.911 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:56.911 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:56.911 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:56.912 INFO:teuthology.orchestra.run.smithi049.stdout: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:41:56.912 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.912 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:56.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.913 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:56.913 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.913 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.913 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697571593.193690_1189.1", 2023-10-17T19:41:56.914 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.914 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 1, 2023-10-17T19:41:56.914 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.914 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:39:53.193690Z", 2023-10-17T19:41:56.915 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.915 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:56.916 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.916 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.916 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:41:56.916 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.916 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.917 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.917 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:41:56.917 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.918 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "", 2023-10-17T19:41:56.919 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.919 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.919 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.919 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "0.000000", 2023-10-17T19:41:56.920 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:56.920 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: "state": 1, 2023-10-17T19:41:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:41:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:56.922 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:56.922 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:41:56.922 INFO:teuthology.orchestra.run.smithi049.stdout: "realm_epoch": 2 2023-10-17T19:41:56.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:56.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:56.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.923 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:56.923 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 1, 2023-10-17T19:41:56.923 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 1 2023-10-17T19:41:56.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:56.924 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:56Z" 2023-10-17T19:41:56.924 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:56.925 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697571593.193690_1189.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-17T19:39:53.193690Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697569413.363964_763.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.363964Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697569413.343990_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:03:33.343990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-17T19:41:56Z" } 2023-10-17T19:41:56.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571593.193690_1189.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:39:53.193690Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:41:56.926 DEBUG:rgw_multi.tests:log_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:56.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1697571593.193690_1189.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:41:56.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-17T19:41:56.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:56.928 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:56.977 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:56.977 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:57.029 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-10-17T19:41:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:41:57.029 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:41:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:41:57.030 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:41:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028936", 2023-10-17T19:41:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:41:56.670001Z" 2023-10-17T19:41:57.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047696", 2023-10-17T19:41:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: "last_update": "2023-10-17T19:41:56.650667Z" 2023-10-17T19:41:57.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.033 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-10-17T19:41:57.034 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:41:57.034 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:41:57.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:57.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:41:57.035 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:41:57.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:41:57.087 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:41:57.133 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:41:57.133 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:41:57.133 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:41:57.133 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:41:57.134 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:41:57.134 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:41:57.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.134 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:41:57.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.135 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:41:57.135 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:57.135 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:41:57.135 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:41:57.136 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:57.136 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:57.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:57.136 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:41:57.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.137 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:41:57.137 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:57.138 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:41:57.138 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:41:57.138 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:57.138 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:41:57.138 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:57.139 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:41:57.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.140 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:41:57.140 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:57.140 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:41:57.140 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:41:57.140 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:57.141 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:41:57.141 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:57.141 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:41:57.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:41:57.142 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:41:57.142 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:41:57.143 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:41:57.143 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:41:57.143 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:41:57.143 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:41:57.144 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:41:57.144 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:41:57.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:41:57.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:41:57.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.145 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:41:57.145 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:41:57.145 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:41:57.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:41:57.146 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:41:57Z" 2023-10-17T19:41:57.146 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:41:57.147 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047556", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:38:18.632571Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:41:57Z" } 2023-10-17T19:41:57.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047556', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.632571Z'}}] 2023-10-17T19:41:57.148 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:41:57.149 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:41:57.149 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000028936 target=00000000000000000000:00000000000000028796, shard=3 master=00000000000000000000:00000000000000047696 target=00000000000000000000:00000000000000047556 2023-10-17T19:42:02.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:02.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:02.151 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:02.212 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:02.265 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:42:02.266 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:42:02.266 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:42:02.266 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:42:02.267 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:42:02.267 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:42:02.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.267 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:42:02.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:02.268 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:42:02.268 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:02.268 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:02.268 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:02.269 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:02.269 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:02.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:02.269 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:02.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:02.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:02.270 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:42:02.271 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:02.271 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:02.271 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:42:02.271 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:02.271 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:02.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:02.272 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:42:02.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:02.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:02.273 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:42:02.273 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:02.273 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:02.273 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:02.274 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:02.274 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:42:02.274 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:02.274 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:02.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:02.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:02.275 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:42:02.275 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:02.276 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:02.276 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:42:02.276 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:02.276 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:42:02.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:02.277 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:42:02.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:02.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:02.277 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:42:02.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.278 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:42:02.278 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:42:02.278 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:42:02.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:02.279 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:42:02Z" 2023-10-17T19:42:02.279 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:42:02.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047556", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:38:18.632571Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:42:02Z" } 2023-10-17T19:42:02.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047556', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.632571Z'}}] 2023-10-17T19:42:02.281 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:42:02.282 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:42:02.282 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000028936 target=00000000000000000000:00000000000000028796, shard=3 master=00000000000000000000:00000000000000047696 target=00000000000000000000:00000000000000047556 2023-10-17T19:42:07.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:07.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:07.284 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:07.346 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:07.347 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:07.397 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:42:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:42:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:42:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:42:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:42:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:42:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:42:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:42:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:42:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:42:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:42:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:42:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:42:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:42:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:42:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:42:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:42:07.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.408 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:42:07.408 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:42:07.408 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:42:07.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:07.409 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:42:07Z" 2023-10-17T19:42:07.409 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:42:07.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047556", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:38:18.632571Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:42:07Z" } 2023-10-17T19:42:07.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047556', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.632571Z'}}] 2023-10-17T19:42:07.411 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:42:07.411 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:42:07.412 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000028936 target=00000000000000000000:00000000000000028796, shard=3 master=00000000000000000000:00000000000000047696 target=00000000000000000000:00000000000000047556 2023-10-17T19:42:12.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:12.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:12.415 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:12.477 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:12.478 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:12.528 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:42:12.528 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:42:12.529 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:42:12.529 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:42:12.529 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:42:12.529 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:42:12.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.530 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:42:12.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:12.530 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:42:12.530 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:12.531 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:12.531 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:12.531 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:12.531 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:12.531 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:12.532 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:42:12.533 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:12.533 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:12.533 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:42:12.533 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:12.533 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:12.534 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:12.534 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:42:12.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:12.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:12.535 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:42:12.535 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:12.535 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:12.535 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:12.535 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:12.536 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:42:12.536 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:12.536 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:12.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:12.537 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047556", 2023-10-17T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:12.538 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.632571Z" 2023-10-17T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.539 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:42:12Z" 2023-10-17T19:42:12.540 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:42:12.541 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028796", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:37:50.926594Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047556", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:38:18.632571Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:42:12Z" } 2023-10-17T19:42:12.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047556', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.632571Z'}}] 2023-10-17T19:42:12.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:42:12.543 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047556'} 2023-10-17T19:42:12.543 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000028936 target=00000000000000000000:00000000000000028796, shard=3 master=00000000000000000000:00000000000000047696 target=00000000000000000000:00000000000000047556 2023-10-17T19:42:17.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:17.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:17.546 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:17.607 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:17.608 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:17.660 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-10-17T19:42:17.661 INFO:teuthology.orchestra.run.smithi049.stdout: "sync_status": { 2023-10-17T19:42:17.661 INFO:teuthology.orchestra.run.smithi049.stdout: "info": { 2023-10-17T19:42:17.661 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "sync", 2023-10-17T19:42:17.662 INFO:teuthology.orchestra.run.smithi049.stdout: "num_shards": 4, 2023-10-17T19:42:17.662 INFO:teuthology.orchestra.run.smithi049.stdout: "instance_id": 16154068986440829441 2023-10-17T19:42:17.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.662 INFO:teuthology.orchestra.run.smithi049.stdout: "markers": [ 2023-10-17T19:42:17.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:17.663 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 0, 2023-10-17T19:42:17.663 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:17.663 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:17.663 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:17.663 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:17.664 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:17.664 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:17.664 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:17.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 1, 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:17.665 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000041402", 2023-10-17T19:42:17.666 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:17.666 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 0, 2023-10-17T19:42:17.666 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:17.666 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:38:18.659371Z" 2023-10-17T19:42:17.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:17.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:17.667 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 2, 2023-10-17T19:42:17.668 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:17.668 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:17.668 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000028936", 2023-10-17T19:42:17.668 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:17.668 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 9, 2023-10-17T19:42:17.669 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:17.669 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:41:56.670001Z" 2023-10-17T19:42:17.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:17.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-10-17T19:42:17.670 INFO:teuthology.orchestra.run.smithi049.stdout: "key": 3, 2023-10-17T19:42:17.670 INFO:teuthology.orchestra.run.smithi049.stdout: "val": { 2023-10-17T19:42:17.670 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "incremental-sync", 2023-10-17T19:42:17.670 INFO:teuthology.orchestra.run.smithi049.stdout: "marker": "00000000000000000000:00000000000000047696", 2023-10-17T19:42:17.670 INFO:teuthology.orchestra.run.smithi049.stdout: "next_step_marker": "", 2023-10-17T19:42:17.671 INFO:teuthology.orchestra.run.smithi049.stdout: "total_entries": 55, 2023-10-17T19:42:17.671 INFO:teuthology.orchestra.run.smithi049.stdout: "pos": 0, 2023-10-17T19:42:17.671 INFO:teuthology.orchestra.run.smithi049.stdout: "timestamp": "2023-10-17T19:41:56.650667Z" 2023-10-17T19:42:17.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:17.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-10-17T19:42:17.672 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-10-17T19:42:17.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.672 INFO:teuthology.orchestra.run.smithi049.stdout: "full_sync": { 2023-10-17T19:42:17.672 INFO:teuthology.orchestra.run.smithi049.stdout: "total": 64, 2023-10-17T19:42:17.672 INFO:teuthology.orchestra.run.smithi049.stdout: "complete": 64 2023-10-17T19:42:17.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2023-10-17T19:42:17.673 INFO:teuthology.orchestra.run.smithi049.stdout: "current_time": "2023-10-17T19:42:17Z" 2023-10-17T19:42:17.673 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-10-17T19:42:17.674 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16154068986440829441 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:27:51.381293Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000041402", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-17T19:38:18.659371Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028936", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-17T19:41:56.670001Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047696", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-17T19:41:56.650667Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-17T19:42:17Z" } 2023-10-17T19:42:17.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041402', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:38:18.659371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028936', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:41:56.670001Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047696', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:41:56.650667Z'}}] 2023-10-17T19:42:17.675 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:42:17.675 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041402', 2: '00000000000000000000:00000000000000028936', 3: '00000000000000000000:00000000000000047696'} 2023-10-17T19:42:17.676 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:17.676 DEBUG:boto:path=/hbmuog-74/ 2023-10-17T19:42:17.677 DEBUG:boto:auth_path=/hbmuog-74/ 2023-10-17T19:42:17.677 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.677 DEBUG:boto:Path: /hbmuog-74/ 2023-10-17T19:42:17.678 DEBUG:boto:Data: 2023-10-17T19:42:17.678 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.679 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.679 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.680 DEBUG:boto:Params: {} 2023-10-17T19:42:17.680 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:17.681 DEBUG:boto:Token: None 2023-10-17T19:42:17.681 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-74/ 2023-10-17T19:42:17.681 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:PI4xMYBpnVwrkZXwIX/aC2iu84U= 2023-10-17T19:42:17.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PI4xMYBpnVwrkZXwIX/aC2iu84U=', 'Content-Length': '0'} 2023-10-17T19:42:17.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b27832371875fda-00652ee399-2940-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.688 DEBUG:boto:path=/hbmuog-74/obj3 2023-10-17T19:42:17.688 DEBUG:boto:auth_path=/hbmuog-74/obj3 2023-10-17T19:42:17.688 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.689 DEBUG:boto:Path: /hbmuog-74/obj3 2023-10-17T19:42:17.689 DEBUG:boto:Data: 2023-10-17T19:42:17.690 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.690 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.691 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.691 DEBUG:boto:Params: {} 2023-10-17T19:42:17.691 DEBUG:boto:Token: None 2023-10-17T19:42:17.692 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-74/obj3 2023-10-17T19:42:17.692 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:rJV3WUktIqefol0Dty5/4QjgHMg= 2023-10-17T19:42:17.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rJV3WUktIqefol0Dty5/4QjgHMg=', 'Content-Length': '0'} 2023-10-17T19:42:17.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ba9cdd0c36883f9-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.696 DEBUG:boto:path=/hbmuog-74/obj3 2023-10-17T19:42:17.696 DEBUG:boto:auth_path=/hbmuog-74/obj3 2023-10-17T19:42:17.697 DEBUG:boto:Method: GET 2023-10-17T19:42:17.697 DEBUG:boto:Path: /hbmuog-74/obj3 2023-10-17T19:42:17.698 DEBUG:boto:Data: 2023-10-17T19:42:17.698 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.699 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.699 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.699 DEBUG:boto:Params: {} 2023-10-17T19:42:17.700 DEBUG:boto:Token: None 2023-10-17T19:42:17.700 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-74/obj3 2023-10-17T19:42:17.701 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:BKeTdkdIsUzJV/4T14Z9ghrrqbs= 2023-10-17T19:42:17.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BKeTdkdIsUzJV/4T14Z9ghrrqbs=', 'Content-Length': '0'} 2023-10-17T19:42:17.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007deff29db084c518-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.704 DEBUG:boto:path=/hbmuog-74/obj4 2023-10-17T19:42:17.705 DEBUG:boto:auth_path=/hbmuog-74/obj4 2023-10-17T19:42:17.705 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.706 DEBUG:boto:Path: /hbmuog-74/obj4 2023-10-17T19:42:17.706 DEBUG:boto:Data: 2023-10-17T19:42:17.706 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.707 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.707 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.708 DEBUG:boto:Params: {} 2023-10-17T19:42:17.708 DEBUG:boto:Token: None 2023-10-17T19:42:17.709 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-74/obj4 2023-10-17T19:42:17.709 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:QaLQ3Zx7ElkV3jAdYNBcuZism4Y= 2023-10-17T19:42:17.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QaLQ3Zx7ElkV3jAdYNBcuZism4Y=', 'Content-Length': '0'} 2023-10-17T19:42:17.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba3d86105165fe7e-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.712 DEBUG:boto:path=/hbmuog-74/obj4 2023-10-17T19:42:17.713 DEBUG:boto:auth_path=/hbmuog-74/obj4 2023-10-17T19:42:17.713 DEBUG:boto:Method: GET 2023-10-17T19:42:17.713 DEBUG:boto:Path: /hbmuog-74/obj4 2023-10-17T19:42:17.714 DEBUG:boto:Data: 2023-10-17T19:42:17.714 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.715 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.715 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.715 DEBUG:boto:Params: {} 2023-10-17T19:42:17.716 DEBUG:boto:Token: None 2023-10-17T19:42:17.716 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-74/obj4 2023-10-17T19:42:17.717 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:7lfAcYXXFFMwMsCPHwMUzxD3274= 2023-10-17T19:42:17.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7lfAcYXXFFMwMsCPHwMUzxD3274=', 'Content-Length': '0'} 2023-10-17T19:42:17.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2b8878276b2adea-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.719 DEBUG:boto:path=/hbmuog-75/ 2023-10-17T19:42:17.720 DEBUG:boto:auth_path=/hbmuog-75/ 2023-10-17T19:42:17.720 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.721 DEBUG:boto:Path: /hbmuog-75/ 2023-10-17T19:42:17.721 DEBUG:boto:Data: 2023-10-17T19:42:17.722 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.722 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.722 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.723 DEBUG:boto:Params: {} 2023-10-17T19:42:17.723 DEBUG:boto:Token: None 2023-10-17T19:42:17.724 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-75/ 2023-10-17T19:42:17.724 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:L3gClJ4ZTyeHADdhPMxAwtQ6KuU= 2023-10-17T19:42:17.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L3gClJ4ZTyeHADdhPMxAwtQ6KuU=', 'Content-Length': '0'} 2023-10-17T19:42:17.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eeca085068c1f0e-00652ee399-2940-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.727 DEBUG:boto:path=/hbmuog-75/obj3 2023-10-17T19:42:17.728 DEBUG:boto:auth_path=/hbmuog-75/obj3 2023-10-17T19:42:17.728 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.729 DEBUG:boto:Path: /hbmuog-75/obj3 2023-10-17T19:42:17.729 DEBUG:boto:Data: 2023-10-17T19:42:17.730 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.730 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.731 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.731 DEBUG:boto:Params: {} 2023-10-17T19:42:17.732 DEBUG:boto:Token: None 2023-10-17T19:42:17.732 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-75/obj3 2023-10-17T19:42:17.732 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:ZXJj8qDawpg4A7L8y96q093CSgc= 2023-10-17T19:42:17.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZXJj8qDawpg4A7L8y96q093CSgc=', 'Content-Length': '0'} 2023-10-17T19:42:17.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063d83630803c2a2d-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.736 DEBUG:boto:path=/hbmuog-75/obj3 2023-10-17T19:42:17.736 DEBUG:boto:auth_path=/hbmuog-75/obj3 2023-10-17T19:42:17.736 DEBUG:boto:Method: GET 2023-10-17T19:42:17.737 DEBUG:boto:Path: /hbmuog-75/obj3 2023-10-17T19:42:17.737 DEBUG:boto:Data: 2023-10-17T19:42:17.738 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.738 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.739 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.739 DEBUG:boto:Params: {} 2023-10-17T19:42:17.740 DEBUG:boto:Token: None 2023-10-17T19:42:17.740 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-75/obj3 2023-10-17T19:42:17.741 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:Qx9DmfszH2Jw/T1WeDrAHodK+iA= 2023-10-17T19:42:17.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Qx9DmfszH2Jw/T1WeDrAHodK+iA=', 'Content-Length': '0'} 2023-10-17T19:42:17.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b384f1ab6c1b6367-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.790 DEBUG:boto:path=/hbmuog-75/obj4 2023-10-17T19:42:17.791 DEBUG:boto:auth_path=/hbmuog-75/obj4 2023-10-17T19:42:17.791 DEBUG:boto:Method: HEAD 2023-10-17T19:42:17.792 DEBUG:boto:Path: /hbmuog-75/obj4 2023-10-17T19:42:17.792 DEBUG:boto:Data: 2023-10-17T19:42:17.793 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.793 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.793 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.794 DEBUG:boto:Params: {} 2023-10-17T19:42:17.794 DEBUG:boto:Token: None 2023-10-17T19:42:17.795 DEBUG:boto:StringToSign: HEAD Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-75/obj4 2023-10-17T19:42:17.795 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:sQ1aD6hZnBUbQ+DoiWvaFeIRjXs= 2023-10-17T19:42:17.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQ1aD6hZnBUbQ+DoiWvaFeIRjXs=', 'Content-Length': '0'} 2023-10-17T19:42:17.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947a2b27ef685567-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.799 DEBUG:boto:path=/hbmuog-75/obj4 2023-10-17T19:42:17.799 DEBUG:boto:auth_path=/hbmuog-75/obj4 2023-10-17T19:42:17.800 DEBUG:boto:Method: GET 2023-10-17T19:42:17.800 DEBUG:boto:Path: /hbmuog-75/obj4 2023-10-17T19:42:17.801 DEBUG:boto:Data: 2023-10-17T19:42:17.801 DEBUG:boto:Headers: {} 2023-10-17T19:42:17.801 DEBUG:boto:Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:17.802 DEBUG:boto:Port: 8001 2023-10-17T19:42:17.802 DEBUG:boto:Params: {} 2023-10-17T19:42:17.803 DEBUG:boto:Token: None 2023-10-17T19:42:17.803 DEBUG:boto:StringToSign: GET Tue, 17 Oct 2023 19:42:17 GMT /hbmuog-75/obj4 2023-10-17T19:42:17.804 DEBUG:boto:Signature: AWS EERKR7CLBH40TV9B4PL4:y02oilPSDhRVndlzkGzvI0fRSUU= 2023-10-17T19:42:17.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:42:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:y02oilPSDhRVndlzkGzvI0fRSUU=', 'Content-Length': '0'} 2023-10-17T19:42:17.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:41:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-74:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9901.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc105d6cbc8c969a-00652ee399-2940-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:42:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:17.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:17.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hbmuog-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:17.808 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hbmuog-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:17.861 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:17.861 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:17.918 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:42:17.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:17.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:17.919 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:17.968 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setuser ceph since I am not root 2023-10-17T19:42:17.968 INFO:teuthology.orchestra.run.smithi049.stderr:ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.016 INFO:teuthology.orchestra.run.smithi049.stdout:{"sync_policy":{"groups":[]}} 2023-10-17T19:42:18.016 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-10-17T19:42:18.017 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.018 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:18.018 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-10-17T19:42:18.019 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:18.020 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:18.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:18.020 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:18.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-10-17T19:42:18.021 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-10-17T19:42:18.021 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-17T19:42:18.022 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:18.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hbmuog-55 2023-10-17T19:42:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.026 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.026 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.026 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.026 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.027 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.027 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SAg9Z2hYeL/uKsL+QuawPOov3ew= 2023-10-17T19:42:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SAg9Z2hYeL/uKsL+QuawPOov3ew=', 'Content-Length': '0'} 2023-10-17T19:42:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000079b6a6a4a6145fc9-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.031 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.032 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.032 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.032 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.033 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.033 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T19:42:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T19:42:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4c9fe7fbfa229b2-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.037 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.038 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.038 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.038 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.039 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.039 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3MFWJEUUghIs90BQAosZO0cbEy4= 2023-10-17T19:42:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3MFWJEUUghIs90BQAosZO0cbEy4='} 2023-10-17T19:42:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000012fd8e6d3d2e8fe-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.043 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.044 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.044 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.044 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.045 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.045 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T19:42:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T19:42:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000321b320fe67b5f51-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.050 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.050 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.050 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.051 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.051 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.052 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vCfwuNlmU5H3TWgeqEgQfHWKmtE= 2023-10-17T19:42:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vCfwuNlmU5H3TWgeqEgQfHWKmtE='} 2023-10-17T19:42:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068b42d41773f4d13-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.056 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.056 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.057 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.057 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.057 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.058 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T19:42:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T19:42:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ede015d66e19b9d-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.062 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.062 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.062 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.063 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.063 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.064 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OO/BNij3RTA9UHTP/jyiT7SMgWY= 2023-10-17T19:42:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OO/BNij3RTA9UHTP/jyiT7SMgWY='} 2023-10-17T19:42:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002077ee5fc1a66ab4-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.068 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.068 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.068 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.069 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.069 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.069 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw= 2023-10-17T19:42:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bQQw/MC3b2weFvh/GU35w8eRbLw=', 'Content-Length': '0'} 2023-10-17T19:42:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f71536734718b53-00652ed613-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.074 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.074 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.074 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.075 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:44:35 GMT 2023-10-17T19:42:18.075 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.076 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GRdVmGpIHLRCYKudlUIftaVmzVE= 2023-10-17T19:42:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:44:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GRdVmGpIHLRCYKudlUIftaVmzVE='} 2023-10-17T19:42:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bef12cbd9b9ef9f8-00652ed613-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:18.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:18.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:18.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568275.712131_636.1", 2023-10-17T19:42:18.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T18:44:35.712131Z" 2023-10-17T19:42:18.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:18.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T19:42:18.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:18.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:18.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:18.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:18.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:18.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:36.044+0000 7f371ee56780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:36.045+0000 7f371ee56780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:18.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:18.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:18.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:18.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:18.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:18.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:18.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568275.706429_633.1", 2023-10-17T19:42:18.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.706429Z", 2023-10-17T19:42:18.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:18.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:18.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:18.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:18.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:18.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:18.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:18.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:18.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:44:36Z" 2023-10-17T19:42:18.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:18.107 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:18.108 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:18.108 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:18.108 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:18.109 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:18.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.109 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.109 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:18.110 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.110 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:18.110 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568275.706429_633.1", 2023-10-17T19:42:18.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.111 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:18.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.112 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:44:35.706429Z", 2023-10-17T19:42:18.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.112 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.113 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.113 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.113 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:18.114 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.114 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:18.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:18.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.116 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.116 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.117 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.117 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:18.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.118 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:18.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.119 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:18.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.119 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.120 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.120 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.120 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:18.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.121 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:18.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.122 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:18.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.123 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.123 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.123 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:18.124 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.124 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:18.124 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:18.124 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:18.125 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.125 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:44:36Z" 2023-10-17T19:42:18.125 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:18.126 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568275.706429_633.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:44:35.706429Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:18.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:18.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568275.706429_633.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:18.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568275.712131_636.1 target=1_1697568275.706429_633.1 2023-10-17T19:42:18.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:18.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:41.167+0000 7faefb953780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:18.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:41.168+0000 7faefb953780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:18.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:18.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:18.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:18.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:18.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:18.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:18.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:18.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568275.712131_636.1", 2023-10-17T19:42:18.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:18.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.712131Z", 2023-10-17T19:42:18.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:18.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:18.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:18.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:18.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:18.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:18.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:18.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:18.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:18.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:18.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:44:41Z" 2023-10-17T19:42:18.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:18.150 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:18.150 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:18.151 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:18.151 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:18.151 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:18.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.152 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.152 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:18.152 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.153 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:18.153 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.153 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568275.712131_636.1", 2023-10-17T19:42:18.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.154 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:18.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.155 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:44:35.712131Z", 2023-10-17T19:42:18.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.155 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.156 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.156 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.156 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:18.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.157 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:18.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:18.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.159 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.159 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.160 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.160 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:18.160 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.161 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:18.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.162 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:18.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.163 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.163 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.163 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:18.164 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:18.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:18.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:18.164 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:18.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:18.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:18.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:18.166 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:18.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:18.166 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.167 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:18.167 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:18.167 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.168 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:18.168 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:18.168 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:18.168 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:18.169 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:44:41Z" 2023-10-17T19:42:18.169 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:18.169 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568275.712131_636.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:44:35.712131Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:18.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:18.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568275.712131_636.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:18.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:18.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T19:42:18.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.171 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:18.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:41.292+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T19:42:18.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:46.294+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T19:42:18.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:51.294+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T19:42:18.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:44:56.295+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T19:42:18.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:01.295+0000 7f6fbb07e780 1 waiting to reach incremental sync.. 2023-10-17T19:42:18.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:06.297+0000 7f6fbb07e780 1 bucket sync caught up with source: 2023-10-17T19:42:18.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000001.26.6, , , , 00000000001.18.6, , , 00000000001.49.6, , 00000000001.110.6] 2023-10-17T19:42:18.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.26.6, , , , 00000000001.18.6, , , 00000000001.49.6, , 00000000001.110.6] 2023-10-17T19:42:18.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:06.297+0000 7f6fbb07e780 0 bucket checkpoint complete 2023-10-17T19:42:18.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T19:42:18.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:18.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:06.420+0000 7f40b8b39780 1 bucket sync caught up with source: 2023-10-17T19:42:18.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000001.146.6, , , , 00000000001.50.6, , , 00000000001.221.6, , 00000000001.406.6] 2023-10-17T19:42:18.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.146.6, , , , 00000000001.50.6, , , 00000000001.221.6, , 00000000001.406.6] 2023-10-17T19:42:18.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:06.420+0000 7f40b8b39780 0 bucket checkpoint complete 2023-10-17T19:42:18.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.182 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.183 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.183 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.183 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.184 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.184 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE= 2023-10-17T19:42:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE=', 'Content-Length': '0'} 2023-10-17T19:42:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000796b1091626f0358-00652ed632-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.188 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.189 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.189 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.189 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.190 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.190 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE= 2023-10-17T19:42:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ez8guKXYAcL+8moOVLp1ZqGv+FE=', 'Content-Length': '0'} 2023-10-17T19:42:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000369086b13ab47f13-00652ed632-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.191 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.196 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.196 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.196 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.197 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.197 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.198 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T19:42:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T19:42:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006493f162474a9bbc-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:18.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:18.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:18.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:18.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.205 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.205 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.205 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.206 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.206 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.206 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T19:42:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T19:42:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2fbc2a85165e71c-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.208 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:18.208 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:18.208 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:18.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.213 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.213 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.213 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.213 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.214 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.214 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T19:42:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T19:42:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf482fe93b76d92-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.220 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.221 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.221 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.221 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.222 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ= 2023-10-17T19:42:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XemFwOVIVOkivjxNdkoVI4maflQ=', 'Content-Length': '0'} 2023-10-17T19:42:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c594de954af1f139-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.223 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.227 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.227 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.228 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.228 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.228 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.229 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T19:42:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T19:42:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919d40f60fb50a03-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.234 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.235 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.235 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.235 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.236 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.236 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T19:42:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T19:42:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001448386bd1bfe9ac-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.241 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.242 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.242 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.242 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.243 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.243 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4= 2023-10-17T19:42:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4=', 'Content-Length': '0'} 2023-10-17T19:42:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413b5994ba002bee-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.248 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.249 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.249 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.249 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.250 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.250 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4= 2023-10-17T19:42:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gQllkulTNqwL3HlKvhncmtQpHo4=', 'Content-Length': '0'} 2023-10-17T19:42:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c874687b8320c44d-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.251 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.256 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.256 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.256 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.257 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.257 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.257 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T19:42:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T19:42:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69f251439fcfdf0-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.263 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.263 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.263 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.264 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.264 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.264 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g= 2023-10-17T19:42:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WeIfVGWXAXogafs/sY+QFjdi/4g=', 'Content-Length': '0'} 2023-10-17T19:42:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdad742a36987599-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.269 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.269 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.270 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.270 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.271 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.271 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034713855188e80a8-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.277 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.277 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.277 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.278 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.278 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.279 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc613644b31e10b-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.283 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.284 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.284 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.284 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.285 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.285 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM= 2023-10-17T19:42:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM=', 'Content-Length': '0'} 2023-10-17T19:42:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfe92f410cd1cc9-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.289 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.290 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.290 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.290 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.291 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.291 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM= 2023-10-17T19:42:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rohleqQ7SQQpxCjYQOsbKc68nXM=', 'Content-Length': '0'} 2023-10-17T19:42:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003734017191c63b3a-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.296 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.296 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.297 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.297 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.297 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.298 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM= 2023-10-17T19:42:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM=', 'Content-Length': '0'} 2023-10-17T19:42:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8572085dedffe5-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.302 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.303 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.303 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.303 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.304 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.304 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM= 2023-10-17T19:42:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QDWichCZYLmBI1sixZuHFQFnApM=', 'Content-Length': '0'} 2023-10-17T19:42:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b1d0ba41394495-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.309 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.309 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.310 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.310 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.310 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.311 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d551f0209795c50-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.316 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.316 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.317 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.317 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.317 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.318 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d5fadd9f3a03563-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.323 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.324 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.324 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.324 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.324 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.325 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T19:42:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T19:42:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7b714a1caa4a10-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.330 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.330 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.331 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.331 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.331 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.333 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T19:42:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T19:42:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e7a04edbff2924-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.338 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.338 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.339 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.339 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.340 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI= 2023-10-17T19:42:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI=', 'Content-Length': '0'} 2023-10-17T19:42:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5784e0b84ca25f-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.345 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.346 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.346 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.346 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.347 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI= 2023-10-17T19:42:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sVBbdz9SETHnFnf4nLTAc5eWpZI=', 'Content-Length': '0'} 2023-10-17T19:42:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1edf61cd3bc4c3-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.348 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.352 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.353 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.353 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.353 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.353 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.354 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T19:42:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T19:42:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952381b147a3b654-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.359 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.359 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.360 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.360 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.360 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.361 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8= 2023-10-17T19:42:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uFg2hVt10hlYqsYXdkMiSlRy/W8=', 'Content-Length': '0'} 2023-10-17T19:42:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c039ba492fb68-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.366 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.366 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.366 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.367 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.367 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.368 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb5d7b04df101da-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.373 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.373 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.374 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.374 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.374 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.375 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef05ec451fb49de-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.379 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.380 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.380 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.380 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.381 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.381 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo= 2023-10-17T19:42:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo=', 'Content-Length': '0'} 2023-10-17T19:42:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd06e845ad135dd-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.386 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.386 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.386 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.387 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.387 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.387 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo= 2023-10-17T19:42:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cBe6bDk+lVyjJeo/zLVZn9un3vo=', 'Content-Length': '0'} 2023-10-17T19:42:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b25f09d8319ec96-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.388 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.392 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.392 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.392 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.393 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.393 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.394 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik= 2023-10-17T19:42:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik=', 'Content-Length': '0'} 2023-10-17T19:42:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dbe62eb65cfeec-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.398 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.398 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.399 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.399 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.399 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.400 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik= 2023-10-17T19:42:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hDN35rVxdYYcOP0Kr1XIwrlqFik=', 'Content-Length': '0'} 2023-10-17T19:42:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e23e9f4771a0e54-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.405 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.405 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.405 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.405 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.406 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.406 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a8d2d7d2b54681-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.411 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.412 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.412 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.412 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.412 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.413 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee403291b8787d5-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.414 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.419 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.419 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.419 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.419 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.420 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.420 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T19:42:18.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T19:42:18.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c80e87a14756e2-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.425 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.425 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.425 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.426 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.426 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.427 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T19:42:18.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T19:42:18.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b60be710b80135-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.431 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.432 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.432 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.432 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.432 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.433 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk= 2023-10-17T19:42:18.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk=', 'Content-Length': '0'} 2023-10-17T19:42:18.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c402c304929b0658-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.437 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.438 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.438 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.438 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.439 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.439 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk= 2023-10-17T19:42:18.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PMgt1aBA8wDXUvNbL4AJzgSaQuk=', 'Content-Length': '0'} 2023-10-17T19:42:18.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095835a57ef358a57-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.440 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:18.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.444 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.444 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.445 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.445 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.445 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.446 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T19:42:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T19:42:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f6bfbf1375f79d-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.451 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.451 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.452 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.452 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.452 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.453 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA= 2023-10-17T19:42:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:B+/rJwlWImyWg4RqzzekWxrVamA=', 'Content-Length': '0'} 2023-10-17T19:42:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e7f7f8e0f26cfe-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.458 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.458 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.458 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.458 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.459 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.459 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9bc04136672fa6f-00652ed632-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.465 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.465 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.465 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.466 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.466 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.466 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs= 2023-10-17T19:42:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VBe3w/Vk3n3q1sqBiUh/+1kmixs=', 'Content-Length': '0'} 2023-10-17T19:42:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f684a54b230c680-00652ed632-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.471 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.471 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.472 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.472 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.472 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:18.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.473 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8= 2023-10-17T19:42:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8=', 'Content-Length': '0'} 2023-10-17T19:42:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068887a8e8e2c0c28-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.477 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.477 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.477 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.478 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.478 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.479 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8= 2023-10-17T19:42:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zpVMv7szKhD6aPqIxMiEPNU2nX8=', 'Content-Length': '0'} 2023-10-17T19:42:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ff2223e03c17d8-00652ed632-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.480 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.483 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.483 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.484 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.484 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:06 GMT 2023-10-17T19:42:18.484 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.485 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3ovWD/7xYZULXZwonzRhROxsxtg= 2023-10-17T19:42:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:06 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3ovWD/7xYZULXZwonzRhROxsxtg=', 'Content-Length': '0'} 2023-10-17T19:42:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b22db588e66406-00652ed632-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.489 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.489 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.490 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.490 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.490 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.491 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SkHs6NR6JRnepy7phJQMWSdCdbI= 2023-10-17T19:42:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SkHs6NR6JRnepy7phJQMWSdCdbI=', 'Content-Length': '0'} 2023-10-17T19:42:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce73fe7f090069c-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.496 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.496 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.496 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.497 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.497 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.498 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a216d18b07d566a1-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.502 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.503 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.503 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.503 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.504 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.504 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006807472f4b895388-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.505 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.509 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.509 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.510 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.510 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.510 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.511 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T19:42:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T19:42:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2a72dc670fdf0e-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.516 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.516 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.516 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.516 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.517 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.517 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T19:42:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T19:42:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f06436aec5001df-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.522 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.522 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.523 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.523 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.523 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.524 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c= 2023-10-17T19:42:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c=', 'Content-Length': '0'} 2023-10-17T19:42:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45b2a9b292b5e2c-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.529 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.529 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.529 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.530 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.530 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.531 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c= 2023-10-17T19:42:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bhr/tKAPJzHnT5n3EEvlQBCz62c=', 'Content-Length': '0'} 2023-10-17T19:42:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417eff1f6b43e5bc-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.531 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.536 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.536 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.536 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.536 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.537 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.537 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T19:42:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T19:42:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541da552fc2c9fde-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.542 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.542 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.543 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.543 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.543 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.544 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw= 2023-10-17T19:42:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TBvA3nFgaFk96yuXijXySNCvvxw=', 'Content-Length': '0'} 2023-10-17T19:42:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ebabc8f10a99e1-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.549 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.549 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.549 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.550 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.550 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.551 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2c302d9570478e-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.556 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.557 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.557 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.558 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa30345ffcc01683-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.562 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.563 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.563 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.563 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.564 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.564 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI= 2023-10-17T19:42:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI=', 'Content-Length': '0'} 2023-10-17T19:42:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc73d99441d6b0eb-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.569 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.569 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.570 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.570 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.570 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.571 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI= 2023-10-17T19:42:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZFaVkj7+UDgT95x5V/GuSEgcsI=', 'Content-Length': '0'} 2023-10-17T19:42:18.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002137673f73d9ba93-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.572 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:18.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:18.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.576 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.576 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.577 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.577 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.577 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:18.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.578 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY= 2023-10-17T19:42:18.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY=', 'Content-Length': '0'} 2023-10-17T19:42:18.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006473954e2ed1c1d3-00652ed633-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:18.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:18.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:18.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.582 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.583 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.583 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.583 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.584 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:18.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.584 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY= 2023-10-17T19:42:18.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TY7Y5JwkbcLLWDjpte0WlZkr2VY=', 'Content-Length': '0'} 2023-10-17T19:42:18.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f5b25df14e119c-00652ed633-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:18.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.589 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.590 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.590 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.590 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.591 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.591 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2926edb0d2836a-00652ed633-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:18.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:18.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.596 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.597 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.597 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.597 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.598 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.598 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk= 2023-10-17T19:42:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HXNc3KlTkyxlQzagEc6RDOJ5OYk=', 'Content-Length': '0'} 2023-10-17T19:42:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f621f5ac89cd02-00652ed633-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.599 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:18.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding": "None", 2023-10-17T19:42:18.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_index": { 2023-10-17T19:42:18.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:18.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "Normal", 2023-10-17T19:42:18.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "normal": { 2023-10-17T19:42:18.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 11, 2023-10-17T19:42:18.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:18.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "logs": [ 2023-10-17T19:42:18.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:18.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:18.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:18.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 11, 2023-10-17T19:42:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:18.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_id": "1#00000000001.26.6", 2023-10-17T19:42:18.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.85525403851221246", 2023-10-17T19:42:18.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op": "write", 2023-10-17T19:42:18.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "object": "a", 2023-10-17T19:42:18.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance": "", 2023-10-17T19:42:18.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": "complete", 2023-10-17T19:42:18.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "index_ver": 1, 2023-10-17T19:42:18.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.729680197Z", 2023-10-17T19:42:18.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "ver": { 2023-10-17T19:42:18.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pool": 9, 2023-10-17T19:42:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 4 2023-10-17T19:42:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bilog_flags": 0, 2023-10-17T19:42:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "versioned": false, 2023-10-17T19:42:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner": "", 2023-10-17T19:42:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner_display_name": "", 2023-10-17T19:42:18.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones_trace": [ 2023-10-17T19:42:18.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T19:42:18.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_id": "5#00000000001.18.6", 2023-10-17T19:42:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.13758827529464904184", 2023-10-17T19:42:18.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op": "write", 2023-10-17T19:42:18.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "object": "d", 2023-10-17T19:42:18.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance": "", 2023-10-17T19:42:18.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": "complete", 2023-10-17T19:42:18.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "index_ver": 1, 2023-10-17T19:42:18.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.863031096Z", 2023-10-17T19:42:18.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "ver": { 2023-10-17T19:42:18.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pool": 9, 2023-10-17T19:42:18.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 1 2023-10-17T19:42:18.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bilog_flags": 0, 2023-10-17T19:42:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "versioned": false, 2023-10-17T19:42:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner": "", 2023-10-17T19:42:18.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner_display_name": "", 2023-10-17T19:42:18.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones_trace": [ 2023-10-17T19:42:18.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T19:42:18.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_id": "8#00000000001.49.6", 2023-10-17T19:42:18.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.2339600626480474804", 2023-10-17T19:42:18.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op": "write", 2023-10-17T19:42:18.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "object": "c", 2023-10-17T19:42:18.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance": "", 2023-10-17T19:42:18.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": "complete", 2023-10-17T19:42:18.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "index_ver": 1, 2023-10-17T19:42:18.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.806992640Z", 2023-10-17T19:42:18.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "ver": { 2023-10-17T19:42:18.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pool": 9, 2023-10-17T19:42:18.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 3 2023-10-17T19:42:18.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bilog_flags": 0, 2023-10-17T19:42:18.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "versioned": false, 2023-10-17T19:42:18.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner": "", 2023-10-17T19:42:18.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner_display_name": "", 2023-10-17T19:42:18.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones_trace": [ 2023-10-17T19:42:18.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T19:42:18.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_id": "10#00000000001.110.6", 2023-10-17T19:42:18.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op_tag": "4c84373e-d451-49bb-a8c9-c37d5e5e2160.5676.7544705034949184787", 2023-10-17T19:42:18.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "op": "write", 2023-10-17T19:42:18.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "object": "b", 2023-10-17T19:42:18.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance": "", 2023-10-17T19:42:18.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": "complete", 2023-10-17T19:42:18.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "index_ver": 1, 2023-10-17T19:42:18.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:44:35.788417331Z", 2023-10-17T19:42:18.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "ver": { 2023-10-17T19:42:18.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pool": 9, 2023-10-17T19:42:18.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 7 2023-10-17T19:42:18.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bilog_flags": 0, 2023-10-17T19:42:18.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "versioned": false, 2023-10-17T19:42:18.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner": "", 2023-10-17T19:42:18.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "owner_display_name": "", 2023-10-17T19:42:18.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones_trace": [ 2023-10-17T19:42:18.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "entry": "4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20" 2023-10-17T19:42:18.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:18.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:18.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-55 2023-10-17T19:42:18.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 4 2023-10-17T19:42:18.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:18.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:18.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding": "None", 2023-10-17T19:42:18.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_index": { 2023-10-17T19:42:18.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:18.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "Normal", 2023-10-17T19:42:18.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "normal": { 2023-10-17T19:42:18.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:18.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:18.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "logs": [ 2023-10-17T19:42:18.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:18.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:18.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:18.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:18.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 11, 2023-10-17T19:42:18.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:18.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:18.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:18.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:18.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:18.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:18.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:18.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:18.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:18.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:18.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:18.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:18.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hbmuog-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:18.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hbmuog-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:18.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [] 2023-10-17T19:42:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.664 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.664 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.664 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.665 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.665 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.666 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T19:42:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T19:42:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000812648bb6c4b0711-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.670 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.671 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.671 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.671 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.672 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.672 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gIWC+zvbjVA2FD5bpzcSuZqfiRU= 2023-10-17T19:42:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gIWC+zvbjVA2FD5bpzcSuZqfiRU='} 2023-10-17T19:42:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000492895b46a57f550-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.677 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.677 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.677 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.678 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.678 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.679 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T19:42:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T19:42:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d93c3e409f49b7b-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.683 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.683 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.684 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.684 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.684 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.685 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BL7UlQ0y5x/sUMnEem7F4jaWFh8= 2023-10-17T19:42:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BL7UlQ0y5x/sUMnEem7F4jaWFh8='} 2023-10-17T19:42:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eac6ee2f2966a98a-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.689 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.689 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.690 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.690 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.690 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.691 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T19:42:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T19:42:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008464733dac03985-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.695 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.695 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.696 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.696 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.697 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9zTdUFeZlwPCQ84Jhxf1k64xfDI= 2023-10-17T19:42:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9zTdUFeZlwPCQ84Jhxf1k64xfDI='} 2023-10-17T19:42:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de8bbcc255ff8c7c-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.701 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.702 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.703 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.703 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA= 2023-10-17T19:42:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Zy0yBDMnpKOYxtJcZ5PcL2XYaCA=', 'Content-Length': '0'} 2023-10-17T19:42:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000415d6192aded757d-00652ed633-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.708 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:18.708 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:18.708 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:18.709 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:45:07 GMT 2023-10-17T19:42:18.709 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.710 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lkG1puv499lpV+VoKJQCZp24aSg= 2023-10-17T19:42:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:45:07 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lkG1puv499lpV+VoKJQCZp24aSg='} 2023-10-17T19:42:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026b748dbfb444e9a-00652ed633-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T19:42:18.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:18.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:07.995+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:18.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:12.996+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:18.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:17.997+0000 7f6a2dc95780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:18.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:22.999+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:27.999+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:33.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:38.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:43.000+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:48.001+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:53.002+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:45:58.004+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , ] 2023-10-17T19:42:18.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:03.005+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:08.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:13.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:18.006+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:23.007+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:28.008+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:33.009+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:38.011+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:43.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:48.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:53.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:46:58.012+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:03.013+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:08.014+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:13.016+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:18.017+0000 7f6a2dc95780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:23.018+0000 7f6a2dc95780 1 bucket sync caught up with source: 2023-10-17T19:42:18.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000003.40.6, 00000000002.36.6, 00000000002.38.6] 2023-10-17T19:42:18.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:23.018+0000 7f6a2dc95780 0 bucket checkpoint complete 2023-10-17T19:42:18.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T19:42:18.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:18.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:18.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:18.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:23.140+0000 7feb5d21b780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000001.406.6] 2023-10-17T19:42:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:18.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:28.142+0000 7feb5d21b780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000001.406.6] 2023-10-17T19:42:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:18.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:33.143+0000 7feb5d21b780 1 bucket sync caught up with source: 2023-10-17T19:42:18.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:18.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:18.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:33.143+0000 7feb5d21b780 0 bucket checkpoint complete 2023-10-17T19:42:18.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.745 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.746 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.746 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.746 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.747 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.747 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg= 2023-10-17T19:42:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg=', 'Content-Length': '0'} 2023-10-17T19:42:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6bdfc182aee8348-00652ed6c5-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.752 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.753 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.753 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.753 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.754 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.754 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg= 2023-10-17T19:42:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:puJHQhmgRcNDOBtmYjrmsy56Jdg=', 'Content-Length': '0'} 2023-10-17T19:42:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c855944fb3c8909-00652ed6c5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.755 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.759 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.759 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.760 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.760 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.760 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.761 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T19:42:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T19:42:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f18734ba029419-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.762 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:18.762 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:18.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:18.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:18.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-17T19:42:18.764 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-17T19:42:18.764 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-17T19:42:18.764 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-17T19:42:18.765 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.769 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.770 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.770 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.770 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.771 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.771 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T19:42:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T19:42:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0acc9a7dd25b2ea-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:18.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:18.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:18.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:18.774 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-17T19:42:18.774 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-17T19:42:18.774 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-17T19:42:18.775 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-17T19:42:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.779 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.780 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.780 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.780 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.781 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.781 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T19:42:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T19:42:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047855ffaecac7f83-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.787 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.787 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.787 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.788 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.788 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:18.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.788 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA= 2023-10-17T19:42:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BqoJGSwKAu+JByGzsKtMX8CigLA=', 'Content-Length': '0'} 2023-10-17T19:42:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1755ba2da10f34-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.794 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.794 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.795 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.795 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.795 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.796 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T19:42:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T19:42:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076eb605c525af276-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.800 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.801 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.801 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.801 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.802 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.802 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T19:42:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T19:42:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b175c7a49137a644-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.807 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.807 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.808 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.808 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.808 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.809 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8= 2023-10-17T19:42:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8=', 'Content-Length': '0'} 2023-10-17T19:42:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d0b45db2a3c5b3-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.813 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.814 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.814 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.814 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.815 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.815 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8= 2023-10-17T19:42:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Szx6S/etrxfFcXQ0+mg1IqdXRK8=', 'Content-Length': '0'} 2023-10-17T19:42:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c437ae2d36565e9-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.816 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.821 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.821 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.821 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.821 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.822 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.822 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T19:42:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T19:42:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac510b6248432e44-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.827 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.827 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.828 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.828 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.828 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:18.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.829 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE= 2023-10-17T19:42:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QL7baOaEkk5l0zUEflFiZxJ8YCE=', 'Content-Length': '0'} 2023-10-17T19:42:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b707310a962f5576-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.833 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.834 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.834 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.834 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.835 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.835 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d0f9941e213595-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.840 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.840 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.841 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.841 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.841 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.842 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011de529f6e8bd495-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.847 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.847 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.847 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.848 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.848 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.849 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU= 2023-10-17T19:42:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU=', 'Content-Length': '0'} 2023-10-17T19:42:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d1f4c3c0f42355-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.853 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.853 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.853 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.854 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.854 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.855 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU= 2023-10-17T19:42:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lvDyLJInAYNr4tG3rlaMGQKN6SU=', 'Content-Length': '0'} 2023-10-17T19:42:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb5d539acac5fdb7-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.859 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.859 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.860 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.860 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.860 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.861 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM= 2023-10-17T19:42:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM=', 'Content-Length': '0'} 2023-10-17T19:42:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1e4574344cc9c5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.865 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.866 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.866 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.866 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.866 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.867 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM= 2023-10-17T19:42:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VJTJmipuCLVHJr1OvzympAmp3cM=', 'Content-Length': '0'} 2023-10-17T19:42:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091084b3aaae70eb7-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.872 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.873 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.873 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.873 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.874 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.874 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53847c0aeff47dd-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.879 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.880 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.880 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.880 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.881 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.881 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085dd4d9b70b1564d-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.882 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.887 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.887 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.887 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.888 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.888 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.888 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T19:42:18.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T19:42:18.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2370ff5913d61b-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.893 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.894 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.894 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.894 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.895 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.895 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T19:42:18.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T19:42:18.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9274b1000ea7d8-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.900 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.901 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.901 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.902 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU= 2023-10-17T19:42:18.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU=', 'Content-Length': '0'} 2023-10-17T19:42:18.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27f7678727afead-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.906 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.907 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.907 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.907 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.908 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.908 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU= 2023-10-17T19:42:18.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/bMZNkiJdgvTo/kbnOgC8jZwrWU=', 'Content-Length': '0'} 2023-10-17T19:42:18.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8aaefa95a26e27-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.909 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.913 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.913 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.914 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.914 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.914 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.915 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T19:42:18.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T19:42:18.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bebf81f522e35e5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:18.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.920 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.920 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.920 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.921 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.921 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:18.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.922 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk= 2023-10-17T19:42:18.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9fG87fQOA1KsvY0RC3KQogfWKZk=', 'Content-Length': '0'} 2023-10-17T19:42:18.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015888f9a9af5e9e5-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.926 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.927 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.927 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.927 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.928 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.928 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf19f9a728460431-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.933 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.933 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.934 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.934 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.934 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.935 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e918e60c9eeea658-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.939 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.940 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.940 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.940 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.940 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.941 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08= 2023-10-17T19:42:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08=', 'Content-Length': '0'} 2023-10-17T19:42:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943570052fc795a8-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.945 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.945 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.946 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.946 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.946 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.947 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08= 2023-10-17T19:42:18.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KYvN5CBH28A1yMQbF77cxIziw08=', 'Content-Length': '0'} 2023-10-17T19:42:18.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267296fec0648bcc-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:18.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.952 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.952 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.952 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.953 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.953 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.954 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g= 2023-10-17T19:42:18.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g=', 'Content-Length': '0'} 2023-10-17T19:42:18.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b576a554aae910e5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:18.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:18.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:18.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.958 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.958 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.958 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.959 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.959 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:18.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.960 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g= 2023-10-17T19:42:18.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XUUvD+rksJJi6CyAsyomWr3i09g=', 'Content-Length': '0'} 2023-10-17T19:42:18.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd70d2889e5c182c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.965 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.965 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.965 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.966 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.966 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.967 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c68dd638f9a273a1-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:18.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:18.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:18.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:18.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.972 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.973 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.973 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.973 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.974 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:18.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.974 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:18.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:18.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72178a3106b1126-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:18.975 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:18.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.979 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.980 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.980 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.980 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.981 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.981 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T19:42:18.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T19:42:18.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf007bf915b97f6-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:18.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.986 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:18.986 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.987 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.987 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.987 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.988 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T19:42:18.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T19:42:18.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8962e3cb529a16-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:18.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:18.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.992 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.993 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.993 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:18.993 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:18.994 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:18.994 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY= 2023-10-17T19:42:18.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY=', 'Content-Length': '0'} 2023-10-17T19:42:18.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f08c74f7687acf6-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:18.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:18.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:18.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:18.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:18.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:18.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:18.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:18.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:18.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:18.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:18.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:18.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:18.999 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:18.999 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.000 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.000 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.000 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.001 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY= 2023-10-17T19:42:19.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ke/sycXR8Ovmp0pQwgCotNmyVGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090426f443226552b-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.002 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:19.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.006 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.006 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.006 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.007 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.007 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.007 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T19:42:19.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T19:42:19.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eede1366b02c119-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.012 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.013 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.013 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.013 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.013 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.014 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g= 2023-10-17T19:42:19.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Rk1Y1UVg9p3VD31X3Yb5SYyaT5g=', 'Content-Length': '0'} 2023-10-17T19:42:19.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8b3feb5ff7652c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.019 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.019 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.020 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.020 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.020 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.021 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:19.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:19.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4a61375342e18e-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.026 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.026 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.027 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.027 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.027 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.028 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:19.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:19.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001738618c10372fef-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.033 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.033 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.033 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.034 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.034 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.035 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE= 2023-10-17T19:42:19.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE=', 'Content-Length': '0'} 2023-10-17T19:42:19.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d325ad86d0b3a718-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.039 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.039 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.039 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.040 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.040 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.041 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE= 2023-10-17T19:42:19.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3uBbSow2KMqPsaJgwUTFV6vDfGE=', 'Content-Length': '0'} 2023-10-17T19:42:19.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670b639f1d46dae6-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:19.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.045 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.045 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.046 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.046 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.047 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo= 2023-10-17T19:42:19.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo=', 'Content-Length': '0'} 2023-10-17T19:42:19.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024861dca3003a326-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.051 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.051 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.052 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.052 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.052 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.053 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo= 2023-10-17T19:42:19.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rMOCZ2jID3ksX+PtD7bUfdD3Zdo=', 'Content-Length': '0'} 2023-10-17T19:42:19.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537f119fa8c112d8-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.058 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.058 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.059 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.059 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.059 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.060 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:19.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:19.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbcd0b3fd60a6209-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.065 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.065 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.066 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.066 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.066 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.067 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:19.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:19.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000552fc3d011067be-00652ed6c5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.068 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.072 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.073 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.073 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.073 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.074 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.074 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T19:42:19.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T19:42:19.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f78a1cc33cd9c5-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.079 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.079 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.080 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.080 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.080 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.081 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T19:42:19.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T19:42:19.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee5a0a86f970b8c-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.086 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.086 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.086 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.087 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.087 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.087 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o= 2023-10-17T19:42:19.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o=', 'Content-Length': '0'} 2023-10-17T19:42:19.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3e258a1530917c-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.092 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.092 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.093 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.093 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.093 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.094 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o= 2023-10-17T19:42:19.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Z8n59+5IKHrv8LqdZG0MosNP65o=', 'Content-Length': '0'} 2023-10-17T19:42:19.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17526dd1ec40c47-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.099 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.099 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.100 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.100 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.100 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.101 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T19:42:19.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T19:42:19.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b53d185aa743b8b-00652ed6c5-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.106 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.106 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.107 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.107 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.107 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.108 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw= 2023-10-17T19:42:19.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Bfob2Ksco9G725sL7jsCinjxrtw=', 'Content-Length': '0'} 2023-10-17T19:42:19.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe57aa79e1978c4a-00652ed6c5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.112 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.112 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.113 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.113 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:33 GMT 2023-10-17T19:42:19.113 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.114 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g= 2023-10-17T19:42:19.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:33 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h8VHhiqOH551TKgI99N7OvLRI7g=', 'Content-Length': '0'} 2023-10-17T19:42:19.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcbbef0839a84755-00652ed6c5-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.119 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.119 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.119 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.120 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.120 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.120 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b72d3a0d17cfdc1-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.125 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.125 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.125 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.126 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.126 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.127 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk= 2023-10-17T19:42:19.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk=', 'Content-Length': '0'} 2023-10-17T19:42:19.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba205cc85cf0942-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.131 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.131 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.131 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.132 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.132 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.132 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk= 2023-10-17T19:42:19.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ij7p2Ph0jI3hAbHgrFWS9IpUcIk=', 'Content-Length': '0'} 2023-10-17T19:42:19.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a0718a1c7b16b4-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.133 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.137 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.137 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.137 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.137 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.138 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.138 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc= 2023-10-17T19:42:19.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc=', 'Content-Length': '0'} 2023-10-17T19:42:19.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180fe22a15533878-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.142 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.142 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.143 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.143 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.143 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.144 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc= 2023-10-17T19:42:19.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QFP99CxweftI7XnLP49zzkEUVKc=', 'Content-Length': '0'} 2023-10-17T19:42:19.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bad0e4eae859d1-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.148 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.148 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.149 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.149 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.149 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.150 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eee05fecddf5c8ab-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.155 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.155 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.156 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.156 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.156 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.157 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a714135991b71cf0-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.158 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:19.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.162 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.162 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.162 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.162 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.163 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.163 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T19:42:19.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T19:42:19.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004722b81b5441e24-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.168 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.168 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.168 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.169 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.169 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.169 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T19:42:19.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T19:42:19.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223709468da86845-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.174 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.174 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.174 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.175 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.175 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.176 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs= 2023-10-17T19:42:19.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs=', 'Content-Length': '0'} 2023-10-17T19:42:19.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb5fdf4f7939e32-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.180 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.181 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.181 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.181 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs= 2023-10-17T19:42:19.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Uub7uZjekDo1ovzpP6ywD2E8TVs=', 'Content-Length': '0'} 2023-10-17T19:42:19.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d89c71beaea7b5-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.182 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:19.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.186 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.186 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.187 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.187 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.187 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.188 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T19:42:19.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T19:42:19.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786ec2077d4fb071-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.192 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.192 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.193 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.193 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.193 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.194 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g= 2023-10-17T19:42:19.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sfOXl1QD4X7V7VKgUkvatTU20g=', 'Content-Length': '0'} 2023-10-17T19:42:19.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001396714b2576766b-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:19.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.199 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.199 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.199 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.200 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.200 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.201 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db2a44c5fb7f9e8-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:19.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.206 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.206 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.206 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.206 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.207 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.207 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5b356d6ab62f7d-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:19.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.211 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.211 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.212 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.212 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.212 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:19.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.213 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k= 2023-10-17T19:42:19.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k=', 'Content-Length': '0'} 2023-10-17T19:42:19.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab333afdb4018292-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:19.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.220 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.221 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.221 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:19.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.222 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k= 2023-10-17T19:42:19.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qOmiVqIboSUZWh8PguG0NvSMY2k=', 'Content-Length': '0'} 2023-10-17T19:42:19.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157b387ec387a965-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.222 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:19.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:19.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.226 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.226 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.226 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.226 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.227 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:19.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.227 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo= 2023-10-17T19:42:19.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo=', 'Content-Length': '0'} 2023-10-17T19:42:19.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5e2d9a308f40976-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:19.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.231 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.232 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.232 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.232 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.233 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:19.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.233 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo= 2023-10-17T19:42:19.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W60pxCifUrEUzQ3WVCL9zRhSEXo=', 'Content-Length': '0'} 2023-10-17T19:42:19.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a97cfdd5f46f30-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:19.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.238 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.238 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.239 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.239 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.239 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.240 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d4c055b9ec0143-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:19.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:19.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.245 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.245 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.246 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.246 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.246 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.247 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba0be29fdc73699-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.248 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:19.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.252 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.252 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.253 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.253 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.253 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.254 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T19:42:19.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T19:42:19.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382e1269d0be4d90-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.259 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.259 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.259 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.260 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.260 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.260 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T19:42:19.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T19:42:19.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8f7b158b794c2a-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.266 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.266 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.266 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.267 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.267 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.268 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g= 2023-10-17T19:42:19.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g=', 'Content-Length': '0'} 2023-10-17T19:42:19.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b87ea3c62392d8-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.272 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.273 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.273 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.273 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.274 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.274 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g= 2023-10-17T19:42:19.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+1ftQIpXn8dGwqlKF7cHVq0I28g=', 'Content-Length': '0'} 2023-10-17T19:42:19.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031909b04d3211495-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.275 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:19.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.280 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.280 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.280 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.280 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.281 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.281 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T19:42:19.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T19:42:19.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7dbd41b2323f80-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:19.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.286 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.286 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.287 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.287 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.287 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:19.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.288 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU= 2023-10-17T19:42:19.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pq86wNvafCWY2pGoApJr7zj9WcU=', 'Content-Length': '0'} 2023-10-17T19:42:19.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58cd1e4ea1f9232-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:19.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.292 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.293 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.293 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.294 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155af481b7698b22-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:19.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.300 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.300 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.300 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.300 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.301 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.301 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81fd388e27600d6-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:19.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.306 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.306 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.306 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.307 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.307 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:19.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.308 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4= 2023-10-17T19:42:19.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4=', 'Content-Length': '0'} 2023-10-17T19:42:19.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2e7b2c047a7f9e-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:19.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.312 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.312 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.313 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.313 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.313 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:19.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.314 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4= 2023-10-17T19:42:19.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GbMUiDAe0EV1LWrD4B+ixEok4D4=', 'Content-Length': '0'} 2023-10-17T19:42:19.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7084b4106eabc4f-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.315 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:19.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:19.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.319 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.319 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.319 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.320 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.320 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:19.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.320 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps= 2023-10-17T19:42:19.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps=', 'Content-Length': '0'} 2023-10-17T19:42:19.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deceb78d2fb1ec6f-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:19.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:19.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:19.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.325 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.325 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.325 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.325 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.326 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:19.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.326 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps= 2023-10-17T19:42:19.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8d4hNHvbzIpg0qkgF7zY8hdxpps=', 'Content-Length': '0'} 2023-10-17T19:42:19.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d3050a6095e6ac-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:19.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.331 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.331 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.331 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.332 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.332 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.332 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9386bf4c9900cf0-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:19.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:19.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.338 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.338 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.339 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.339 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.340 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7ad71c21989349-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.341 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:19.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.345 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.345 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.346 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.346 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.346 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.347 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T19:42:19.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T19:42:19.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6f220d768d0ece-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.351 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.352 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.352 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.352 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.352 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.353 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T19:42:19.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T19:42:19.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1d816a0dc2b1e4-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.358 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.358 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.359 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.359 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.359 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.360 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY= 2023-10-17T19:42:19.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY=', 'Content-Length': '0'} 2023-10-17T19:42:19.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6cadaa244877539-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.365 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.365 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.366 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.366 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.366 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.367 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY= 2023-10-17T19:42:19.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dTtY6nJltN8Cvunq0JMSiETquoY=', 'Content-Length': '0'} 2023-10-17T19:42:19.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c33a047b1f2a265-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.368 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:19.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.373 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.373 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.374 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.374 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.374 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.375 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T19:42:19.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T19:42:19.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139689e13212956d-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:19.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.380 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.380 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.380 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.381 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.381 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:19.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.381 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU= 2023-10-17T19:42:19.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hiuAfWQCZYJXOhsdC+hDjzOKKjU=', 'Content-Length': '0'} 2023-10-17T19:42:19.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013bf83f635e39ea9-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:19.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.387 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.387 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.387 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.387 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.388 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.388 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ec26f7d63487ed-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:19.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.394 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.394 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.395 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.395 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.395 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.396 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de056c84f440c87-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:19.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.400 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.401 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.401 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.401 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.402 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:19.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.402 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI= 2023-10-17T19:42:19.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI=', 'Content-Length': '0'} 2023-10-17T19:42:19.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d21227c40784d2-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:19.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.407 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.407 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.407 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.407 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.408 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:19.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.408 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI= 2023-10-17T19:42:19.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWeDAQUeOXEugAUbsZVqqB5x2pI=', 'Content-Length': '0'} 2023-10-17T19:42:19.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078256ee6e2940741-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.409 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:19.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:19.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.413 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.413 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.413 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.414 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.414 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:19.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.415 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo= 2023-10-17T19:42:19.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo=', 'Content-Length': '0'} 2023-10-17T19:42:19.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003effad1d8b7f2897-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:19.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:19.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:19.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.419 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.419 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.420 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.420 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.420 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:19.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.421 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo= 2023-10-17T19:42:19.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:knIXx9wKcRBavpSXr8DuTLhsGoo=', 'Content-Length': '0'} 2023-10-17T19:42:19.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c6794bd3746c0b-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.426 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.426 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.427 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.427 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.427 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.428 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d086f3f155ebad2-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:19.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:19.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.433 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.433 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.434 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.434 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.434 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.435 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faadae5ed672d7ba-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.436 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:19.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.441 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.441 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.441 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.441 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.442 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.442 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T19:42:19.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T19:42:19.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033814b698cf7811f-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.447 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.447 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.447 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.448 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.448 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.448 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T19:42:19.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T19:42:19.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a06001c78093a3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.453 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.453 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.454 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.454 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.454 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.455 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28= 2023-10-17T19:42:19.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28=', 'Content-Length': '0'} 2023-10-17T19:42:19.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a891d502da2a9451-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.460 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.460 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.460 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.460 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.461 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.461 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28= 2023-10-17T19:42:19.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UBZtt/8Z0IaC8paKwfgTzSVCq28=', 'Content-Length': '0'} 2023-10-17T19:42:19.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa1fd45e9716db3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.462 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:19.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.467 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.467 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.468 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.468 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.468 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.469 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T19:42:19.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T19:42:19.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0291277b055e30c-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:19.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.474 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.474 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.474 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.475 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.475 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:19.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.475 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw= 2023-10-17T19:42:19.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q/Wc9wIt7HX7tRmqRIn/zsgj2Vw=', 'Content-Length': '0'} 2023-10-17T19:42:19.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed564413bfdf7b7-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:19.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.481 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.481 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.481 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.481 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.482 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.482 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a24f7f0d1901146f-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:19.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.488 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.488 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.489 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.489 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.489 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.490 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dff311ce203747-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:19.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.494 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.495 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.495 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.495 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.496 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:19.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.496 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74= 2023-10-17T19:42:19.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74=', 'Content-Length': '0'} 2023-10-17T19:42:19.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b088521ac9b599-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:19.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.501 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.501 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.501 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.502 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.502 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:19.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.503 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74= 2023-10-17T19:42:19.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6V+9tE5zmGGuErZSeCpbOdlWi74=', 'Content-Length': '0'} 2023-10-17T19:42:19.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006750a63c2263388-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.504 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:19.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:19.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.507 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.508 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.508 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.508 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.508 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:19.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.509 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8= 2023-10-17T19:42:19.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8=', 'Content-Length': '0'} 2023-10-17T19:42:19.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e0a95a679c07e4-00652ed6c6-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:19.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:19.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:19.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.513 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.514 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.514 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.514 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.514 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:19.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.515 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8= 2023-10-17T19:42:19.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BX+y3soQd4DrE++ObD0YW/lqv+8=', 'Content-Length': '0'} 2023-10-17T19:42:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8fd6715252dee3-00652ed6c6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:19.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.520 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.520 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.520 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.521 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.521 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.522 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c439e3775824ff29-00652ed6c6-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:19.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:19.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.527 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.528 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.528 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.528 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:34 GMT 2023-10-17T19:42:19.528 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.529 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY= 2023-10-17T19:42:19.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:34 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kVbmtwR9/RlJrV/hpcnVWbg6gGY=', 'Content-Length': '0'} 2023-10-17T19:42:19.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76b0f7e6de3931b-00652ed6c6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:19.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:19.531 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hbmuog-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:19.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hbmuog-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:19.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:19.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-55 2023-10-17T19:42:19.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 8 2023-10-17T19:42:19.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:19.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:19.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:19.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:19.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding": "None", 2023-10-17T19:42:19.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_index": { 2023-10-17T19:42:19.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:19.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "Normal", 2023-10-17T19:42:19.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "normal": { 2023-10-17T19:42:19.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:19.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:19.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "logs": [ 2023-10-17T19:42:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:19.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:19.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:19.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:19.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 0, 2023-10-17T19:42:19.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 11, 2023-10-17T19:42:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:19.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:19.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:19.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:19.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:19.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:19.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:19.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:19.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:19.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:19.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:19.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-55 2023-10-17T19:42:19.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:19.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:19.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:19.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:19.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:19.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:35.190+0000 7f4fce5df780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:19.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:40.191+0000 7f4fce5df780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:19.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:45.191+0000 7f4fce5df780 1 bucket sync caught up with empty source 2023-10-17T19:42:19.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:45.191+0000 7f4fce5df780 0 bucket checkpoint complete 2023-10-17T19:42:19.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-55 2023-10-17T19:42:19.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:19.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:19.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:19.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:19.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:19.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:45.318+0000 7fa6f3413780 1 bucket sync caught up with source: 2023-10-17T19:42:19.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:19.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000002.150.6, , , 00000000001.114.6, 00000000001.50.6, , , 00000000002.225.6, , 00000000002.410.6] 2023-10-17T19:42:19.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:47:45.318+0000 7fa6f3413780 0 bucket checkpoint complete 2023-10-17T19:42:19.557 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:19.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:19.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.560 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.561 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.561 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.561 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.562 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.562 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8= 2023-10-17T19:42:19.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8=', 'Content-Length': '0'} 2023-10-17T19:42:19.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b2106f5be232f1-00652ed6d1-1630-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/ 2023-10-17T19:42:19.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.567 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.567 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.567 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.568 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.568 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.569 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8= 2023-10-17T19:42:19.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7IX+qFeJHQDiWnavJq9dGCCdkl8=', 'Content-Length': '0'} 2023-10-17T19:42:19.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046f2d888c701c053-00652ed6d1-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.569 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:19.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.574 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.574 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.575 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.575 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.575 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:19.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.576 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T19:42:19.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T19:42:19.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da90932d2938ae1-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.577 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:19.578 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:19.578 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:19.578 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:19.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-17T19:42:19.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-17T19:42:19.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-17T19:42:19.580 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-17T19:42:19.580 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:19.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.584 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.585 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.585 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.585 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.585 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:19.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.586 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T19:42:19.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T19:42:19.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad0a81d6659045d-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-17T19:42:19.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-17T19:42:19.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-17T19:42:19.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-17T19:42:19.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-17T19:42:19.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-17T19:42:19.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-17T19:42:19.590 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-17T19:42:19.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.594 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.594 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.594 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.595 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.595 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:19.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.596 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T19:42:19.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T19:42:19.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15b59dba366a2c5-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?versions&max-keys=999 2023-10-17T19:42:19.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.601 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.601 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.602 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.602 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.602 INFO:tasks.rgw_multisite_tests:/hbmuog-55/?versions 2023-10-17T19:42:19.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.603 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E= 2023-10-17T19:42:19.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Mq+ekT+nfZrXsLt+eUDJtfjEQ8E=', 'Content-Length': '0'} 2023-10-17T19:42:19.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c956598b62f1562-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55999falseanulltrue2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.604 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:19.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.608 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.609 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.609 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.609 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.610 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.610 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T19:42:19.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T19:42:19.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526525169338971d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.615 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.616 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.616 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.616 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.616 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.617 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T19:42:19.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T19:42:19.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042119550987ea522-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.622 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.622 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.622 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.623 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.623 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.624 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA= 2023-10-17T19:42:19.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA=', 'Content-Length': '0'} 2023-10-17T19:42:19.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8167fe159fba5e-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.630 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.631 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.631 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.631 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.632 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.632 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA= 2023-10-17T19:42:19.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IP2BDdCIycbu2G1dyPCoFKnYQjA=', 'Content-Length': '0'} 2023-10-17T19:42:19.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005971789b6b85d1ca-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.633 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:19.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.638 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.638 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.639 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.639 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.639 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.640 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T19:42:19.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T19:42:19.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103cbce04635ccb3-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a?versionId=null 2023-10-17T19:42:19.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.645 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.645 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.646 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.646 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.646 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a?versionId=null 2023-10-17T19:42:19.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.647 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM= 2023-10-17T19:42:19.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OttnNC9Au3qCp+wM1O8Kie/HpkM=', 'Content-Length': '0'} 2023-10-17T19:42:19.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2f0b120445c27c-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:19.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.652 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.652 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.652 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.652 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.653 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.653 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb224465411f255-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.658 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.659 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.659 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.659 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.660 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.660 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2327f02599ac8ca-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.665 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.665 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.665 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.666 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.666 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:19.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.667 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4= 2023-10-17T19:42:19.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4=', 'Content-Length': '0'} 2023-10-17T19:42:19.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300124897607a0c8-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.671 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.672 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.672 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.672 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.672 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:19.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.673 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4= 2023-10-17T19:42:19.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vlQ7lLtFTQo9b/Lygvzo+WlBUh4=', 'Content-Length': '0'} 2023-10-17T19:42:19.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b6c45b4150a375-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.674 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-17T19:42:19.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:19.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.678 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.678 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.679 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.679 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.679 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:19.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.680 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8= 2023-10-17T19:42:19.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8=', 'Content-Length': '0'} 2023-10-17T19:42:19.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b45d3143d33f60e-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/a 2023-10-17T19:42:19.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/a 2023-10-17T19:42:19.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/a 2023-10-17T19:42:19.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.685 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.685 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.685 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.686 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.686 INFO:tasks.rgw_multisite_tests:/hbmuog-55/a 2023-10-17T19:42:19.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.687 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8= 2023-10-17T19:42:19.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:di0D3N5cMSW3I6Vk2eAVr2nFFZ8=', 'Content-Length': '0'} 2023-10-17T19:42:19.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47686cdb141dc61-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.692 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.692 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.692 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.692 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.693 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.693 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b883eff959f2d513-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=a 2023-10-17T19:42:19.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=a 2023-10-17T19:42:19.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.699 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.700 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.701 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.701 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca165f99dd66d04b-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55a1000falsea2023-10-17T18:44:35.729Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.703 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:19.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.707 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.707 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.707 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.708 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.708 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.708 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T19:42:19.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T19:42:19.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21b4146ae4b59e1-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.713 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.714 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.714 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.714 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.715 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.715 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T19:42:19.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T19:42:19.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043fc6b0dd1c0686-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.720 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.720 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.720 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.721 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.721 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.722 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE= 2023-10-17T19:42:19.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE=', 'Content-Length': '0'} 2023-10-17T19:42:19.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ccb1940c0d7c92-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.727 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.727 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.728 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.728 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.728 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.729 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE= 2023-10-17T19:42:19.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jHUmH5jzOCfW8Y7CDgjDDHWKLUE=', 'Content-Length': '0'} 2023-10-17T19:42:19.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b586543a35f10fcc-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.730 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:19.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.734 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.735 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.735 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.736 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.736 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.737 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T19:42:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T19:42:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cedf5e7299e94948-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b?versionId=null 2023-10-17T19:42:19.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.742 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.742 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.742 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.743 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.743 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b?versionId=null 2023-10-17T19:42:19.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.744 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw= 2023-10-17T19:42:19.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cPC4MMaRAJUNuKjh7OwrZB1m9Gw=', 'Content-Length': '0'} 2023-10-17T19:42:19.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a1194a5883d191-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:19.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.748 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.749 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.749 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.749 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.750 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.750 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b183cdc7c33bde-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.756 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.756 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.756 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.756 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.757 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.757 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755f76514725a8b2-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.762 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.762 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.762 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.763 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.763 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:19.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.763 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU= 2023-10-17T19:42:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU=', 'Content-Length': '0'} 2023-10-17T19:42:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030994c0f7e3ca020-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.768 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.768 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.768 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.769 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.769 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:19.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.769 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU= 2023-10-17T19:42:19.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mActvrIdSRdxJGOlQlufXsg2xnU=', 'Content-Length': '0'} 2023-10-17T19:42:19.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50354faa7761a7b-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.770 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-17T19:42:19.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.774 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.774 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.775 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.775 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.775 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:19.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.776 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw= 2023-10-17T19:42:19.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw=', 'Content-Length': '0'} 2023-10-17T19:42:19.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274b4693e0e08cbc-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/b 2023-10-17T19:42:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/b 2023-10-17T19:42:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/b 2023-10-17T19:42:19.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.780 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.780 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.781 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.781 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.781 INFO:tasks.rgw_multisite_tests:/hbmuog-55/b 2023-10-17T19:42:19.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.782 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw= 2023-10-17T19:42:19.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qlsT69sPvUjykiAcFFWXQvhsoWw=', 'Content-Length': '0'} 2023-10-17T19:42:19.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001965f80a85676e3c-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.787 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.787 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.787 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.788 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.788 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.789 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007077fab4e7fad2ed-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=b 2023-10-17T19:42:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=b 2023-10-17T19:42:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.794 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.794 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.795 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.795 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.795 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.796 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6db61aede6a3676-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55b1000falseb2023-10-17T18:44:35.788Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.798 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:19.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.802 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.802 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.802 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.803 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.803 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.804 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T19:42:19.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T19:42:19.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092122c5305f1aa5d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.809 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.809 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.809 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.810 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.810 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.811 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T19:42:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T19:42:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e3de75bab4abe54-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.816 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.816 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.816 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.817 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.817 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc= 2023-10-17T19:42:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc=', 'Content-Length': '0'} 2023-10-17T19:42:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee4f70e189c3708-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.823 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.823 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.824 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.824 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.824 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.825 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc= 2023-10-17T19:42:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hoQAMO/Q3tPjtaH1HmVpukrS1Cc=', 'Content-Length': '0'} 2023-10-17T19:42:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037fde36f1ac647d-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.826 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:19.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.830 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.830 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.830 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.831 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.831 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.833 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T19:42:19.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T19:42:19.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3fe020d0286c9c-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c?versionId=null 2023-10-17T19:42:19.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.839 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.839 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.839 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.840 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.840 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c?versionId=null 2023-10-17T19:42:19.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.841 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0= 2023-10-17T19:42:19.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j4h7VQzJrDPRhLDlEp9JYmeMyk0=', 'Content-Length': '0'} 2023-10-17T19:42:19.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099a30dd2d938261-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.846 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.846 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.846 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.847 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.847 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.848 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bf7c0e2fd529b03-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.853 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.853 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.854 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.854 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.854 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.855 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ee874b3c302c73-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.859 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.859 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.860 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.860 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.860 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.861 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ= 2023-10-17T19:42:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ=', 'Content-Length': '0'} 2023-10-17T19:42:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebef9aa44f68638c-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.865 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.865 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.866 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.866 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.866 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.867 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ= 2023-10-17T19:42:19.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EAf+vOOYi91RUJ9spfrCtqVBNyQ=', 'Content-Length': '0'} 2023-10-17T19:42:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1c01e4e11117a0-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.868 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-17T19:42:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.871 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.872 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.872 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.872 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.873 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.873 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY= 2023-10-17T19:42:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY=', 'Content-Length': '0'} 2023-10-17T19:42:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68d9b3e3b63aa9a-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/c 2023-10-17T19:42:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/c 2023-10-17T19:42:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/c 2023-10-17T19:42:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.877 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.878 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.878 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.878 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.879 INFO:tasks.rgw_multisite_tests:/hbmuog-55/c 2023-10-17T19:42:19.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.879 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY= 2023-10-17T19:42:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Dgw8f0fokjSxFif93F46ut7N1FY=', 'Content-Length': '0'} 2023-10-17T19:42:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbe036514adc240-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.884 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.885 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.885 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.885 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.885 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.886 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbc129615e22c38-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=c 2023-10-17T19:42:19.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=c 2023-10-17T19:42:19.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.892 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.892 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.893 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.893 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.893 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.894 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6e76237462ae8c-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55c1000falsec2023-10-17T18:44:35.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.895 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.900 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.901 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.901 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.901 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T19:42:19.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T19:42:19.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067467a7feb109b2d-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.907 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.907 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.907 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.908 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.908 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.908 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T19:42:19.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T19:42:19.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ce24bfcc64e261-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.914 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.914 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.914 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.915 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.915 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.916 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU= 2023-10-17T19:42:19.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU=', 'Content-Length': '0'} 2023-10-17T19:42:19.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70b41858bcaf99f-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.920 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.921 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.921 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.921 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.922 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.922 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU= 2023-10-17T19:42:19.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CxyrbXY+TKfxb7KzAdHwo/vjfMU=', 'Content-Length': '0'} 2023-10-17T19:42:19.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2743754ee007a3-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.928 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.928 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.928 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.929 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.929 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.929 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T19:42:19.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T19:42:19.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a44b44f6036c4f-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d?versionId=null 2023-10-17T19:42:19.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.935 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.935 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.935 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.936 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.936 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d?versionId=null 2023-10-17T19:42:19.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.937 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM= 2023-10-17T19:42:19.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BSIHBTPA9h3bDaq0YfFdDx7kigM=', 'Content-Length': '0'} 2023-10-17T19:42:19.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e5b84201b1b33f-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.941 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.942 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.942 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.942 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.943 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.943 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea471cdc1c82049-00652ed6d1-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.948 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.949 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.949 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.949 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.950 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.950 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI= 2023-10-17T19:42:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TS6BTljV2PRmKF6TSyLvl1t+bAI=', 'Content-Length': '0'} 2023-10-17T19:42:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a193a557426003a-00652ed6d1-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.955 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.955 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.955 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.956 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.956 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.957 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w= 2023-10-17T19:42:19.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w=', 'Content-Length': '0'} 2023-10-17T19:42:19.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d865a24e8f8b648-00652ed6d1-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.961 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:19.961 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.961 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.962 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:45 GMT 2023-10-17T19:42:19.962 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.963 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w= 2023-10-17T19:42:19.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CkdejJOU1dzaGjsBngoXssNw+5w=', 'Content-Length': '0'} 2023-10-17T19:42:19.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1657d90769aa48b-00652ed6d1-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.963 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-17T19:42:19.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.967 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.968 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.968 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.968 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:19.968 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.969 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM= 2023-10-17T19:42:19.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM=', 'Content-Length': '0'} 2023-10-17T19:42:19.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047cfbdd590327078-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/d 2023-10-17T19:42:19.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/d 2023-10-17T19:42:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/d 2023-10-17T19:42:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.973 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.974 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.974 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.974 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:19.975 INFO:tasks.rgw_multisite_tests:/hbmuog-55/d 2023-10-17T19:42:19.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.975 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM= 2023-10-17T19:42:19.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:U47LHFNZ4ho1lk+NvxDuhh2LqaM=', 'Content-Length': '0'} 2023-10-17T19:42:19.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2583f90960c04fb-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.980 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.980 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.981 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.981 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:19.981 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.982 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:19.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:19.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d27deff34a54f0a-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=d 2023-10-17T19:42:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=d 2023-10-17T19:42:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:19.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.987 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.988 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.988 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.988 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:19.989 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:19.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.989 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e57a306fdd8d81-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55d1000falsed2023-10-17T18:44:35.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:19.991 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:19.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:19.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:19.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:19.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:19.995 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:19.995 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.995 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:19.995 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:19.996 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:19.996 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T19:42:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T19:42:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abcd46ca349291cd-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:19.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:19.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:19.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.001 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.001 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.001 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.002 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.002 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.003 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T19:42:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T19:42:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000506cc5dee8b73fa-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.008 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.008 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.009 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.009 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.009 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.010 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4= 2023-10-17T19:42:20.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4=', 'Content-Length': '0'} 2023-10-17T19:42:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5f63ed59b61af0-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:20.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.015 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.016 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.016 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.016 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.017 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4= 2023-10-17T19:42:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NlgPZmi07Ng/LcwitoEBma1RlP4=', 'Content-Length': '0'} 2023-10-17T19:42:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe49dceb4d6ed33-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.018 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:20.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.022 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.023 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.023 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.023 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.024 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.024 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T19:42:20.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T19:42:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5d7a08a67bb9ad-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e?versionId=null 2023-10-17T19:42:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.029 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.029 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.030 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.030 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.030 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e?versionId=null 2023-10-17T19:42:20.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.031 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0= 2023-10-17T19:42:20.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N9BO2GBymze3lfI22fodKANKMf0=', 'Content-Length': '0'} 2023-10-17T19:42:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34f35f11d8353e6-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:20.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.038 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.038 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.039 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.039 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.039 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.040 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e29aaa68a8ac21-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:20.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.045 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.046 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.047 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.047 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2905cee1d55bd7-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:20.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.052 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.052 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.052 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.053 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.053 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:20.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.054 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg= 2023-10-17T19:42:20.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg=', 'Content-Length': '0'} 2023-10-17T19:42:20.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e026a8cd13916a50-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:20.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.059 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.059 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.059 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.059 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.060 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:20.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.060 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg= 2023-10-17T19:42:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Tbfvw5Z/ByNOwCb/sAVdM4t/CGg=', 'Content-Length': '0'} 2023-10-17T19:42:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f97bf7728d78f0a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.061 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-17T19:42:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:20.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.065 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.065 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.065 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.066 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.066 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:20.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.067 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc= 2023-10-17T19:42:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc=', 'Content-Length': '0'} 2023-10-17T19:42:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c7b888552ba585-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/e 2023-10-17T19:42:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/e 2023-10-17T19:42:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/e 2023-10-17T19:42:20.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.071 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.072 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.072 INFO:tasks.rgw_multisite_tests:/hbmuog-55/e 2023-10-17T19:42:20.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.073 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc= 2023-10-17T19:42:20.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:f6uvljrfJFR78nuPJbP2iu4w6zc=', 'Content-Length': '0'} 2023-10-17T19:42:20.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e86dd4b364c619-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:20.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.078 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.078 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.078 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.079 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.079 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.080 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1016a2223f25b7-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=e 2023-10-17T19:42:20.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=e 2023-10-17T19:42:20.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.085 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.085 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.085 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.086 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.086 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.087 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003579807aa1de8d0e-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55e1000falsee2023-10-17T18:45:07.811Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.088 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:20.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.092 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.093 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.093 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.093 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.093 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.094 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T19:42:20.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T19:42:20.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f91dad0835db16-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.099 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.099 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.099 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.100 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.100 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.101 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T19:42:20.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T19:42:20.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da9da4d9dcf7509-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.106 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.106 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.107 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.107 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.107 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.108 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ= 2023-10-17T19:42:20.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ=', 'Content-Length': '0'} 2023-10-17T19:42:20.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae1bcb9c8d51f95-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.113 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.113 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.113 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.114 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.114 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.115 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ= 2023-10-17T19:42:20.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EJ8Qhb4mP31QqspAM/xysgBBamQ=', 'Content-Length': '0'} 2023-10-17T19:42:20.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9f2ccc65f2653a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.116 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:20.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.120 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.120 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.120 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.120 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.121 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.121 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T19:42:20.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T19:42:20.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e616d9ea347d06-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f?versionId=null 2023-10-17T19:42:20.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.126 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.127 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.127 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.127 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.128 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f?versionId=null 2023-10-17T19:42:20.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.128 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so= 2023-10-17T19:42:20.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GhVCZDbUSZHI/6mJz1e8ajjs7so=', 'Content-Length': '0'} 2023-10-17T19:42:20.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022398fa3d924d629-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:20.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.134 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.134 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.134 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.135 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.135 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.136 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6146685d25fa0d0-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:20.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.141 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.141 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.141 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.141 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.142 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.142 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cbb415a00944de9-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:20.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.147 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.147 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.147 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.148 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.148 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:20.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.148 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0= 2023-10-17T19:42:20.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0=', 'Content-Length': '0'} 2023-10-17T19:42:20.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08f11755f4d40d2-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:20.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.153 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.153 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.154 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.154 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.155 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:20.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.155 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0= 2023-10-17T19:42:20.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dXpFsM7DrnQKZo5jAKla5+NamF0=', 'Content-Length': '0'} 2023-10-17T19:42:20.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd97810354d2e52c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.156 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-17T19:42:20.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:20.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.159 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.160 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.160 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.160 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.161 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:20.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.161 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY= 2023-10-17T19:42:20.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY=', 'Content-Length': '0'} 2023-10-17T19:42:20.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf68f9960c12fa88-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/f 2023-10-17T19:42:20.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/f 2023-10-17T19:42:20.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/f 2023-10-17T19:42:20.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.165 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.166 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.166 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.166 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.167 INFO:tasks.rgw_multisite_tests:/hbmuog-55/f 2023-10-17T19:42:20.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.167 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY= 2023-10-17T19:42:20.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:eK9S+ie2ps6rmkFmY7J9XcHwVpY=', 'Content-Length': '0'} 2023-10-17T19:42:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008183b595d6004e1a-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.172 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.173 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.173 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.174 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b725ad6438a1559a-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=f 2023-10-17T19:42:20.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=f 2023-10-17T19:42:20.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.179 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.180 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.181 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.181 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7ac35083727adb-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55f1000falsef2023-10-17T18:45:07.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.182 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:20.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.187 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.187 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.187 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.188 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.188 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.189 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T19:42:20.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T19:42:20.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091fe22073ee20df6-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.193 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.194 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.194 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.194 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.195 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.195 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T19:42:20.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T19:42:20.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caf522369b6127ec-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.200 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.201 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.201 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.201 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.202 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.202 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU= 2023-10-17T19:42:20.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU=', 'Content-Length': '0'} 2023-10-17T19:42:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb9290ec389f004-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.207 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.207 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.207 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.208 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.208 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.209 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU= 2023-10-17T19:42:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WqtOv5CozVM4VpQ4xp/TauN4QPU=', 'Content-Length': '0'} 2023-10-17T19:42:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a15bfd46ba6b63-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:20.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.214 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.214 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.215 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.215 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.215 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.216 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T19:42:20.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T19:42:20.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ead69aa1b4c8f1-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g?versionId=null 2023-10-17T19:42:20.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.221 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.221 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.221 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.221 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.222 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g?versionId=null 2023-10-17T19:42:20.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.222 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs= 2023-10-17T19:42:20.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:gXxVGcfUzvtiJmcvFhtiZ+ijDqs=', 'Content-Length': '0'} 2023-10-17T19:42:20.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483daeeb72586345-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:20.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.227 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.228 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.228 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.228 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.228 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.229 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b6d328cce702822-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:20.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.234 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.234 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.234 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.235 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.235 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.236 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf06eb6cc27ba5a4-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:20.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.241 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.242 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.242 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:20.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.242 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI= 2023-10-17T19:42:20.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI=', 'Content-Length': '0'} 2023-10-17T19:42:20.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4681e75edb1f659-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.250 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.250 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.251 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.251 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.251 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:20.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.252 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI= 2023-10-17T19:42:20.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rccTCs2KLtFwGrB42KNBm0gcjsI=', 'Content-Length': '0'} 2023-10-17T19:42:20.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080efcff4fcbecb9e-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.253 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-17T19:42:20.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.257 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.257 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.258 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.258 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.258 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:20.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.259 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A= 2023-10-17T19:42:20.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A=', 'Content-Length': '0'} 2023-10-17T19:42:20.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ffb02d111155e4-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/g 2023-10-17T19:42:20.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/g 2023-10-17T19:42:20.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/g 2023-10-17T19:42:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.263 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.264 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.264 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.264 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.265 INFO:tasks.rgw_multisite_tests:/hbmuog-55/g 2023-10-17T19:42:20.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.265 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A= 2023-10-17T19:42:20.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ref7UdV9W5rIZ+1oL1nDDM2VE2A=', 'Content-Length': '0'} 2023-10-17T19:42:20.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010181189f7ef020c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:20.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.271 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.271 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.271 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.272 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.272 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.273 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c24d642dc4c93a9-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=g 2023-10-17T19:42:20.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=g 2023-10-17T19:42:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.278 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.279 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.279 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.279 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.279 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.280 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb76d5d49fce50b-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55g1000falseg2023-10-17T18:45:07.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.281 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.286 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.286 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.286 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.287 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.287 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.288 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T19:42:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T19:42:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd990e2ac0a770c-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.292 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.293 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.294 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.294 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T19:42:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T19:42:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b0ae97fc6bfbc5-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.299 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.300 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.300 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.300 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.301 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.301 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw= 2023-10-17T19:42:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw=', 'Content-Length': '0'} 2023-10-17T19:42:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000074a932944dccdfa-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.306 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.306 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.307 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.307 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.307 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.308 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw= 2023-10-17T19:42:20.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:C1Gbq/xQzU4f5UpQHTgB8xX8YZw=', 'Content-Length': '0'} 2023-10-17T19:42:20.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d297f060a2bf95-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.309 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:20.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.313 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.313 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.314 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.314 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.314 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.315 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T19:42:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T19:42:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f250f3b37829dc4-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h?versionId=null 2023-10-17T19:42:20.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.320 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.320 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.320 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.321 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.321 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h?versionId=null 2023-10-17T19:42:20.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.321 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4= 2023-10-17T19:42:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1jj9jJ/LEHLwFH2vU0EwV+30FH4=', 'Content-Length': '0'} 2023-10-17T19:42:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794f8af1451982fc-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:20.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.327 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.327 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.327 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.328 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.328 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.328 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099929e60eb6fd41b-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:20.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.334 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.334 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.335 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.335 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.335 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.336 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e4de7a0e00e2156-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:20.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.340 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.341 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.341 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.341 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.342 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:20.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.342 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k= 2023-10-17T19:42:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k=', 'Content-Length': '0'} 2023-10-17T19:42:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001948caa28ebc160e-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.347 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:20.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.348 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.348 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:20.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.349 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k= 2023-10-17T19:42:20.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FiMsqJ0NQ+qY33+Jg6tlV00q23k=', 'Content-Length': '0'} 2023-10-17T19:42:20.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad8255c4d993591c-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.350 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-17T19:42:20.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:20.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.354 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.354 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.354 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.355 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.355 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:20.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.356 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0= 2023-10-17T19:42:20.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0=', 'Content-Length': '0'} 2023-10-17T19:42:20.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-55:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec26ef752fcddb2f-00652ed6d2-1630-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/h 2023-10-17T19:42:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/h 2023-10-17T19:42:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/h 2023-10-17T19:42:20.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.360 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.360 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.361 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.361 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.361 INFO:tasks.rgw_multisite_tests:/hbmuog-55/h 2023-10-17T19:42:20.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.362 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0= 2023-10-17T19:42:20.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4ZrjqZHXu9bvvPCqMhA9BFOiND0=', 'Content-Length': '0'} 2023-10-17T19:42:20.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:45:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041797f1099257606-00652ed6d2-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:20.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:20.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.367 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.368 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.369 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.369 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a590c5cef114d1da-00652ed6d2-1630-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/ 2023-10-17T19:42:20.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/ 2023-10-17T19:42:20.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-55/?prefix=h 2023-10-17T19:42:20.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:20.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-55/?prefix=h 2023-10-17T19:42:20.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.374 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:20.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.375 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:47:46 GMT 2023-10-17T19:42:20.376 INFO:tasks.rgw_multisite_tests:/hbmuog-55/ 2023-10-17T19:42:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.376 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE= 2023-10-17T19:42:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:47:46 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:S4D1iSjI9Np4ZtG18BVVYvUiZEE=', 'Content-Length': '0'} 2023-10-17T19:42:20.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb055c670e01e3a-00652ed6d2-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:47:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-55h1000falseh2023-10-17T18:45:07.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:20.377 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-55 zones={a2, a1} 2023-10-17T19:42:20.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:20.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.379 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:46.979+0000 7f5e948b0700 -1 req 4164692305357405680 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-1:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.1,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.380 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:46.979+0000 7f5ea78d6700 -1 req 8993893449878962159 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-30:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.30,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.380 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.031+0000 7f5e9a0bb700 -1 req 16793100318319100887 0.000999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-43:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.8,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.380 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.031+0000 7f5e7987a700 -1 req 2406639096585865596 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-4:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.381 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.076+0000 7f5e57035700 -1 req 278019542600793309 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-41:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.381 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.081+0000 7f5e4f826700 -1 req 3692153941252346995 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-23:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.23,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:20.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding": "None", 2023-10-17T19:42:20.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_index": { 2023-10-17T19:42:20.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "Normal", 2023-10-17T19:42:20.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "normal": { 2023-10-17T19:42:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "logs": [ 2023-10-17T19:42:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:20.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:20.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:20.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:20.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.398 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.331+0000 7f5e67055700 -1 req 15020052301451087982 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-2:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.398 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.331+0000 7f5e40808700 -1 req 7782735937252187366 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-25:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.25,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.398 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.331+0000 7f5e5c03f700 -1 req 5516016212627392655 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-40:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.5,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.399 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.331+0000 7f5e54830700 -1 req 14088950252049687166 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-39:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.4,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.399 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.333+0000 7f5e31feb700 -1 req 7998560759297130222 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-19:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.19,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.399 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.333+0000 7f5e46814700 -1 req 16492183496669393951 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-22:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.22,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.400 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.333+0000 7f5e227cc700 -1 req 3758129637006002652 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-6:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.6,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.400 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.333+0000 7f5e3e003700 -1 req 17395515403808682674 0.000999972s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-24:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.24,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.400 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.426+0000 7f5e2c7e0700 -1 req 7477457638819705725 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-42:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.7,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.400 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.427+0000 7f5e277d6700 -1 req 8245506584526227205 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-3:4c84373e-d451-49bb-a8c9-c37d5e5e2160.4373.3,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.401 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T18:47:47.427+0000 7f5e1c7c0700 -1 req 7234563436123005083 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=4c84373e-d451-49bb-a8c9-c37d5e5e2160,az=0},d={b=hbmuog-37:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.2,z=8f644c5a-db9c-4565-b32b-e3a6790e3bcc,az=0}} returned ret=-2 2023-10-17T19:42:20.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hbmuog-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hbmuog-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding": "None", 2023-10-17T19:42:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_index": { 2023-10-17T19:42:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "Normal", 2023-10-17T19:42:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "normal": { 2023-10-17T19:42:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "logs": [ 2023-10-17T19:42:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 1, 2023-10-17T19:42:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "type": "InIndex", 2023-10-17T19:42:20.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "in_index": { 2023-10-17T19:42:20.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "gen": 2, 2023-10-17T19:42:20.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "layout": { 2023-10-17T19:42:20.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 3, 2023-10-17T19:42:20.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hash_type": "Mod" 2023-10-17T19:42:20.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.416 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:20.416 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.417 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:20.417 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2023-10-17T19:42:20.417 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:20.419 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:20.419 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:20.419 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:20.419 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 1667, in test_zap_init_bucket_sync_run 2023-10-17T19:42:20.420 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-17T19:42:20.420 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-17T19:42:20.420 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-17T19:42:20.421 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-17T19:42:20.421 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-17T19:42:20.421 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-17T19:42:20.422 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-17T19:42:20.422 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-17T19:42:20.422 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:20.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:20.426 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:20.426 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.426 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.427 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:54:25 GMT 2023-10-17T19:42:20.427 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:20.428 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:th/h0vYaiC9QEIukF9vy6Kkfub0= 2023-10-17T19:42:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:54:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:th/h0vYaiC9QEIukF9vy6Kkfub0=', 'Content-Length': '0'} 2023-10-17T19:42:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000054eed05f41aa77d2-00652ed861-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:54:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:20.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hbmuog-59 2023-10-17T19:42:20.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:20.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:20.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568865.677043_735.1", 2023-10-17T19:42:20.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T18:54:25.677043Z" 2023-10-17T19:42:20.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T18:17:34.606901Z" 2023-10-17T19:42:20.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:20.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:20.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:20.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:25.879+0000 7fd67ba8b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:25.880+0000 7fd67ba8b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:25Z" 2023-10-17T19:42:20.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.463 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.463 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.464 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.464 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.464 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.465 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.465 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.466 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.466 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.466 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.467 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.468 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.469 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.469 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.470 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.470 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.471 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.471 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.473 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.473 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.474 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.474 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.474 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.475 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.476 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.477 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.477 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.478 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.478 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.478 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.479 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.480 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.480 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.480 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.481 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.481 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.482 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.482 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.482 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.483 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.483 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.483 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.484 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:25Z" 2023-10-17T19:42:20.484 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.484 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:31.007+0000 7f7977856780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:31.008+0000 7f7977856780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:31Z" 2023-10-17T19:42:20.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.507 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.507 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.507 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.508 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.508 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.509 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.509 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.509 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.510 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.510 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.511 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.512 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.512 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.513 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.513 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.513 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.514 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.516 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.516 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.517 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.517 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.517 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.518 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.519 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.520 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.520 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.520 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.521 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.521 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.522 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.523 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.523 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.524 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.524 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.524 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.524 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.525 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.525 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.525 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.526 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:31Z" 2023-10-17T19:42:20.526 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.526 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:36.132+0000 7f6c858af780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:36.133+0000 7f6c858af780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:36Z" 2023-10-17T19:42:20.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.552 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.553 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.553 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.554 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.554 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.555 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.555 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.557 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.558 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.559 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.559 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.559 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.560 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.561 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.561 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.562 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.562 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.563 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.563 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.563 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.564 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.565 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.566 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.566 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.566 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.567 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.567 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.568 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.569 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.569 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.570 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.570 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.571 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.571 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.571 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.571 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.572 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.572 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:36Z" 2023-10-17T19:42:20.572 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.573 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:41.262+0000 7f998ad9d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:41.263+0000 7f998ad9d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:41Z" 2023-10-17T19:42:20.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.598 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.598 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.599 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.599 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.599 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.600 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.600 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.600 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.601 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.601 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.602 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.603 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.604 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.604 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.604 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.604 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.605 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.606 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.607 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.607 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.607 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.607 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.608 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.609 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.610 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.610 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.610 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.611 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.612 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.613 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.613 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.614 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.614 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.614 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.615 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.615 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.615 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.616 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.616 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:41Z" 2023-10-17T19:42:20.616 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.617 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:46.389+0000 7f076fc01780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:46.390+0000 7f076fc01780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:46Z" 2023-10-17T19:42:20.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.641 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.641 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.642 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.642 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.642 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.643 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.643 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.644 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.644 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.644 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.646 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.646 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.647 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.647 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.648 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.648 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.648 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.649 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.649 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.650 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.650 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.650 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.651 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.651 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.651 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.653 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.654 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.655 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.656 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.656 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.657 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.657 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.657 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.658 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.659 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.660 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.660 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.661 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.661 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.662 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.662 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.663 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:46Z" 2023-10-17T19:42:20.663 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.663 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:51.517+0000 7fe7e2081780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:51.518+0000 7fe7e2081780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:51Z" 2023-10-17T19:42:20.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.688 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.689 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.689 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.690 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.690 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.691 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.691 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.691 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.692 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.692 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.694 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.694 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.694 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.695 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.697 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.698 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.698 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.699 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.699 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.700 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.701 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.701 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.702 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.702 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.703 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.704 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.705 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.705 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.705 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.706 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.706 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.706 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.706 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.707 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.707 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:51Z" 2023-10-17T19:42:20.707 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.707 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:56.643+0000 7f9487f24780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:54:56.644+0000 7f9487f24780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:54:56Z" 2023-10-17T19:42:20.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.731 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.732 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.732 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.732 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.733 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.733 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.734 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.734 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.734 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.735 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.736 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.737 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.738 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.738 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.738 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.741 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.741 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.742 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.742 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.742 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.744 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.745 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.745 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.746 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.746 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.746 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.747 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.748 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.749 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.749 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.749 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.750 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.750 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.750 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.751 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.751 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.751 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:54:56Z" 2023-10-17T19:42:20.752 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.752 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:01.771+0000 7fab4a045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:01.772+0000 7fab4a045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:01Z" 2023-10-17T19:42:20.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.777 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.777 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.777 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.778 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.778 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.779 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.780 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.780 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.780 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.782 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.783 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.784 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.784 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.786 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.788 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.789 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.789 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.790 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.791 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.792 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.793 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.794 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.794 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.796 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.797 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.798 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.799 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.799 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.800 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.801 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.801 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.801 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.802 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:01Z" 2023-10-17T19:42:20.802 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.803 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:06.893+0000 7fa4a2dd6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:06.894+0000 7fa4a2dd6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:06Z" 2023-10-17T19:42:20.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.827 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.828 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.828 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.828 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.829 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.829 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.830 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.832 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.833 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.834 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.834 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.834 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.835 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.837 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.837 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.838 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.838 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.838 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.839 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.840 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.841 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.841 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.841 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.842 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.842 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.843 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.844 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.845 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.845 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.845 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.845 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.846 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.846 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.846 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.847 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:06Z" 2023-10-17T19:42:20.847 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.847 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:12.020+0000 7fe1543e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:12.021+0000 7fe1543e7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:12Z" 2023-10-17T19:42:20.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.875 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.875 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.876 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.876 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.876 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.877 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.877 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.877 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.878 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.878 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.878 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.879 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.880 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.881 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.881 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.881 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.882 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.884 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.884 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.884 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.885 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.885 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.886 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.887 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.888 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.888 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.888 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.889 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.889 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.890 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.891 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.892 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.892 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.893 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.893 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.893 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.893 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.894 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.894 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.894 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:12Z" 2023-10-17T19:42:20.895 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.895 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:17.143+0000 7fba7639f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:17.144+0000 7fba7639f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:17Z" 2023-10-17T19:42:20.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.919 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.920 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.920 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.920 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.921 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.921 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.922 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.922 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.923 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.924 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.925 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.926 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.926 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.926 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.927 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.929 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.930 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.930 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.931 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.932 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.932 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.933 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.933 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.934 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.934 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.935 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.935 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.935 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.937 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.937 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.937 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.938 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.938 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.938 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.939 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.939 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.939 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.939 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:17Z" 2023-10-17T19:42:20.940 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.940 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:22.267+0000 7fcbf240e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:22.269+0000 7fcbf240e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:20.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:20.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:20.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:22Z" 2023-10-17T19:42:20.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:20.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:20.963 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:20.964 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:20.964 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:20.964 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.965 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.965 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:20.965 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.966 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:20.966 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.967 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:20.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.969 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.969 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.969 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.970 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:20.970 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.972 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.973 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.973 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.973 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:20.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.974 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.976 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.976 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.976 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:20.976 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:20.977 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:20.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:20.977 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:20.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:20.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:20.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:20.979 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:20.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:20.979 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.979 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:20.980 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:20.980 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.980 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:20.981 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:20.981 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:20.981 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:20.982 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:22Z" 2023-10-17T19:42:20.982 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:20.982 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:20.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:20.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:20.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:20.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:20.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:20.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:20.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:20.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:27.393+0000 7f009765d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:27.394+0000 7f009765d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:20.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:20.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:20.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:20.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:20.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:20.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:20.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:20.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:20.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:20.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:20.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:20.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:20.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:20.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:20.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:20.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:20.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:20.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:20.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:20.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:20.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:20.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:20.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:20.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:20.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:27Z" 2023-10-17T19:42:21.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.007 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.007 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.008 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.008 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.008 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.009 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.009 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.010 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.010 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.012 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.013 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.013 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.013 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.014 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.014 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.017 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.017 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.017 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.018 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.018 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.019 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.020 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.020 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.021 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.021 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.021 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.022 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.022 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.024 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.025 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.025 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.025 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.026 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.026 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.026 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.027 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.027 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.027 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:27Z" 2023-10-17T19:42:21.028 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.028 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:32.521+0000 7f33c28cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:32.523+0000 7f33c28cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:32Z" 2023-10-17T19:42:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.052 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.052 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.052 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.053 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.053 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.054 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.054 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.054 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.055 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.055 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.056 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.057 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.058 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.058 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.058 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.058 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.059 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.060 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.061 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.061 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.061 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.062 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.063 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.064 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.066 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.066 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.066 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.067 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.069 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.070 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.070 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.070 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.071 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.071 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.071 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.072 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.072 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:32Z" 2023-10-17T19:42:21.072 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.073 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:37.650+0000 7f8773bd9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:37.651+0000 7f8773bd9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:37Z" 2023-10-17T19:42:21.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.097 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.097 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.098 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.098 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.098 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.099 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.099 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.100 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.100 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.100 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.102 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.103 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.103 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.103 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.104 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.104 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.106 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.107 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.107 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.108 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.108 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.110 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.110 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.110 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.111 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.111 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.111 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.112 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.113 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.113 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.114 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.114 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.114 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.115 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.115 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.115 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.116 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.116 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.116 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:37Z" 2023-10-17T19:42:21.117 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.117 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:42.775+0000 7f14986ce780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:42.776+0000 7f14986ce780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:42Z" 2023-10-17T19:42:21.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.142 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.142 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.142 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.143 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.144 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.144 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.144 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.144 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.146 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.147 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.148 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.148 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.148 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.148 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.151 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.151 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.151 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.152 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.153 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.154 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.155 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.155 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.155 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.155 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.156 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.158 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.158 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.159 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.159 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.159 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.160 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.160 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.160 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.160 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.161 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:42Z" 2023-10-17T19:42:21.161 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.161 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:47.906+0000 7fccf348d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:47.908+0000 7fccf348d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:47Z" 2023-10-17T19:42:21.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.186 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.186 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.186 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.186 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.187 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.187 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.188 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.188 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.188 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.189 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.190 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.191 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.191 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.192 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.192 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.194 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.195 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.195 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.195 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.196 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.197 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.198 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.198 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.199 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.199 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.200 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.200 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.201 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.202 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.203 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.203 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.203 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.203 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.204 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.204 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.204 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.205 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:47Z" 2023-10-17T19:42:21.205 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.205 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:53.032+0000 7f75958dc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:53.033+0000 7f75958dc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:53Z" 2023-10-17T19:42:21.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.230 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.230 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.230 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.231 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.231 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.232 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.232 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.232 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.233 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568816.812003_700.1", 2023-10-17T19:42:21.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.234 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:53:36.812003Z", 2023-10-17T19:42:21.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.236 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.236 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.236 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.236 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.237 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.239 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.240 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.240 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.240 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.241 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.241 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.243 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.243 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.244 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.244 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.244 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.245 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.246 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.247 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.247 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.248 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.248 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.248 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.248 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.249 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.249 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.249 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:53Z" 2023-10-17T19:42:21.250 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.250 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568816.812003_700.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:53:36.812003Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568816.812003_700.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568816.812003_700.1 2023-10-17T19:42:21.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:58.160+0000 7f29fb1e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:55:58.161+0000 7f29fb1e9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568841.365181_703.1", 2023-10-17T19:42:21.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:54:01.365181Z", 2023-10-17T19:42:21.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:55:58Z" 2023-10-17T19:42:21.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.274 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.274 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.274 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.275 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.275 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.276 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.276 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.276 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.276 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568841.365181_703.1", 2023-10-17T19:42:21.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.278 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:54:01.365181Z", 2023-10-17T19:42:21.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.279 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.279 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.280 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.280 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.283 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.283 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.283 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.284 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.284 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.285 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.285 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.286 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.286 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.287 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.287 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.288 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.289 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.290 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.290 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.291 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.291 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.291 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.292 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.292 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.292 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.292 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:55:58Z" 2023-10-17T19:42:21.293 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568841.365181_703.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:54:01.365181Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568841.365181_703.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697568865.677043_735.1 target=1_1697568841.365181_703.1 2023-10-17T19:42:21.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:21.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:21.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:03.288+0000 7f412a38e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:03.290+0000 7f412a38e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:21.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:21.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:21.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:21.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:21.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:21.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:21.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697568865.677043_735.1", 2023-10-17T19:42:21.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:21.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:54:25.677043Z", 2023-10-17T19:42:21.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:21.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:21.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:21.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:21.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:21.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:21.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:21.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:21.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:21.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:21.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:21.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:21.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:21.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:21.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:21.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:21.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:21.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T18:56:03Z" 2023-10-17T19:42:21.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:21.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:21.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:21.318 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:21.318 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:21.319 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:21.319 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:21.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.320 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.320 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:21.320 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.320 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:21.321 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1697568865.677043_735.1", 2023-10-17T19:42:21.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.322 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:21.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:54:25.677043Z", 2023-10-17T19:42:21.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.323 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.323 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.324 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.324 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:21.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.325 INFO:tasks.rgw_multisite_tests: "marker": "1_1697566654.606901_130.1", 2023-10-17T19:42:21.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T18:17:34.606901Z", 2023-10-17T19:42:21.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.327 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.327 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.328 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.328 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:21.328 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.329 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.330 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.331 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.331 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.331 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:21.332 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:21.332 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:21.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:21.332 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:21.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:21.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:21.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:21.334 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:21.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:21.334 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.335 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:21.335 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:21.335 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.336 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:21.336 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:21.336 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:21.337 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:21.337 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T18:56:03Z" 2023-10-17T19:42:21.337 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:21.338 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697568865.677043_735.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T18:54:25.677043Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697566654.606901_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T18:17:34.606901Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-17T19:42:21.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697568865.677043_735.1', 1: '1_1697566654.606901_130.1', 2: '', 3: ''} 2023-10-17T19:42:21.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.343 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.343 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.343 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.344 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.344 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.345 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T19:42:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T19:42:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ee7d734d2251824-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.349 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.349 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.350 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.350 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.350 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.351 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GmMkVJogk5r6wfjVekxbq8dvVvE= 2023-10-17T19:42:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GmMkVJogk5r6wfjVekxbq8dvVvE='} 2023-10-17T19:42:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e4b1a9bb0872f8-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.355 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.355 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.356 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.356 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.356 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.357 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T19:42:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T19:42:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225a805abdd81031-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.361 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.362 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.362 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.362 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.363 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.363 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/Zj7ugw/EGQASmX7VpgDBmGvH9s= 2023-10-17T19:42:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/Zj7ugw/EGQASmX7VpgDBmGvH9s='} 2023-10-17T19:42:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b205584a34381f-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.368 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.369 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.369 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T19:42:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T19:42:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e608a69cebc63e79-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.374 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.374 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.374 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.375 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.375 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.376 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hlBZEqZYr99AF15rJsV6rXST2J0= 2023-10-17T19:42:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hlBZEqZYr99AF15rJsV6rXST2J0='} 2023-10-17T19:42:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb8dcf2b2804a72e-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.380 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.380 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.381 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.381 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.381 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.382 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T19:42:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T19:42:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b846ea310e598f5f-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.386 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.386 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.387 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.387 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.387 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.388 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ydryy37zKLeiF/knNNa27gvYGUU= 2023-10-17T19:42:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ydryy37zKLeiF/knNNa27gvYGUU='} 2023-10-17T19:42:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000319013750d9b34f4-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.392 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.393 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.393 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.393 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.394 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.394 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78= 2023-10-17T19:42:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:imdnvyA1uf2ilJ5klIjgOnjPs78=', 'Content-Length': '0'} 2023-10-17T19:42:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a4820f761b980d5-00652ed8c3-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.398 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.399 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.399 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:03 GMT 2023-10-17T19:42:21.400 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.400 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SvSfDvltaEHSj02HGwVcsSXsbJA= 2023-10-17T19:42:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:03 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvSfDvltaEHSj02HGwVcsSXsbJA='} 2023-10-17T19:42:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036a3d1fe9fd0aa68-00652ed8c3-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T19:42:21.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:21.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:21.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:21.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:03.550+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:08.550+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:13.551+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:18.552+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:23.553+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:28.552+0000 7fcc492ad780 1 waiting to reach incremental sync.. 2023-10-17T19:42:21.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:33.554+0000 7fcc492ad780 1 bucket sync caught up with source: 2023-10-17T19:42:21.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , 00000000001.74.6, 00000000002.76.6, 00000000002.48.6] 2023-10-17T19:42:21.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.74.6, 00000000002.76.6, 00000000002.48.6] 2023-10-17T19:42:21.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:33.554+0000 7fcc492ad780 0 bucket checkpoint complete 2023-10-17T19:42:21.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-59 2023-10-17T19:42:21.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:21.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:21.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:21.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:33.678+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:21.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:38.679+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:43.681+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:21.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:48.682+0000 7f09990f4780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:21.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:53.684+0000 7f09990f4780 1 bucket sync caught up with source: 2023-10-17T19:42:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.383.6, 00000000002.175.6, 00000000002.206.6] 2023-10-17T19:42:21.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:53.684+0000 7f09990f4780 0 bucket checkpoint complete 2023-10-17T19:42:21.414 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-59 zones={a2, a1} 2023-10-17T19:42:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.418 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.418 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.419 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.419 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.419 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.420 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI= 2023-10-17T19:42:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI=', 'Content-Length': '0'} 2023-10-17T19:42:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acccc2dcbae9e7ac-00652ed8f5-1b46-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.425 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.425 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.426 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.426 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.426 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.427 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI= 2023-10-17T19:42:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pnFZV/lyPZcTbd3BlTCDSD9gIlI=', 'Content-Length': '0'} 2023-10-17T19:42:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000274bfedb3f6267e8-00652ed8f5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.428 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.433 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.433 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.433 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.434 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.434 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.435 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T19:42:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T19:42:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907a50b1daa8c931-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.436 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:21.436 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:21.437 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:21.437 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:21.437 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:21.438 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.442 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.442 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.443 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.443 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.443 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.444 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T19:42:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T19:42:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e255e1d361e9e5-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:21.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:21.446 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:21.446 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:21.446 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.451 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.451 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.451 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.452 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.452 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.452 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T19:42:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T19:42:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007f2ebb61ad68e69-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.458 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.459 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.459 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.459 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.460 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.460 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E= 2023-10-17T19:42:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:H8YkV+cK4oLeJBNf8ZxcTAd9F/E=', 'Content-Length': '0'} 2023-10-17T19:42:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91af2fb4c23ba42-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.461 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.465 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.466 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.466 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.466 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.467 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.470 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T19:42:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T19:42:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d10b73d54d1236-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.475 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.475 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.475 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.476 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.476 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.476 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T19:42:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T19:42:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286fd603ead895ee-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.481 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.482 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.482 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.482 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.483 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.483 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM= 2023-10-17T19:42:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM=', 'Content-Length': '0'} 2023-10-17T19:42:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6574dd9e215d73d-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.488 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.488 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.489 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.489 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.489 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.490 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM= 2023-10-17T19:42:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1AjXPlmPBcpDPLslIqKrPkd3kRM=', 'Content-Length': '0'} 2023-10-17T19:42:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf473a7b5db3592-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.491 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.495 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.495 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.496 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.496 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.496 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.497 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T19:42:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T19:42:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b45ba7210919f2b-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.502 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.502 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.503 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.503 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.503 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.504 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4= 2023-10-17T19:42:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cCHhKCe0BxVLHk6Nt/x/hNfIDf4=', 'Content-Length': '0'} 2023-10-17T19:42:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015537fe6649643a7-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.509 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.509 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.509 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.510 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.510 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.511 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T19:42:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T19:42:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce63c4b33681d8ee-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.516 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.517 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.517 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.517 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.518 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.518 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T19:42:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T19:42:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ade935389c834c-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.523 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.524 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.524 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.524 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.525 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.525 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE= 2023-10-17T19:42:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE=', 'Content-Length': '0'} 2023-10-17T19:42:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006405065e7bb7e1a7-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.530 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.530 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.530 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.531 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.531 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.532 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE= 2023-10-17T19:42:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0kEY4iantp0PTjYdVA5OqndazGE=', 'Content-Length': '0'} 2023-10-17T19:42:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a4e4645089d5af-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.533 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.537 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.537 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.537 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.538 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.538 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.539 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM= 2023-10-17T19:42:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM=', 'Content-Length': '0'} 2023-10-17T19:42:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000943a84a9cacfbaba-00652ed8f5-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.543 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.543 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.544 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.544 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.544 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.545 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM= 2023-10-17T19:42:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6eXPuhr66XLf66e0MZA413DVfEM=', 'Content-Length': '0'} 2023-10-17T19:42:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8fefe7755f03486-00652ed8f5-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.550 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.550 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.551 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.551 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.551 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.552 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T19:42:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T19:42:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060d559aeaa91e382-00652ed8f5-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.557 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.557 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.557 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.557 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:53 GMT 2023-10-17T19:42:21.558 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.558 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I= 2023-10-17T19:42:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7s3y9r6N8Nc8O5NbhjnDwfZIM2I=', 'Content-Length': '0'} 2023-10-17T19:42:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000084885bd0166a9c7-00652ed8f5-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.559 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.564 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.565 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.565 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.565 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.566 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.566 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T19:42:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T19:42:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc070428713d31f-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.572 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.572 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.572 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.572 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.573 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.573 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T19:42:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T19:42:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5934142d6f9f466-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.578 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.578 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.579 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.579 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.579 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.580 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0= 2023-10-17T19:42:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0=', 'Content-Length': '0'} 2023-10-17T19:42:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af21f2c5ff0e4010-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.585 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.585 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.585 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.586 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.586 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.587 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0= 2023-10-17T19:42:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L8VkBgoNzEqjaauFezG/jSI7ke0=', 'Content-Length': '0'} 2023-10-17T19:42:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d7e1b2c8e94294-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.592 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.592 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.592 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.593 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.593 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.594 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T19:42:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T19:42:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064017165a57366a9-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.599 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.599 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.599 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.600 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.600 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.600 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk= 2023-10-17T19:42:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oJeSwSQ1GqzUxjQUIcHBkClsabk=', 'Content-Length': '0'} 2023-10-17T19:42:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f28efaf94910d2-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.605 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.606 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.606 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.606 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.607 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.607 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75c926c5fb2a222-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.613 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.613 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.613 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.614 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.614 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.615 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002cb46bed3f6b66-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.619 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.620 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.620 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.620 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.620 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.621 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g= 2023-10-17T19:42:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g=', 'Content-Length': '0'} 2023-10-17T19:42:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011714f76066aada6-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.625 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.626 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.626 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.626 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.627 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.627 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g= 2023-10-17T19:42:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7O9N2IaqyUEKPrEXSuuPEFl0O0g=', 'Content-Length': '0'} 2023-10-17T19:42:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7ac6cc1b3a9706-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.628 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.632 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.632 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.632 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.633 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.633 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.634 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw= 2023-10-17T19:42:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw=', 'Content-Length': '0'} 2023-10-17T19:42:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee5555128f36d57-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.638 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.639 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.639 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.639 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.640 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.640 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw= 2023-10-17T19:42:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qPIN7u6hR7fQmZbxWiJEizU4Vzw=', 'Content-Length': '0'} 2023-10-17T19:42:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d9eae0c83016b0-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.645 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.646 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.646 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.646 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.647 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.647 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5d80c2c1ee72de-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.652 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.652 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.653 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.653 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.653 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.654 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8a06206108c5d1-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.655 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.659 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.659 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.660 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.660 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.660 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.661 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T19:42:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T19:42:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a57f75621012c5d-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.666 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.666 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.666 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.667 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.667 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.668 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T19:42:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T19:42:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002395ce9f720a0067-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.674 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.674 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.675 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.675 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.675 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.676 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA= 2023-10-17T19:42:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA=', 'Content-Length': '0'} 2023-10-17T19:42:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fcb3487eca092a3-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.681 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.681 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.682 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.682 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.682 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.683 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA= 2023-10-17T19:42:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7y0fo+QQgiWwUA7xzCgvXfd7aPA=', 'Content-Length': '0'} 2023-10-17T19:42:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102c58fbde5aff15-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.684 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.689 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.689 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.690 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.690 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.690 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.691 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T19:42:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T19:42:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd4bd5030606413-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.696 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.696 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.697 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.697 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.697 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.698 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY= 2023-10-17T19:42:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:j0dnJMV/l4I/gMWWB+fraML8QUY=', 'Content-Length': '0'} 2023-10-17T19:42:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3906afe4e81ee71-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.703 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.703 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.703 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.704 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.704 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.705 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c45c1fade8c665-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.710 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.711 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.711 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.711 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.712 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.712 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f26fd2874d24f67-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.717 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.717 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.717 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.717 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.718 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs= 2023-10-17T19:42:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs=', 'Content-Length': '0'} 2023-10-17T19:42:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c538763ae6d57731-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.723 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.723 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.723 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.723 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.724 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.724 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs= 2023-10-17T19:42:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZUOUITgrurI8Yde70j/NUMyaavs=', 'Content-Length': '0'} 2023-10-17T19:42:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b7707307931827-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.725 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.729 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.729 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.729 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.730 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.730 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.730 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0= 2023-10-17T19:42:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0=', 'Content-Length': '0'} 2023-10-17T19:42:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13a525292fec653-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.735 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.735 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.736 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.736 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.736 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.737 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0= 2023-10-17T19:42:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GdbCGinZOfMTJdiGu+2ZDpJmAb0=', 'Content-Length': '0'} 2023-10-17T19:42:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b7c28637aad379-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.742 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.742 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.742 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.742 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.743 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.743 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb71b443de1fb51d-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.749 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.749 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.750 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.750 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.750 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.751 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a208e00b63855d-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.752 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.756 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.757 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.757 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.757 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.758 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.758 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T19:42:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T19:42:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6d85a0630c7ff3-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.763 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.763 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.764 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.764 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.764 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.765 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T19:42:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T19:42:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198f52aababbc24c-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.770 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.770 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.770 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.771 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.771 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.772 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM= 2023-10-17T19:42:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM=', 'Content-Length': '0'} 2023-10-17T19:42:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b3f58d1aae7985-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.777 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.777 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.777 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.778 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.778 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.779 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM= 2023-10-17T19:42:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nRlA3HhTJE71Ydp1tShOnc37HXM=', 'Content-Length': '0'} 2023-10-17T19:42:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ab8e263b573afa-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.780 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.784 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.785 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.785 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.785 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.786 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.786 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T19:42:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T19:42:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63e75a1e4435313-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.791 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.792 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.792 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.792 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.792 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.793 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ= 2023-10-17T19:42:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SvEJsk07sZgKAUHedvPSrwQ+4vQ=', 'Content-Length': '0'} 2023-10-17T19:42:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da82ef902a95bd2-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.798 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.798 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.799 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.799 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.799 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.800 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad6c73e2a0e0487-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.806 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.806 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.806 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.807 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.807 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.807 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eec324d860473720-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.813 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.813 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.813 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.814 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.814 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.814 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg= 2023-10-17T19:42:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg=', 'Content-Length': '0'} 2023-10-17T19:42:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e493d105b6bcb0b-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.819 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.819 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.820 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.820 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.820 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.821 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg= 2023-10-17T19:42:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8gbNqueNhFBy+b9ruULnhK1Cowg=', 'Content-Length': '0'} 2023-10-17T19:42:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a5e206eea057af-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.825 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.826 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.826 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.826 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.827 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.827 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw= 2023-10-17T19:42:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw=', 'Content-Length': '0'} 2023-10-17T19:42:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e2f98ae14e3ce8e-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.832 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.832 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.833 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.833 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.834 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.834 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw= 2023-10-17T19:42:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:9LY1hLAC9hJ8+NFej1tGeJkkLGw=', 'Content-Length': '0'} 2023-10-17T19:42:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095c0b08ce40d534-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.839 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.840 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.840 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.840 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.841 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.841 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712fa29a17fb8adb-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.847 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.847 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.847 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.848 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.848 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.848 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa14d184f158d54-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.849 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.854 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.854 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.854 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.854 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.855 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.855 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T19:42:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T19:42:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f0b0c130078677-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.861 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.861 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.861 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.861 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.862 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.862 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T19:42:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T19:42:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26e93db29a29284-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.868 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.868 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.868 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.868 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.869 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds= 2023-10-17T19:42:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds=', 'Content-Length': '0'} 2023-10-17T19:42:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec1f19022af33a8-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.874 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.874 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.875 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.875 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.875 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.876 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds= 2023-10-17T19:42:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:p9lvyYltw7yrRldc3h4wfkCoZds=', 'Content-Length': '0'} 2023-10-17T19:42:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006388f1d6d95859d7-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.877 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.883 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.883 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.883 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.884 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.884 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.884 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T19:42:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T19:42:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1d9ff61d6d2a5c-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.889 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.890 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.890 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.890 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.890 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.891 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8= 2023-10-17T19:42:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Jegwmj9OgJmKyIQs1txHhrhsrX8=', 'Content-Length': '0'} 2023-10-17T19:42:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441f269621c135d4-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.896 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.896 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.896 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.897 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.897 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.898 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009098963d60ff4623-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.903 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.904 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.904 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.904 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.905 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.905 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f1dc55b1abc51f-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.910 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.910 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.910 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.911 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.911 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.912 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0= 2023-10-17T19:42:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0=', 'Content-Length': '0'} 2023-10-17T19:42:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe134ba75171cf2-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.916 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.916 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.917 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.917 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.917 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.918 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0= 2023-10-17T19:42:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DkKSBjUhOOxMFRiWnaS2LEuOzR0=', 'Content-Length': '0'} 2023-10-17T19:42:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d13071c24261066-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.919 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.923 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.923 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.923 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.923 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.924 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.924 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g= 2023-10-17T19:42:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g=', 'Content-Length': '0'} 2023-10-17T19:42:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8979d3751495104-00652ed8f6-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.929 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.929 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.929 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.930 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.930 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.931 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g= 2023-10-17T19:42:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Xj6i8Af2EO4OqItkvlu1RTN/j2g=', 'Content-Length': '0'} 2023-10-17T19:42:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf4b73c8e81adc8-00652ed8f6-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.935 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.936 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.936 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.936 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.937 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.937 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e6fdd5c6a9eafe2-00652ed8f6-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.942 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:21.943 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.943 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.943 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:54 GMT 2023-10-17T19:42:21.944 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.944 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY= 2023-10-17T19:42:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:0ZtyaN77PBfPREK0PJJHw/8WbPY=', 'Content-Length': '0'} 2023-10-17T19:42:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d87f96a2defa62ad-00652ed8f6-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:56:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:21.945 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-59 zones={a2, a1} 2023-10-17T19:42:21.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:21.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:21.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:21.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:21.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:21.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:21.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-59 2023-10-17T19:42:21.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 5 2023-10-17T19:42:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.952 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.952 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.952 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.953 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.953 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.953 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T19:42:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T19:42:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa00845e07e62fbe-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17 2023-10-17T19:42:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.958 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.958 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.958 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.959 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.959 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17 2023-10-17T19:42:21.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.960 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3SkwdlXJKTSlyLt3F9HUUHyZjxk= 2023-10-17T19:42:21.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3SkwdlXJKTSlyLt3F9HUUHyZjxk='} 2023-10-17T19:42:21.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004eb75eac1674b90e-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.964 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.964 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.964 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.965 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.965 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.965 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T19:42:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T19:42:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000499a44e737db71dd-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34 2023-10-17T19:42:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.970 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.970 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.971 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.971 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.971 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34 2023-10-17T19:42:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.972 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bmg7f26GqRUPssuvwxb7cGFMgPo= 2023-10-17T19:42:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bmg7f26GqRUPssuvwxb7cGFMgPo='} 2023-10-17T19:42:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3e2ca4aad15f273-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.976 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.977 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.977 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.977 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.978 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T19:42:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T19:42:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006581bcbb3b15ef6a-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51 2023-10-17T19:42:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.982 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.983 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.983 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.983 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.984 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51 2023-10-17T19:42:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.984 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ibt3dzDMCj/HEZoS1ndI8aPuiII= 2023-10-17T19:42:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ibt3dzDMCj/HEZoS1ndI8aPuiII='} 2023-10-17T19:42:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d193ebf346935e3-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.989 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:21.989 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.989 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:21.990 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.990 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.991 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T19:42:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T19:42:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e746dabc1c7cd24-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68 2023-10-17T19:42:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:21.995 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:21.995 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:21.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:21.996 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:21.996 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68 2023-10-17T19:42:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:21.997 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:s0+j0GIcLrYk5x80btP+IpM6UEo= 2023-10-17T19:42:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:s0+j0GIcLrYk5x80btP+IpM6UEo='} 2023-10-17T19:42:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a02a88bd1046ef1c-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.001 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.002 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.002 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.002 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:22.003 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.003 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E= 2023-10-17T19:42:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hnH9Falvb9YAUZHxa7/Garg429E=', 'Content-Length': '0'} 2023-10-17T19:42:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021838152fb3180aa-00652ed8f7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85 2023-10-17T19:42:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.008 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:22.008 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:22.008 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:22.008 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:56:55 GMT 2023-10-17T19:42:22.009 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85 2023-10-17T19:42:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.009 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:guQP2slNtf1AMh+rmim8Sv4pJFU= 2023-10-17T19:42:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:56:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:guQP2slNtf1AMh+rmim8Sv4pJFU='} 2023-10-17T19:42:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce11d71e323fec88-00652ed8f7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T19:42:22.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:22.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:56:55.347+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:00.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:05.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:10.348+0000 7f4bc9f56780 1 waiting to reach latest gen 1, on 0.. 2023-10-17T19:42:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:15.350+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T19:42:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:20.352+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T19:42:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:25.353+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T19:42:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:30.356+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , , , , ] 2023-10-17T19:42:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:35.357+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:40.357+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:45.358+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:50.360+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:57:55.361+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:00.363+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:05.365+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000001.53.6, , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:10.367+0000 7f4bc9f56780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000001.53.6, , , , , , , 00000000001.60.6, , , ] 2023-10-17T19:42:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:15.369+0000 7f4bc9f56780 1 bucket sync caught up with source: 2023-10-17T19:42:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [00000000001.51.6, , , , , , 00000000001.53.6, , 00000000001.39.6, , , , 00000000001.44.6, 00000000001.60.6, , , ] 2023-10-17T19:42:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:15.369+0000 7f4bc9f56780 0 bucket checkpoint complete 2023-10-17T19:42:22.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-59 2023-10-17T19:42:22.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:22.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:15.494+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T19:42:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:20.494+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T19:42:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:25.496+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T19:42:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:30.497+0000 7fd26ed66780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , 00000000001.195.6, , , , , , 00000000001.383.6, 00000000002.175.6, 00000000003.214.6] 2023-10-17T19:42:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:35.498+0000 7fd26ed66780 1 bucket sync caught up with source: 2023-10-17T19:42:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, 00000000001.155.6, 00000000001.195.6, , , 00000000001.169.6, , , 00000000001.383.6, 00000000003.197.6, 00000000003.214.6] 2023-10-17T19:42:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:35.498+0000 7fd26ed66780 0 bucket checkpoint complete 2023-10-17T19:42:22.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-59 zones={a2, a1} 2023-10-17T19:42:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.036 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.037 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.037 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.037 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.038 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.038 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0= 2023-10-17T19:42:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0=', 'Content-Length': '0'} 2023-10-17T19:42:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb8177c373b87688-00652ed95b-1b46-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.043 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.043 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.044 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.044 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.044 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.045 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0= 2023-10-17T19:42:22.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:L2yM/CncG0sMXQcwBJYyk0IfwY0=', 'Content-Length': '0'} 2023-10-17T19:42:22.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e853f1499cc01c30-00652ed95b-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.045 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:22.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.050 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.050 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.051 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.051 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.051 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:22.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.052 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T19:42:22.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T19:42:22.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b33c10dd8d1c5b-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.053 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:22.053 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2023-10-17T19:42:22.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:22.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:22.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2023-10-17T19:42:22.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:22.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2023-10-17T19:42:22.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:22.056 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2023-10-17T19:42:22.056 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2023-10-17T19:42:22.056 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:22.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.060 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.061 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.061 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.061 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.062 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.062 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T19:42:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T19:42:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db987f21832655eb-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:22.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2023-10-17T19:42:22.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:22.064 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:22.065 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2023-10-17T19:42:22.065 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:22.065 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj51 2023-10-17T19:42:22.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:22.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj68 2023-10-17T19:42:22.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj85 2023-10-17T19:42:22.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.071 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.072 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.072 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:22.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.073 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T19:42:22.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T19:42:22.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a9c0884297d5c0-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?versions&max-keys=999 2023-10-17T19:42:22.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.078 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.078 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.079 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.079 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.079 INFO:tasks.rgw_multisite_tests:/hbmuog-59/?versions 2023-10-17T19:42:22.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.080 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0= 2023-10-17T19:42:22.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h+ysSvXWVeWlVNcr4vQltfn1Pa0=', 'Content-Length': '0'} 2023-10-17T19:42:22.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8da6364700b1b0-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59999falseobj11nulltrue2023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.081 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:22.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.086 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.086 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.086 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.087 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.087 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.087 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T19:42:22.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1edf8943516eb8-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.092 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.093 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.093 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.093 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.094 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.094 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T19:42:22.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6685ef802c71391-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.099 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.100 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.100 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.100 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.100 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.101 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI= 2023-10-17T19:42:22.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI=', 'Content-Length': '0'} 2023-10-17T19:42:22.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea338c4c13e5abe-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.106 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.106 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.107 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.107 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.107 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.108 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI= 2023-10-17T19:42:22.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7VT5KE/gQMQFoCwvyeDxyw6fjkI=', 'Content-Length': '0'} 2023-10-17T19:42:22.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a686b9bf142489b-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.109 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:22.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.113 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.114 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.114 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.114 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.115 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.115 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T19:42:22.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096466aa678f06d50-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.120 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.120 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.121 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.121 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.121 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11?versionId=null 2023-10-17T19:42:22.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.122 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c= 2023-10-17T19:42:22.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jbTgr8ODSmUVs9BTuepgzLQdb0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca807ce5c910eb0a-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.127 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.127 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.127 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.128 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.128 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.129 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b985ed6529481a4-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.134 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.134 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.134 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.135 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.135 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.136 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e8388b000f7e369-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:22.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.140 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.141 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.141 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.141 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.141 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:22.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.142 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM= 2023-10-17T19:42:22.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM=', 'Content-Length': '0'} 2023-10-17T19:42:22.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc46d02148fc6e8f-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:22.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.146 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.147 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.147 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.147 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.148 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:22.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.148 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM= 2023-10-17T19:42:22.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4aeQZ4JXJB/5yC/FLqCOMV0yAM=', 'Content-Length': '0'} 2023-10-17T19:42:22.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73e5fce63122cef-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:22.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:22.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.153 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.153 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.153 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.154 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.154 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:22.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.155 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k= 2023-10-17T19:42:22.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k=', 'Content-Length': '0'} 2023-10-17T19:42:22.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001381fd6928405102-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj11 2023-10-17T19:42:22.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj11 2023-10-17T19:42:22.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj11 2023-10-17T19:42:22.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.159 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.159 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.159 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.159 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.160 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj11 2023-10-17T19:42:22.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.160 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k= 2023-10-17T19:42:22.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:o+soKEaSk9MqgwXPKLQ4VWBCm9k=', 'Content-Length': '0'} 2023-10-17T19:42:22.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67889a90d4ca1eb-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.165 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.165 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.166 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.166 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.166 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.167 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b607a18b85e1dcba-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj11 2023-10-17T19:42:22.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.172 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.173 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.174 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.174 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059fc9fd148cfc9f5-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj111000falseobj112023-10-17T18:56:03.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-10-17T19:42:22.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.180 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.181 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.181 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.181 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.182 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.182 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T19:42:22.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T19:42:22.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb0950abd4fc22c-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.187 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.188 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.188 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.188 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.189 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.189 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T19:42:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T19:42:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f399fa2935895881-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.195 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.195 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.195 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.195 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.196 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.196 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8= 2023-10-17T19:42:22.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8=', 'Content-Length': '0'} 2023-10-17T19:42:22.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde8c36e014c97c8-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.201 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.201 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.202 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.202 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.202 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.203 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8= 2023-10-17T19:42:22.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:1aa87iajV2Ko9rufvlW1HvMb2i8=', 'Content-Length': '0'} 2023-10-17T19:42:22.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d09bd8b00cf9cac-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.204 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-10-17T19:42:22.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.208 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.208 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.209 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.209 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.209 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.210 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T19:42:22.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T19:42:22.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f231be7b788f6e-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.215 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.215 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.215 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.216 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.216 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17?versionId=null 2023-10-17T19:42:22.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.217 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90= 2023-10-17T19:42:22.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XRzvKRNU0o5amSCu76ihsu1fD90=', 'Content-Length': '0'} 2023-10-17T19:42:22.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd77547394b93930-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.222 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.222 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.222 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.223 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.223 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.224 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffbc3c573f4f9a2-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.229 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.229 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.229 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.229 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.230 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.231 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b97fe2a60ffe7cc-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17 2023-10-17T19:42:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.235 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.235 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.236 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.236 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.236 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17 2023-10-17T19:42:22.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.237 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM= 2023-10-17T19:42:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM=', 'Content-Length': '0'} 2023-10-17T19:42:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e97ca148471cce-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17 2023-10-17T19:42:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.241 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.242 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.242 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17 2023-10-17T19:42:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.243 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM= 2023-10-17T19:42:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BBsxnU7nm9xQtv88sBzePMboLLM=', 'Content-Length': '0'} 2023-10-17T19:42:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6910e0da035903-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.244 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2023-10-17T19:42:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17 2023-10-17T19:42:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.247 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.248 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.248 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.248 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.248 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17 2023-10-17T19:42:22.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.249 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c= 2023-10-17T19:42:22.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001923d72c9b789572-00652ed95b-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj17 2023-10-17T19:42:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj17 2023-10-17T19:42:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj17 2023-10-17T19:42:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.253 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.253 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.254 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.254 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.254 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj17 2023-10-17T19:42:22.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.255 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c= 2023-10-17T19:42:22.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cR1WtD0tAtzNuYkJe6QCxY9sz0c=', 'Content-Length': '0'} 2023-10-17T19:42:22.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609f6d9b832deccd-00652ed95b-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.260 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.260 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.260 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.260 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.261 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.261 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3dcd6bcbfbc3370-00652ed95b-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj17 2023-10-17T19:42:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.266 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.267 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.267 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.267 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:35 GMT 2023-10-17T19:42:22.268 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.268 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0= 2023-10-17T19:42:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MbabFXZqd9lu2j9P80rVkYoavh0=', 'Content-Length': '0'} 2023-10-17T19:42:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7b95cd2d64fb03-00652ed95b-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj171000falseobj172023-10-17T18:56:55.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.274 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.274 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.274 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.274 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.275 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.275 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T19:42:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T19:42:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000695d450faf4e994c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.280 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.280 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.280 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.281 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.281 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.282 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T19:42:22.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T19:42:22.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed07b73276fd915d-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.287 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.287 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.287 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.288 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.288 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.289 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8= 2023-10-17T19:42:22.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8=', 'Content-Length': '0'} 2023-10-17T19:42:22.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b315fe3b8576b495-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.294 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.294 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.294 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.295 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.295 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.295 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8= 2023-10-17T19:42:22.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O/VVHq2lLClMr+tmD+ux7T8m5v8=', 'Content-Length': '0'} 2023-10-17T19:42:22.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01bd65d26c1564e-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.296 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:22.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.301 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.301 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.301 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.302 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.302 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.303 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T19:42:22.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T19:42:22.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629c249767dd8609-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.307 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.307 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.308 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.308 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.308 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22?versionId=null 2023-10-17T19:42:22.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.309 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE= 2023-10-17T19:42:22.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3sReEZxCx2PxwKMaUPvA8i28dZE=', 'Content-Length': '0'} 2023-10-17T19:42:22.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb30b59008fff8d2-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.314 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.314 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.314 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.315 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.315 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.316 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7e975fc410b049-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.321 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.321 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.321 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.322 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.322 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.323 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554a75144170b382-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.327 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.328 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.328 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.328 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.328 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:22.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.329 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o= 2023-10-17T19:42:22.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o=', 'Content-Length': '0'} 2023-10-17T19:42:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e20013195dd4f01-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.333 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.334 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.334 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.334 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.335 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:22.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.335 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o= 2023-10-17T19:42:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tb//o6qfVAfLd6zJlNQqKtHqH4o=', 'Content-Length': '0'} 2023-10-17T19:42:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb4457affab5de1-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.336 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.340 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.340 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.341 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.341 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.341 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:22.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.342 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo= 2023-10-17T19:42:22.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo=', 'Content-Length': '0'} 2023-10-17T19:42:22.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e5bbc4c1807e52-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj22 2023-10-17T19:42:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj22 2023-10-17T19:42:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj22 2023-10-17T19:42:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.346 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.347 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.348 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj22 2023-10-17T19:42:22.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.348 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo= 2023-10-17T19:42:22.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ihFKLRcKxguISoaPrC8u79oJDZo=', 'Content-Length': '0'} 2023-10-17T19:42:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60b19cbad6b4ed4-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.353 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.353 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.354 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.354 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.354 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.355 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2789a8b2df48105-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj22 2023-10-17T19:42:22.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.360 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.361 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.361 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.361 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.362 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.362 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497218669d78d30f-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj221000falseobj222023-10-17T18:56:03.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.364 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:22.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.368 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.368 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.369 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.369 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.370 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T19:42:22.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T19:42:22.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082198f4c2fb9a0f5-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.374 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.375 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.375 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.376 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T19:42:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T19:42:22.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc8f99d67d725a6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.381 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.381 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.381 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.382 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.382 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.383 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU= 2023-10-17T19:42:22.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU=', 'Content-Length': '0'} 2023-10-17T19:42:22.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1d96c3ceb7b66c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.387 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.388 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.388 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.388 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.389 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.389 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU= 2023-10-17T19:42:22.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7DE03BQarHz/gFCMp6GmohXLYTU=', 'Content-Length': '0'} 2023-10-17T19:42:22.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb9cec84289804b-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.390 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:22.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.395 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.395 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.395 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.396 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.396 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.397 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T19:42:22.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T19:42:22.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a270ea18bfb8bf-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.402 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.402 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.402 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.403 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.403 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33?versionId=null 2023-10-17T19:42:22.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.404 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M= 2023-10-17T19:42:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ao01VsRhZgddgSZ0uKTix+mT/M=', 'Content-Length': '0'} 2023-10-17T19:42:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c792ab48be0ef9a7-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.409 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.409 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.409 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.410 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.410 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.411 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a88d2438f2d0e4-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.416 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.416 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.416 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.417 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.417 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.418 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008bb8be87eceb550-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:22.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.422 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.423 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.423 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.423 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.424 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:22.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.424 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc= 2023-10-17T19:42:22.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc=', 'Content-Length': '0'} 2023-10-17T19:42:22.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a1daa99fae9957-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:22.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.429 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.429 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.429 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.429 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.430 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:22.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.430 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc= 2023-10-17T19:42:22.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qgGNwU3etyLzofbu2S/BhRqkodc=', 'Content-Length': '0'} 2023-10-17T19:42:22.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7f1ce3de875359-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.431 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:22.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.435 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.435 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.436 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.436 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.436 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.437 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo= 2023-10-17T19:42:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo=', 'Content-Length': '0'} 2023-10-17T19:42:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f852028dbf2017c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj33 2023-10-17T19:42:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj33 2023-10-17T19:42:22.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj33 2023-10-17T19:42:22.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.441 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.442 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.442 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.442 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.443 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj33 2023-10-17T19:42:22.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.443 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo= 2023-10-17T19:42:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CXPHE0v0edKNjcMAnsCXfUAjywo=', 'Content-Length': '0'} 2023-10-17T19:42:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813844f625902132-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.448 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.449 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.449 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.449 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.450 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.450 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf2912c34681aae-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj33 2023-10-17T19:42:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.456 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.456 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.456 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.457 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.457 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.458 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb41ac52bc2d1642-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj331000falseobj332023-10-17T18:56:03.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.459 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-10-17T19:42:22.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.464 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.464 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.464 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.465 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.465 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T19:42:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T19:42:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ad23ea537027cd-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.470 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.471 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.471 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.471 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.472 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.472 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T19:42:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T19:42:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddc24d98881c9e6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.477 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.478 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.478 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.478 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.479 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.479 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro= 2023-10-17T19:42:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro=', 'Content-Length': '0'} 2023-10-17T19:42:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006350c1730857725b-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.484 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.484 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.485 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.485 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.485 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.486 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro= 2023-10-17T19:42:22.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3VXCltj3gfL+sdOtX2T4HMtHyro=', 'Content-Length': '0'} 2023-10-17T19:42:22.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c56dd7c6475069-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.487 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-10-17T19:42:22.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.491 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.492 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.492 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.492 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.492 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.493 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T19:42:22.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T19:42:22.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d501b09f6fe4e9b-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.501 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.501 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.501 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.501 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.502 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34?versionId=null 2023-10-17T19:42:22.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.502 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo= 2023-10-17T19:42:22.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oWHDquHw9deIOuMKLrhFaQvjpvo=', 'Content-Length': '0'} 2023-10-17T19:42:22.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073e3f2b7251ce27-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.508 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.509 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.509 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.509 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.510 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.510 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac03cdd9ab78f43-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.515 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.516 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.516 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.516 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.517 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.517 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063324a00e86953a2-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34 2023-10-17T19:42:22.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.522 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.523 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.523 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.523 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.524 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34 2023-10-17T19:42:22.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.524 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac= 2023-10-17T19:42:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac=', 'Content-Length': '0'} 2023-10-17T19:42:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2f4146d211edd0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34 2023-10-17T19:42:22.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.529 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.529 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.529 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.530 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.530 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34 2023-10-17T19:42:22.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.531 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac= 2023-10-17T19:42:22.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cLMcw+TRWdf+gbgm3ZL9auicuac=', 'Content-Length': '0'} 2023-10-17T19:42:22.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699f9022aee3060e-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.532 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2023-10-17T19:42:22.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34 2023-10-17T19:42:22.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.536 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.536 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.536 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.537 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.537 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34 2023-10-17T19:42:22.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.537 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg= 2023-10-17T19:42:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg=', 'Content-Length': '0'} 2023-10-17T19:42:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39d56e22ef42ce0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj34 2023-10-17T19:42:22.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj34 2023-10-17T19:42:22.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj34 2023-10-17T19:42:22.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.542 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.542 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.542 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.543 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.543 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj34 2023-10-17T19:42:22.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.544 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg= 2023-10-17T19:42:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iWVjx2sythQtaPsRIrujqaKxdOg=', 'Content-Length': '0'} 2023-10-17T19:42:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea27ad39fd47bf4-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.549 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.549 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.549 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.550 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.550 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.551 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043fe17162d443ab2-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj34 2023-10-17T19:42:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.556 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.557 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.557 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.558 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae11823fc6b8e497-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj341000falseobj342023-10-17T18:56:55.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.559 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:22.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.564 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.564 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.564 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.565 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.565 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.565 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T19:42:22.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T19:42:22.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a092e68f9674c5e0-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.571 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.571 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.571 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.572 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.572 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.573 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T19:42:22.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T19:42:22.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6b09b6c6e9d3b9-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.578 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.579 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.579 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.579 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.580 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.580 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw= 2023-10-17T19:42:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw=', 'Content-Length': '0'} 2023-10-17T19:42:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50b828e19251d87-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.585 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.586 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.586 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.586 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.586 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.587 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw= 2023-10-17T19:42:22.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lg7raoXcDXJM5uU4gfZ6+y9jBdw=', 'Content-Length': '0'} 2023-10-17T19:42:22.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047367c69776b42a3-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:22.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.592 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.593 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.593 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.593 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.593 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.594 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T19:42:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T19:42:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2a2b5822614e3f-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.599 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.599 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.599 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.600 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.600 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44?versionId=null 2023-10-17T19:42:22.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.601 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE= 2023-10-17T19:42:22.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JBrRp3t5copsxcQdCCixsm9DskE=', 'Content-Length': '0'} 2023-10-17T19:42:22.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fff54a8483b2c6-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.606 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.606 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.606 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.607 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.607 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.607 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003326dd232c658ce1-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.613 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.613 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.613 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.614 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.614 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.615 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d3d83556dff240b-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.620 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.620 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.620 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.621 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.621 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:22.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.622 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg= 2023-10-17T19:42:22.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg=', 'Content-Length': '0'} 2023-10-17T19:42:22.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253e01ebfee8fbb7-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:22.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.626 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.626 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.627 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.627 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.627 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:22.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.628 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg= 2023-10-17T19:42:22.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kpuU9spwXs9P/MHnjDYy5ZTlgbg=', 'Content-Length': '0'} 2023-10-17T19:42:22.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aae61fc328e4e92-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.629 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:22.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:22.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.632 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.633 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.633 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.633 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.634 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:22.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.634 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw= 2023-10-17T19:42:22.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw=', 'Content-Length': '0'} 2023-10-17T19:42:22.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34191b03e83e185-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj44 2023-10-17T19:42:22.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj44 2023-10-17T19:42:22.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj44 2023-10-17T19:42:22.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.639 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.639 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.639 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.640 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.640 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj44 2023-10-17T19:42:22.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.641 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw= 2023-10-17T19:42:22.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6WFA5kQATwz7MN8x7Xw3dwHCeiw=', 'Content-Length': '0'} 2023-10-17T19:42:22.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe1e4612f29f8fa-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.646 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.646 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.646 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.646 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.647 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.648 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f602144aad000d28-00652ed95c-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj44 2023-10-17T19:42:22.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.653 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.653 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.654 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.654 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.654 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.655 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde02b8739160daf-00652ed95c-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj441000falseobj442023-10-17T18:56:03.398Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.656 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-10-17T19:42:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.661 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.661 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.661 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.662 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.662 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.662 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T19:42:22.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T19:42:22.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9ebb56f5c17331-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.667 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.667 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.668 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.668 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.668 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.669 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T19:42:22.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T19:42:22.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b3eadfaf35aab1-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.674 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.674 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.674 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.674 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.675 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.675 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U= 2023-10-17T19:42:22.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U=', 'Content-Length': '0'} 2023-10-17T19:42:22.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad0e41ea51f31b6-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.680 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.681 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.681 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.681 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.681 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.682 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U= 2023-10-17T19:42:22.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2+UfeQGGVLmn+S0ns7FsqFsKi+U=', 'Content-Length': '0'} 2023-10-17T19:42:22.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76644d838ce2ded-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.683 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-10-17T19:42:22.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.687 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.688 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.688 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.688 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.689 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.689 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T19:42:22.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T19:42:22.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016531567f7f8c09c-00652ed95c-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.694 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.695 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.695 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.695 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.695 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51?versionId=null 2023-10-17T19:42:22.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.696 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ= 2023-10-17T19:42:22.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xvhqC0HMmJFf/ZW9Gr48KyJtvmQ=', 'Content-Length': '0'} 2023-10-17T19:42:22.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0002c7bb6d3740a-00652ed95c-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.701 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.702 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:36 GMT 2023-10-17T19:42:22.703 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.703 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A= 2023-10-17T19:42:22.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AeTLubd9ento8rJ5ObFNRKbx75A=', 'Content-Length': '0'} 2023-10-17T19:42:22.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ada77c8ee8686d-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.709 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.709 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.709 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.710 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.710 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.711 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008623119eb8298872-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51 2023-10-17T19:42:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.716 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.716 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.717 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.717 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51 2023-10-17T19:42:22.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.718 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g= 2023-10-17T19:42:22.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g=', 'Content-Length': '0'} 2023-10-17T19:42:22.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c73fd298413a04cd-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51 2023-10-17T19:42:22.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.722 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.722 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.722 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.723 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.723 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51 2023-10-17T19:42:22.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.724 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g= 2023-10-17T19:42:22.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QkcyWCOBuSU443R+BsZUVrL0W+g=', 'Content-Length': '0'} 2023-10-17T19:42:22.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09c462ac517afe6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.724 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj51 2023-10-17T19:42:22.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51 2023-10-17T19:42:22.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.728 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.728 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.728 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.729 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.729 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51 2023-10-17T19:42:22.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.730 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y= 2023-10-17T19:42:22.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y=', 'Content-Length': '0'} 2023-10-17T19:42:22.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef9f7f45e242b99-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj51 2023-10-17T19:42:22.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj51 2023-10-17T19:42:22.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj51 2023-10-17T19:42:22.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.734 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.734 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.735 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.735 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.735 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj51 2023-10-17T19:42:22.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.736 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y= 2023-10-17T19:42:22.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tvWK4EJiZUDJ1P0mMpxLbM5Kd1Y=', 'Content-Length': '0'} 2023-10-17T19:42:22.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000355b67403242942c-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.740 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.741 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.741 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.741 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.742 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.742 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9316ed4540498ab-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj51 2023-10-17T19:42:22.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.747 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.748 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.748 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.748 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.749 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.749 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000947bd5a24a204c52-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj511000falseobj512023-10-17T18:56:55.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.751 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:22.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.755 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.756 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.756 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.756 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.757 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.757 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T19:42:22.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T19:42:22.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd603a879bcf97e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.762 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.762 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.763 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.763 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.763 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.764 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T19:42:22.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T19:42:22.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fda66bab7b427d9-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.768 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.769 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.769 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.769 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.770 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.770 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI= 2023-10-17T19:42:22.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI=', 'Content-Length': '0'} 2023-10-17T19:42:22.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f86fac30f6eb1e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.775 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.776 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.776 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.776 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.777 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.777 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI= 2023-10-17T19:42:22.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ZhX/jY5azK05QDVAE4T/ruCt7NI=', 'Content-Length': '0'} 2023-10-17T19:42:22.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66fdb7419a52790-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.778 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:22.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.783 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.783 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.783 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.784 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.784 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.785 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T19:42:22.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T19:42:22.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a364016d67cb55f-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.790 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.790 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.790 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.790 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.791 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55?versionId=null 2023-10-17T19:42:22.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.791 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms= 2023-10-17T19:42:22.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:v12VOp7IT42i8mXTdWnXUMAghms=', 'Content-Length': '0'} 2023-10-17T19:42:22.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32abd6e294dd75b-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.796 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.797 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.797 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.797 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.797 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.798 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057c94f4c63c67241-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.803 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.804 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.804 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.804 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.805 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.805 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca42267650ea019-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:22.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.810 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.810 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.811 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.811 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.811 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:22.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.812 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo= 2023-10-17T19:42:22.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo=', 'Content-Length': '0'} 2023-10-17T19:42:22.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53ba684a9775b02-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.816 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.817 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.817 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.817 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:22.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.818 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo= 2023-10-17T19:42:22.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:WzETGMyGMfmex4hw/hUIusOMVFo=', 'Content-Length': '0'} 2023-10-17T19:42:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f70f6ade4e3c0ca-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.819 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.823 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.823 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.824 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.824 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.824 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:22.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.825 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As= 2023-10-17T19:42:22.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As=', 'Content-Length': '0'} 2023-10-17T19:42:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c07f295136e9ba5a-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj55 2023-10-17T19:42:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj55 2023-10-17T19:42:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj55 2023-10-17T19:42:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.829 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.830 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.830 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.830 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.830 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj55 2023-10-17T19:42:22.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.831 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As= 2023-10-17T19:42:22.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wfhXbj09F3UUjAqU9vV1sO6o1As=', 'Content-Length': '0'} 2023-10-17T19:42:22.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a11e4db9149cc85-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.836 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.836 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.837 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.837 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.837 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.838 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b24bb304467fd0e0-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj55 2023-10-17T19:42:22.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.843 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.844 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.844 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.844 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.845 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.845 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000417ca94cb8ac5f61-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj551000falseobj552023-10-17T18:56:03.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.847 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-10-17T19:42:22.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.851 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.851 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.851 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.852 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.852 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.853 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T19:42:22.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T19:42:22.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0646ed6e1ccd18-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.858 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.858 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.858 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.859 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.859 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.859 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T19:42:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T19:42:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd864c6b21f03b3b-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.864 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.864 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.865 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.865 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.865 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.866 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs= 2023-10-17T19:42:22.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs=', 'Content-Length': '0'} 2023-10-17T19:42:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f22d0c850e85ba3-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.871 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.871 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.872 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.872 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.872 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.873 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs= 2023-10-17T19:42:22.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SLnnlun8r51SYmOUfbfR3aC4rhs=', 'Content-Length': '0'} 2023-10-17T19:42:22.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d677eba75ce2fd-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.874 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-10-17T19:42:22.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.878 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.879 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.879 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.879 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.880 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.880 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T19:42:22.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T19:42:22.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d3794dc634b50e-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.885 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.886 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.886 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.886 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.886 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68?versionId=null 2023-10-17T19:42:22.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.887 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58= 2023-10-17T19:42:22.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7oaydaZjGSqT8z6D/FacWgcdQ58=', 'Content-Length': '0'} 2023-10-17T19:42:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d654f2daecd7b0a4-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.892 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.893 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.893 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.893 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.893 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.894 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2312f6f0d813d3c-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.900 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.900 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.901 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.901 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003304d24c3702a73e-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68 2023-10-17T19:42:22.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.906 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.906 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.907 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.907 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.907 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68 2023-10-17T19:42:22.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.908 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4= 2023-10-17T19:42:22.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4=', 'Content-Length': '0'} 2023-10-17T19:42:22.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eba0d64f6a2def3-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68 2023-10-17T19:42:22.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.912 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.912 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.913 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.913 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.913 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68 2023-10-17T19:42:22.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.914 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4= 2023-10-17T19:42:22.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PE/21tTBqa4KeB2RPPLCPeIUg/4=', 'Content-Length': '0'} 2023-10-17T19:42:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ac2a52296d7db6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.915 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj68 2023-10-17T19:42:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68 2023-10-17T19:42:22.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.919 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.919 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.919 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.920 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.920 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68 2023-10-17T19:42:22.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.920 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso= 2023-10-17T19:42:22.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso=', 'Content-Length': '0'} 2023-10-17T19:42:22.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a018f1e20aff42bc-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj68 2023-10-17T19:42:22.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj68 2023-10-17T19:42:22.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj68 2023-10-17T19:42:22.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.925 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.925 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.925 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.926 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.926 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj68 2023-10-17T19:42:22.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.927 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso= 2023-10-17T19:42:22.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nWct6yhvaSkNj5JblrDnEbeSEso=', 'Content-Length': '0'} 2023-10-17T19:42:22.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4768fc2d605bc7-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.931 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.931 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.932 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.932 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.932 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.933 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d8d0e8a19382808-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj68 2023-10-17T19:42:22.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.939 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.939 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.940 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.940 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.940 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.941 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688f43b0890e9603-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj681000falseobj682023-10-17T18:56:55.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.942 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-10-17T19:42:22.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.947 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.947 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.947 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.948 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.948 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.949 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T19:42:22.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T19:42:22.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e61f1b7a702d111-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.954 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.954 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.955 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.955 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.955 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.956 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T19:42:22.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T19:42:22.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0a8024069bb1f12-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.961 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.961 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.961 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.961 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.962 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.962 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw= 2023-10-17T19:42:22.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw=', 'Content-Length': '0'} 2023-10-17T19:42:22.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa8e2d212e3aa79-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:22.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.967 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:22.968 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.968 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.968 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.969 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.969 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw= 2023-10-17T19:42:22.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:isFAhnTby/CAdbEBkmnshfeC2rw=', 'Content-Length': '0'} 2023-10-17T19:42:22.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130ec1770f442698-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.970 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-10-17T19:42:22.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.974 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.975 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.975 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.975 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.976 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.976 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T19:42:22.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T19:42:22.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2c13bd08da2866-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:22.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.981 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.982 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.982 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.982 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.983 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85?versionId=null 2023-10-17T19:42:22.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.983 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis= 2023-10-17T19:42:22.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/3qwvEYsSJ2GbxwvYKNz8UX9Sis=', 'Content-Length': '0'} 2023-10-17T19:42:22.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf094c623afb6f6-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:22.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:22.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.988 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.989 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.989 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.989 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.990 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.990 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c29d7258d32cd10-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:22.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:22.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:22.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj85 2023-10-17T19:42:22.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:22.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:22.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:22.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:22.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:22.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:22.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:22.996 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:22.996 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.996 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:22.997 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:22.997 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:22.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:22.998 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:22.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:22.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003743595cf03327dd-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:23.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85 2023-10-17T19:42:23.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.002 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.003 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.003 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.003 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.004 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85 2023-10-17T19:42:23.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.004 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI= 2023-10-17T19:42:23.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI=', 'Content-Length': '0'} 2023-10-17T19:42:23.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6a902e2896efc5-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:23.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:23.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85 2023-10-17T19:42:23.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.008 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.009 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.009 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.009 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.010 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85 2023-10-17T19:42:23.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.010 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI= 2023-10-17T19:42:23.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VLyk9DcVCZbnHCRcZ/3Gywn9nxI=', 'Content-Length': '0'} 2023-10-17T19:42:23.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3183cc32c322c0-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.011 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj85 2023-10-17T19:42:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85 2023-10-17T19:42:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.015 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.015 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.016 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.016 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.016 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85 2023-10-17T19:42:23.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.017 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM= 2023-10-17T19:42:23.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM=', 'Content-Length': '0'} 2023-10-17T19:42:23.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-59:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d873bda0cb81a20d-00652ed95d-1b46-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj85 2023-10-17T19:42:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj85 2023-10-17T19:42:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj85 2023-10-17T19:42:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.021 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.021 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.022 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.022 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.022 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj85 2023-10-17T19:42:23.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.023 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM= 2023-10-17T19:42:23.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uKK0YI11eWSKyugWbelsBnTexyM=', 'Content-Length': '0'} 2023-10-17T19:42:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 18:56:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090feeb1b5c0168b8-00652ed95d-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.028 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.029 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.029 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.030 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.030 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.031 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:23.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:23.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194735063eb802f5-00652ed95d-1b46-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/?prefix=obj85 2023-10-17T19:42:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.036 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.036 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.037 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.037 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.037 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.038 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE= 2023-10-17T19:42:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:or/0IVb7ocTNuueZXS0+4bQeppE=', 'Content-Length': '0'} 2023-10-17T19:42:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259ea40f0dcfc338-00652ed95d-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-59obj851000falseobj852023-10-17T18:56:55.210Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.039 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-59 zones={a2, a1} 2023-10-17T19:42:23.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-59 2023-10-17T19:42:23.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 10 2023-10-17T19:42:23.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:23.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.046 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.047 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.047 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.047 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s= 2023-10-17T19:42:23.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s=', 'Content-Length': '0'} 2023-10-17T19:42:23.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005794cf73f72db3f6-00652ed95d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj19 2023-10-17T19:42:23.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj19 2023-10-17T19:42:23.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj19 2023-10-17T19:42:23.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.052 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.052 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.053 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.053 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.053 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj19 2023-10-17T19:42:23.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.054 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GqFtDwo2pWVMVnhr27t1E3r2mww= 2023-10-17T19:42:23.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqFtDwo2pWVMVnhr27t1E3r2mww='} 2023-10-17T19:42:23.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fff93f3e896d5de9-00652ed95d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:23.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.058 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.059 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.059 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.059 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.059 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.060 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s= 2023-10-17T19:42:23.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g4HSYOUMb4v5MUGsTBq3Xc+DU7s=', 'Content-Length': '0'} 2023-10-17T19:42:23.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064d66cd75faf2ad2-00652ed95d-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj38 2023-10-17T19:42:23.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj38 2023-10-17T19:42:23.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj38 2023-10-17T19:42:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.064 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.064 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.064 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.065 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:37 GMT 2023-10-17T19:42:23.065 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj38 2023-10-17T19:42:23.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.066 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Yq3mt6qSu4Ufjhw2FeACKgzV5Lk= 2023-10-17T19:42:23.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:37 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yq3mt6qSu4Ufjhw2FeACKgzV5Lk='} 2023-10-17T19:42:23.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ff964cd3cad62e2-00652ed95d-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:23.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.070 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.071 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.071 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.071 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.072 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T19:42:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T19:42:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bef770e4524e609f-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj57 2023-10-17T19:42:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj57 2023-10-17T19:42:23.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj57 2023-10-17T19:42:23.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.077 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.077 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.077 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.078 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.078 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj57 2023-10-17T19:42:23.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.079 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:DprTkP43vvfJUHt+hsp7fRRUKFk= 2023-10-17T19:42:23.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:DprTkP43vvfJUHt+hsp7fRRUKFk='} 2023-10-17T19:42:23.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5aa5d4a10fe8c82-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:23.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.083 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.083 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.084 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.084 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.084 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.085 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T19:42:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T19:42:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f2095c90c138574-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj76 2023-10-17T19:42:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj76 2023-10-17T19:42:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj76 2023-10-17T19:42:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.089 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.090 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.090 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.090 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.091 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj76 2023-10-17T19:42:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.091 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PUdUDpv8obVBxSlfOK3pRd7/4ac= 2023-10-17T19:42:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PUdUDpv8obVBxSlfOK3pRd7/4ac='} 2023-10-17T19:42:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbde9a80f76c1e4e-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/ 2023-10-17T19:42:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/ 2023-10-17T19:42:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/ 2023-10-17T19:42:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.095 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.096 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.096 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.096 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.097 INFO:tasks.rgw_multisite_tests:/hbmuog-59/ 2023-10-17T19:42:23.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.097 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk= 2023-10-17T19:42:23.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tFYnRsS/1+RJ8/WoGAPhJBNcSpk=', 'Content-Length': '0'} 2023-10-17T19:42:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080867f0a79e52643-00652ed95e-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-59/obj95 2023-10-17T19:42:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-59/obj95 2023-10-17T19:42:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-59/obj95 2023-10-17T19:42:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.102 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.102 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.102 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.103 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 18:58:38 GMT 2023-10-17T19:42:23.103 INFO:tasks.rgw_multisite_tests:/hbmuog-59/obj95 2023-10-17T19:42:23.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.103 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:yLmfQchb06wLvB6XfViqXbd89EE= 2023-10-17T19:42:23.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 18:58:38 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:yLmfQchb06wLvB6XfViqXbd89EE='} 2023-10-17T19:42:23.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a62d0fc3981a50a7-00652ed95e-162c-a1'), ('Date', 'Tue, 17 Oct 2023 18:58:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-59 2023-10-17T19:42:23.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.105 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:38.204+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:23.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:43.206+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:48.207+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:53.206+0000 7fc68f5ff780 1 waiting to reach latest gen 2, on 1.. 2023-10-17T19:42:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:58:58.208+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-17T19:42:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:03.209+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:08.211+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:13.212+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:18.215+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:23.217+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:28.217+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:33.219+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:38.221+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:43.223+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:48.224+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:53.227+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T18:59:58.229+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:03.230+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:08.231+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:13.233+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:18.235+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:23.237+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:28.240+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, , , , , , , , , , , , ] 2023-10-17T19:42:23.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:33.242+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:38.243+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:43.244+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:48.245+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:53.247+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:00:58.249+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:03.250+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:08.253+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:13.256+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:18.256+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:23.258+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:28.260+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:33.261+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:38.263+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:43.266+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:48.267+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:53.268+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:01:58.270+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:03.271+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:08.273+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:13.275+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:18.277+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:23.279+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:28.281+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:33.281+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:38.283+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:43.285+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:48.287+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:53.289+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:02:58.291+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:03.293+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:08.293+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:13.295+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:18.297+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:23.298+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:28.300+0000 7fc68f5ff780 1 waiting for incremental sync to catch up: 2023-10-17T19:42:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , 00000000002.52.6, 00000000001.60.6, , , , , , , , , , , ] 2023-10-17T19:42:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , 00000000002.52.6, 00000000004.66.6, , , , , , , , , , , ] 2023-10-17T19:42:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:33.303+0000 7fc68f5ff780 0 bucket checkpoint timed out waiting for incremental sync to catch up 2023-10-17T19:42:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:33.303+0000 7fc68f5ff780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:03:33.303+0000 7fc68f5ff780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-17T19:42:23.161 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:23.162 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.162 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:23.162 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-17T19:42:23.163 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:23.163 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:23.164 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:23.164 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:23.164 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-17T19:42:23.165 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-17T19:42:23.165 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-17T19:42:23.165 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-17T19:42:23.166 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-17T19:42:23.166 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-17T19:42:23.166 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-17T19:42:23.166 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-17T19:42:23.167 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-17T19:42:23.167 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:23.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.171 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.171 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.171 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.171 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:47 GMT 2023-10-17T19:42:23.172 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.172 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:EmOj19ly0lWQlGc//nunKKejrgk= 2023-10-17T19:42:23.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:47 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:EmOj19ly0lWQlGc//nunKKejrgk=', 'Content-Length': '0'} 2023-10-17T19:42:23.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a1413bf0fe8a5942-00652edcaf-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hbmuog-63 2023-10-17T19:42:23.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:23.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:23.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:23.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569967.530068_834.1", 2023-10-17T19:42:23.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:12:47.530068Z" 2023-10-17T19:42:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:23.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:23.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:23.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:12:47.716+0000 7feef5c09780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:12:47.717+0000 7feef5c09780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:23.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:23.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:23.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569967.523870_831.1", 2023-10-17T19:42:23.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:12:47.523870Z", 2023-10-17T19:42:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:23.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:23.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:12:47Z" 2023-10-17T19:42:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:23.205 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:23.205 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:23.205 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:23.205 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:23.206 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.206 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.207 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:23.207 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.207 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:23.208 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.208 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569967.523870_831.1", 2023-10-17T19:42:23.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.209 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:23.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:12:47.523870Z", 2023-10-17T19:42:23.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.210 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.211 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.211 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.211 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:23.212 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.212 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.214 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.214 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.215 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.215 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:23.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.216 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:23.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.217 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:23.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.218 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.218 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.218 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.219 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:23.219 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.222 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.222 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.222 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:23.223 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.223 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:23.223 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:23.223 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:23.224 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.224 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:12:47Z" 2023-10-17T19:42:23.224 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:23.225 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569967.523870_831.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:12:47.523870Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:23.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697569967.523870_831.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697569967.530068_834.1 target=1_1697569967.523870_831.1 2023-10-17T19:42:23.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.227 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:23.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:12:52.845+0000 7f3c9bf1d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:12:52.846+0000 7f3c9bf1d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:23.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569967.530068_834.1", 2023-10-17T19:42:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:12:47.530068Z", 2023-10-17T19:42:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:23.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:23.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:23.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:12:52Z" 2023-10-17T19:42:23.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:23.250 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:23.250 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:23.250 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:23.251 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:23.251 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.252 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.252 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:23.252 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.253 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:23.253 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569967.530068_834.1", 2023-10-17T19:42:23.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.254 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:23.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:12:47.530068Z", 2023-10-17T19:42:23.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.255 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.256 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.256 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.256 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:23.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.259 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.259 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.260 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.260 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:23.260 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.261 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:23.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.262 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:23.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.263 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.263 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.263 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.264 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:23.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.266 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.267 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.267 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:23.267 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.268 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:23.268 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:23.268 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:23.269 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.269 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:12:52Z" 2023-10-17T19:42:23.269 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:23.270 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697569967.530068_834.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:12:47.530068Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:23.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697569967.530068_834.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:23.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.275 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.275 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.275 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.276 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.276 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.276 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:42:23.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:42:23.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000022118850d1d6947-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj11 2023-10-17T19:42:23.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj11 2023-10-17T19:42:23.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj11 2023-10-17T19:42:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.281 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.281 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.281 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.282 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.282 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj11 2023-10-17T19:42:23.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.283 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:X4VOAnJ431mp0T1Fkq/newNtn5s= 2023-10-17T19:42:23.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:X4VOAnJ431mp0T1Fkq/newNtn5s='} 2023-10-17T19:42:23.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcbdd1e9314cff61-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.287 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.287 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.287 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.288 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.288 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.288 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:42:23.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:42:23.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba5348b87a8049c-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj22 2023-10-17T19:42:23.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj22 2023-10-17T19:42:23.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj22 2023-10-17T19:42:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.293 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.293 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.293 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.294 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.294 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj22 2023-10-17T19:42:23.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.294 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ePPKi79EB2AwDUpn+YY11XzJwGI= 2023-10-17T19:42:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ePPKi79EB2AwDUpn+YY11XzJwGI='} 2023-10-17T19:42:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7e7fafcbe807be7-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.299 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.299 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.299 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.300 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.300 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.301 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:42:23.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:42:23.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007147f49790f7f6bb-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj33 2023-10-17T19:42:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj33 2023-10-17T19:42:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj33 2023-10-17T19:42:23.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.305 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.305 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.306 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.306 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.307 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj33 2023-10-17T19:42:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.307 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:h6TrHmSBPPtbpJ4Eq7gPAidS8K4= 2023-10-17T19:42:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:h6TrHmSBPPtbpJ4Eq7gPAidS8K4='} 2023-10-17T19:42:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000addc6be10cee1190-00652edcb4-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.311 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.311 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.312 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.312 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.312 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.313 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ= 2023-10-17T19:42:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2EuVdQ03gAMwssLqqHBJCvRxPiQ=', 'Content-Length': '0'} 2023-10-17T19:42:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdd20b207a6c3c80-00652edcb4-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj44 2023-10-17T19:42:23.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj44 2023-10-17T19:42:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj44 2023-10-17T19:42:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.317 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.318 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.318 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.318 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:52 GMT 2023-10-17T19:42:23.319 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj44 2023-10-17T19:42:23.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.319 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:wLwOSAa/zK1HZNabQ/G+9VNr5+A= 2023-10-17T19:42:23.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:52 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wLwOSAa/zK1HZNabQ/G+9VNr5+A='} 2023-10-17T19:42:23.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1c375656795a35e-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.323 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.324 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.324 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.324 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.325 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.325 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c395d2d4343f8b9-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj55 2023-10-17T19:42:23.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj55 2023-10-17T19:42:23.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj55 2023-10-17T19:42:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.330 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.330 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.331 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.331 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.331 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj55 2023-10-17T19:42:23.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.332 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jMa8as22mMr4/cW3a32h+gLUCYc= 2023-10-17T19:42:23.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jMa8as22mMr4/cW3a32h+gLUCYc='} 2023-10-17T19:42:23.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aed2f822fdf863a1-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 5 2023-10-17T19:42:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.339 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.339 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.340 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.340 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d0c0f11f133e6d4-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj17 2023-10-17T19:42:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj17 2023-10-17T19:42:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj17 2023-10-17T19:42:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.345 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.345 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.346 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.346 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.346 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj17 2023-10-17T19:42:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.347 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:xK3iM9cTLv2tLaL91hm2TNQFbJ8= 2023-10-17T19:42:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:xK3iM9cTLv2tLaL91hm2TNQFbJ8='} 2023-10-17T19:42:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002470bed8ba409cc1-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.352 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.352 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.352 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.353 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.353 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000466fbf998b5a3db7-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj34 2023-10-17T19:42:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj34 2023-10-17T19:42:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj34 2023-10-17T19:42:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.358 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.358 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.359 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.359 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.359 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj34 2023-10-17T19:42:23.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.360 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rr2eEDfl7QI1BoFCgDYoOfV6Pms= 2023-10-17T19:42:23.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rr2eEDfl7QI1BoFCgDYoOfV6Pms='} 2023-10-17T19:42:23.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb2be91392f92750-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.364 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.365 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.365 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.365 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.366 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.366 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c9677f742e2b304-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj51 2023-10-17T19:42:23.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj51 2023-10-17T19:42:23.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj51 2023-10-17T19:42:23.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.371 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.371 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.372 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.372 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.372 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj51 2023-10-17T19:42:23.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.373 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Aw+cfBI7g+xEDBrD4OJhZLmLsc8= 2023-10-17T19:42:23.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Aw+cfBI7g+xEDBrD4OJhZLmLsc8='} 2023-10-17T19:42:23.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000237d5467646bc278-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.377 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.377 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.378 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.378 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.378 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.379 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ded2a2b032291c6-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj68 2023-10-17T19:42:23.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj68 2023-10-17T19:42:23.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj68 2023-10-17T19:42:23.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.383 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.383 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.383 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.384 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.384 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj68 2023-10-17T19:42:23.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.385 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Cn9JIMoF7cyoWDwgeUF0Vtd8oC0= 2023-10-17T19:42:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Cn9JIMoF7cyoWDwgeUF0Vtd8oC0='} 2023-10-17T19:42:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021297b4e501733db-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.389 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.389 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.389 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.390 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.390 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.391 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c2a059245010964-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj85 2023-10-17T19:42:23.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj85 2023-10-17T19:42:23.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj85 2023-10-17T19:42:23.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.395 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.395 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.395 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.396 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.396 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj85 2023-10-17T19:42:23.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.397 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cKsiXoZU3IosbyNhZtksGH+kO9w= 2023-10-17T19:42:23.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cKsiXoZU3IosbyNhZtksGH+kO9w='} 2023-10-17T19:42:23.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a98be37e17dbf785-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 10 2023-10-17T19:42:23.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.403 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.404 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.404 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.404 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.404 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.405 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc15875be281094d-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj19 2023-10-17T19:42:23.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj19 2023-10-17T19:42:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj19 2023-10-17T19:42:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.409 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.410 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.410 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.410 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.411 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj19 2023-10-17T19:42:23.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.411 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:I4QBaBnCwazh5QxOO/42CMxNsT8= 2023-10-17T19:42:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:I4QBaBnCwazh5QxOO/42CMxNsT8='} 2023-10-17T19:42:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a5471a0329804c-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.416 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.416 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.416 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.417 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.417 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.418 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006e28914e9a1a7c8-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj38 2023-10-17T19:42:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj38 2023-10-17T19:42:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj38 2023-10-17T19:42:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.422 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.423 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.423 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.423 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.424 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj38 2023-10-17T19:42:23.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.424 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2YHUNhqi4s4kysb3yUnWRn6Ie3Y= 2023-10-17T19:42:23.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2YHUNhqi4s4kysb3yUnWRn6Ie3Y='} 2023-10-17T19:42:23.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e99a823e97536351-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.429 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.429 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.429 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.430 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.430 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.431 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3caf8e0d60ae03e-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj57 2023-10-17T19:42:23.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj57 2023-10-17T19:42:23.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj57 2023-10-17T19:42:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.435 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.435 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.435 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.436 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.436 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj57 2023-10-17T19:42:23.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.436 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vtaNPWGFMFxh0nLfbLL0DEsc/a8= 2023-10-17T19:42:23.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vtaNPWGFMFxh0nLfbLL0DEsc/a8='} 2023-10-17T19:42:23.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5cf50f1045b6fee-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.441 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.441 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.441 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.442 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.442 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.442 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67e15752570d161-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj76 2023-10-17T19:42:23.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj76 2023-10-17T19:42:23.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj76 2023-10-17T19:42:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.447 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.447 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.448 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.448 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.448 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj76 2023-10-17T19:42:23.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.449 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GrD79vHtf01f/jSxSFjv8bqJIuQ= 2023-10-17T19:42:23.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GrD79vHtf01f/jSxSFjv8bqJIuQ='} 2023-10-17T19:42:23.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9edd056705e4f00-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.453 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.453 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.454 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.454 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.454 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.455 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073d76fc93a7c2bd0-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj95 2023-10-17T19:42:23.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj95 2023-10-17T19:42:23.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj95 2023-10-17T19:42:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.459 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.460 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.460 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.460 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.460 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj95 2023-10-17T19:42:23.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.461 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4YSq8s8VmtvPo2NRfiF4m7IlzV8= 2023-10-17T19:42:23.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4YSq8s8VmtvPo2NRfiF4m7IlzV8='} 2023-10-17T19:42:23.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000088e6bb75a4a1252-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 15 2023-10-17T19:42:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.468 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.468 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.469 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.469 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.469 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.470 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001b5f11f18fad35d-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj23 2023-10-17T19:42:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj23 2023-10-17T19:42:23.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj23 2023-10-17T19:42:23.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.474 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.475 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.475 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.475 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.476 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj23 2023-10-17T19:42:23.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.476 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:a/6F7Dv+F0A0+/fBPT0XjLa77GE= 2023-10-17T19:42:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:a/6F7Dv+F0A0+/fBPT0XjLa77GE='} 2023-10-17T19:42:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050b551531d08545f-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.481 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.481 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.481 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.482 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.482 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.483 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083473a796585242c-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj46 2023-10-17T19:42:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj46 2023-10-17T19:42:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj46 2023-10-17T19:42:23.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.487 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.487 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.488 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.488 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj46 2023-10-17T19:42:23.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.489 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PbsMFiOK4YwhdNWBDxqLiZ1HmLU= 2023-10-17T19:42:23.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PbsMFiOK4YwhdNWBDxqLiZ1HmLU='} 2023-10-17T19:42:23.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aed36297b490c6ce-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.493 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.493 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.493 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.494 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.494 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.495 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225314936f8bd1fe-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj69 2023-10-17T19:42:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj69 2023-10-17T19:42:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj69 2023-10-17T19:42:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.499 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.499 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.499 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.499 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.500 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj69 2023-10-17T19:42:23.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.500 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zA/LEZAdscZNkGBGaUz31zk9kRs= 2023-10-17T19:42:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zA/LEZAdscZNkGBGaUz31zk9kRs='} 2023-10-17T19:42:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fd924f792b16312-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.505 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.505 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.506 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.506 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.507 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo= 2023-10-17T19:42:23.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:SUD+V4UkgeAEw32HjttpmW37BBo=', 'Content-Length': '0'} 2023-10-17T19:42:23.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9c1ef388f7c6f45-00652edcb5-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj92 2023-10-17T19:42:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj92 2023-10-17T19:42:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj92 2023-10-17T19:42:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.511 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.511 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.512 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.512 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:53 GMT 2023-10-17T19:42:23.512 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj92 2023-10-17T19:42:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.513 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:LS1IVM8up4gHXNTXy0GZjHy/vYk= 2023-10-17T19:42:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:53 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:LS1IVM8up4gHXNTXy0GZjHy/vYk='} 2023-10-17T19:42:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f3cb137b63c5ae2-00652edcb5-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.517 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.518 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.518 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.518 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.519 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.519 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000234ed1958a08bd3a-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj115 2023-10-17T19:42:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj115 2023-10-17T19:42:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj115 2023-10-17T19:42:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.524 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.524 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.525 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.525 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.525 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj115 2023-10-17T19:42:23.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.526 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:udJ3x2v9Cz6gS2qsp29vWgIMS88= 2023-10-17T19:42:23.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:udJ3x2v9Cz6gS2qsp29vWgIMS88='} 2023-10-17T19:42:23.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb22d061c8b5e401-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 20 2023-10-17T19:42:23.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.533 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.533 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.533 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.534 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.534 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.534 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f081a8addecf9521-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj29 2023-10-17T19:42:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj29 2023-10-17T19:42:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj29 2023-10-17T19:42:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.539 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.539 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.540 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.540 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj29 2023-10-17T19:42:23.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.540 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:iMExVY69L5fjlJJIdhWD3ycLlxg= 2023-10-17T19:42:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:iMExVY69L5fjlJJIdhWD3ycLlxg='} 2023-10-17T19:42:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd3bd5e2d1c2d281-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.545 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.545 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.545 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.546 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.546 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.547 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007df01919c44158f0-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj58 2023-10-17T19:42:23.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj58 2023-10-17T19:42:23.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj58 2023-10-17T19:42:23.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.551 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.551 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.552 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.552 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj58 2023-10-17T19:42:23.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.553 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8c/dThgnUS9NCY0/rSiVwMI3lsU= 2023-10-17T19:42:23.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8c/dThgnUS9NCY0/rSiVwMI3lsU='} 2023-10-17T19:42:23.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f973fed276f1325b-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.557 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.558 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.558 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.558 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.559 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.559 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e912522fbf5f5b5-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj87 2023-10-17T19:42:23.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj87 2023-10-17T19:42:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj87 2023-10-17T19:42:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.563 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.564 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.564 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.564 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.565 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj87 2023-10-17T19:42:23.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.565 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hSMQ7CbgzTZHD5HyolJWjejc+hE= 2023-10-17T19:42:23.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hSMQ7CbgzTZHD5HyolJWjejc+hE='} 2023-10-17T19:42:23.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd1cf8c0fde5a2f-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.570 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.570 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.570 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.570 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.571 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.571 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b847658bce39b9-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj116 2023-10-17T19:42:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj116 2023-10-17T19:42:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj116 2023-10-17T19:42:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.576 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.576 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.576 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.577 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.577 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj116 2023-10-17T19:42:23.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.577 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:N18AT6MhnhWeHmmZ0J2K2w9JzgE= 2023-10-17T19:42:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:N18AT6MhnhWeHmmZ0J2K2w9JzgE='} 2023-10-17T19:42:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ddd764b158d4505-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.582 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.582 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.583 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.583 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.583 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.584 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be2b32041317930e-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj145 2023-10-17T19:42:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj145 2023-10-17T19:42:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj145 2023-10-17T19:42:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.588 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.588 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.589 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.589 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.589 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj145 2023-10-17T19:42:23.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.590 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hOs3Tq7amC3ifotsmgXcr6SKwjc= 2023-10-17T19:42:23.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hOs3Tq7amC3ifotsmgXcr6SKwjc='} 2023-10-17T19:42:23.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb7951bea5e9e03c-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 25 2023-10-17T19:42:23.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.597 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.597 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.598 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.598 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.598 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.599 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001803f100f57add0-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj31 2023-10-17T19:42:23.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj31 2023-10-17T19:42:23.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj31 2023-10-17T19:42:23.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.603 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.604 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.604 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.604 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.605 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj31 2023-10-17T19:42:23.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.605 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:wrKfsWpPqurMtCWI4I9P82bwVzE= 2023-10-17T19:42:23.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:wrKfsWpPqurMtCWI4I9P82bwVzE='} 2023-10-17T19:42:23.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000960866292ab729ef-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.610 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.610 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.611 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.611 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.611 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.612 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c27f5760e288fb62-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj62 2023-10-17T19:42:23.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj62 2023-10-17T19:42:23.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj62 2023-10-17T19:42:23.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.616 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.617 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.617 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.617 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.618 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj62 2023-10-17T19:42:23.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.619 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2BwDpJ/fdA3Q7+0pfDCMN8hHfMk= 2023-10-17T19:42:23.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2BwDpJ/fdA3Q7+0pfDCMN8hHfMk='} 2023-10-17T19:42:23.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bbdfd04dd65629c-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.623 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.623 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.624 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.624 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.624 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.625 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209966d0ebf50e80-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj93 2023-10-17T19:42:23.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj93 2023-10-17T19:42:23.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj93 2023-10-17T19:42:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.629 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.630 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.630 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.630 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.630 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj93 2023-10-17T19:42:23.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.631 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:YCiUIuGplwHMcj/U9SIUEnqtiDI= 2023-10-17T19:42:23.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:YCiUIuGplwHMcj/U9SIUEnqtiDI='} 2023-10-17T19:42:23.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc45f8c07a87db93-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.636 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.636 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.636 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.637 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.637 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d2aef81c7028db-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj124 2023-10-17T19:42:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj124 2023-10-17T19:42:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj124 2023-10-17T19:42:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.642 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.642 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.642 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.643 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.643 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj124 2023-10-17T19:42:23.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.644 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:hKRDHXNo7c2d7It6wjjzSk1jA7g= 2023-10-17T19:42:23.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:hKRDHXNo7c2d7It6wjjzSk1jA7g='} 2023-10-17T19:42:23.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c267673b1bdb32f-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.649 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.649 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.649 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.650 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.650 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.651 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003197e7259131b781-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj155 2023-10-17T19:42:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj155 2023-10-17T19:42:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj155 2023-10-17T19:42:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.655 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.655 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.655 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.656 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.656 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj155 2023-10-17T19:42:23.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.657 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Y8qELTEyvahGHt1rK4HO4nio6+c= 2023-10-17T19:42:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Y8qELTEyvahGHt1rK4HO4nio6+c='} 2023-10-17T19:42:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079f4fca71e1c8442-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:23.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:23.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:23.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-63 2023-10-17T19:42:23.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 30 2023-10-17T19:42:23.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.663 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.664 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.664 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.664 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.665 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.665 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfb6e48ec1d6b44c-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj37 2023-10-17T19:42:23.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj37 2023-10-17T19:42:23.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj37 2023-10-17T19:42:23.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.670 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.670 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.670 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.671 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.671 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj37 2023-10-17T19:42:23.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.672 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:OUxceuf2nsV4FQmCcF6Lv7iEr/M= 2023-10-17T19:42:23.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:OUxceuf2nsV4FQmCcF6Lv7iEr/M='} 2023-10-17T19:42:23.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008046291ffb5550b8-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.676 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.677 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.677 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.677 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.678 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.678 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A= 2023-10-17T19:42:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:UG0T+JcA8Cq7z6rt8LrWl6hpG9A=', 'Content-Length': '0'} 2023-10-17T19:42:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a396fe478dc6cc28-00652edcb6-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj74 2023-10-17T19:42:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj74 2023-10-17T19:42:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj74 2023-10-17T19:42:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.683 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.683 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.683 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.683 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:54 GMT 2023-10-17T19:42:23.684 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj74 2023-10-17T19:42:23.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.684 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vzJOH3FOpTotFoIboOoIlJpNFhI= 2023-10-17T19:42:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:54 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vzJOH3FOpTotFoIboOoIlJpNFhI='} 2023-10-17T19:42:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e474278bf9d3cef0-00652edcb6-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.689 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.689 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.689 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.690 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.690 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.690 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:42:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:42:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8de06b70da1674e-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj111 2023-10-17T19:42:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj111 2023-10-17T19:42:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj111 2023-10-17T19:42:23.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.695 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.695 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.696 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.696 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj111 2023-10-17T19:42:23.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.697 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zU0Kw+bLn8wkNDA94jPT1goGzeg= 2023-10-17T19:42:23.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zU0Kw+bLn8wkNDA94jPT1goGzeg='} 2023-10-17T19:42:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a61d7f7b1ccb266-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.702 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.702 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.703 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.703 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.704 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:42:23.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:42:23.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042daa328b4b22e60-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj148 2023-10-17T19:42:23.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj148 2023-10-17T19:42:23.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj148 2023-10-17T19:42:23.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.708 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.708 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.709 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.709 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.709 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj148 2023-10-17T19:42:23.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.710 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ea6o1y89+c8mTjEyaTB2Lt9Zx38= 2023-10-17T19:42:23.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ea6o1y89+c8mTjEyaTB2Lt9Zx38='} 2023-10-17T19:42:23.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000601d7b8f5ec18f2a-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/ 2023-10-17T19:42:23.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/ 2023-10-17T19:42:23.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/ 2023-10-17T19:42:23.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.714 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.714 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.715 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.715 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.715 INFO:tasks.rgw_multisite_tests:/hbmuog-63/ 2023-10-17T19:42:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.716 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0= 2023-10-17T19:42:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/BSAkKupMfHHq+pBCsYE2nWxhV0=', 'Content-Length': '0'} 2023-10-17T19:42:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000214e5c6cb9f2e252-00652edcb7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-63/obj185 2023-10-17T19:42:23.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-63/obj185 2023-10-17T19:42:23.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-63/obj185 2023-10-17T19:42:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.720 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.720 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.720 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.721 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:12:55 GMT 2023-10-17T19:42:23.721 INFO:tasks.rgw_multisite_tests:/hbmuog-63/obj185 2023-10-17T19:42:23.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.722 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sQbeXEzKAbJI3JFcfVzNtJJSQlI= 2023-10-17T19:42:23.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:12:55 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sQbeXEzKAbJI3JFcfVzNtJJSQlI='} 2023-10-17T19:42:23.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0e11cab72c120c2-00652edcb7-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-63 2023-10-17T19:42:23.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:12:55.248+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:00.249+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:05.250+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:10.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:15.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:20.251+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:25.252+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:30.253+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:35.254+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:40.255+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:45.256+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:50.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:13:55.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:00.257+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:05.258+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:10.259+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:15.260+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:20.261+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:25.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:30.261+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:35.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:40.262+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:45.263+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:50.264+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:14:55.265+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:00.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:05.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:10.266+0000 7fadb5999780 1 waiting to reach latest gen 6, on 2.. 2023-10-17T19:42:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:15.267+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:20.267+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:25.268+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:30.269+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:35.270+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:40.271+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:45.270+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:50.271+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:15:55.272+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:00.272+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:05.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:10.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:15.276+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:20.275+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:25.276+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:30.277+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:35.277+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:40.278+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:45.279+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:50.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:16:55.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:00.280+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:05.281+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:10.281+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:15.283+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:20.284+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:25.285+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:30.284+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:35.285+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:40.286+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:45.287+0000 7fadb5999780 1 waiting to reach latest gen 6, on 3.. 2023-10-17T19:42:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.288+0000 7fadb5999780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-17T19:42:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.288+0000 7fadb5999780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.288+0000 7fadb5999780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-17T19:42:23.744 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:23.745 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.745 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:23.746 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-17T19:42:23.746 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:23.747 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:23.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:23.748 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:23.748 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-17T19:42:23.748 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-17T19:42:23.749 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-17T19:42:23.749 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-17T19:42:23.749 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-17T19:42:23.750 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-17T19:42:23.750 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-17T19:42:23.750 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-17T19:42:23.751 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-17T19:42:23.751 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:23.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:23.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.755 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.755 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.755 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.756 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.756 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.756 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ohoEmQ8NrhPucVc4lWZW4b+mnEk= 2023-10-17T19:42:23.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ohoEmQ8NrhPucVc4lWZW4b+mnEk=', 'Content-Length': '0'} 2023-10-17T19:42:23.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000632af682a691b72f-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hbmuog-64 2023-10-17T19:42:23.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:23.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697570270.335817_873.1", 2023-10-17T19:42:23.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:17:50.335817Z" 2023-10-17T19:42:23.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:23.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:23.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:23.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:23.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:23.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.536+0000 7f511e104780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.537+0000 7f511e104780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:23.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697570270.335817_873.1", 2023-10-17T19:42:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:17:50.335817Z", 2023-10-17T19:42:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:17:50Z" 2023-10-17T19:42:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:23.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:23.789 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:23.789 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:23.790 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:23.790 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:23.790 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:23.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.791 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:23.792 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.792 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:23.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1697570270.335817_873.1", 2023-10-17T19:42:23.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.794 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:23.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:17:50.335817Z", 2023-10-17T19:42:23.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.795 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.795 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.796 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.796 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:23.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:23.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:23.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.799 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.799 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.799 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.800 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:23.800 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.800 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:23.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.802 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:23.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.802 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.803 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.803 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:23.803 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:23.804 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:23.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:23.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:23.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:23.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:23.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:23.806 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:23.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:23.806 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.807 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:23.807 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:23.807 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.808 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:23.808 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:23.808 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:23.809 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:23.809 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:17:50Z" 2023-10-17T19:42:23.809 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:23.810 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570270.335817_873.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:17:50.335817Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:23.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697570270.335817_873.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:23.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:23.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.815 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.815 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.815 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.816 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.816 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.816 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:42:23.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:42:23.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f584df89f6d04b8-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11 2023-10-17T19:42:23.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.821 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.821 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.821 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.822 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.822 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11 2023-10-17T19:42:23.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.823 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:q38I4sEkUlx3+iS384+ktxu34Ps= 2023-10-17T19:42:23.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:q38I4sEkUlx3+iS384+ktxu34Ps='} 2023-10-17T19:42:23.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d9e11caf2ce72e5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.827 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.827 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.827 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.828 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.828 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.829 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:42:23.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:42:23.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004464e8d368521ca-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:23.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:23.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22 2023-10-17T19:42:23.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.833 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.833 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.834 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.834 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.834 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22 2023-10-17T19:42:23.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.835 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:urgUbFqKRzW9AD1XmfO0l9HFfoA= 2023-10-17T19:42:23.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:urgUbFqKRzW9AD1XmfO0l9HFfoA='} 2023-10-17T19:42:23.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b50c11b982931b4-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.839 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.839 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.840 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.840 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.840 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.841 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:42:23.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:42:23.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082623cd3f58820ea-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:23.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:23.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33 2023-10-17T19:42:23.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.845 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.846 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.846 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.846 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.847 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33 2023-10-17T19:42:23.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.847 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:cni1zWucfyHJxLAtWA1JgAq7kwo= 2023-10-17T19:42:23.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:cni1zWucfyHJxLAtWA1JgAq7kwo='} 2023-10-17T19:42:23.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd8b8354ce8bd8e5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.852 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.852 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.852 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.853 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.853 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.854 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:42:23.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:42:23.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f799dcf04e8cbf21-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:23.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:23.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44 2023-10-17T19:42:23.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.858 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.859 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.859 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.859 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.860 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44 2023-10-17T19:42:23.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.860 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oYQpYkY7Kqhsa9vJ0ouJHamj6bQ= 2023-10-17T19:42:23.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oYQpYkY7Kqhsa9vJ0ouJHamj6bQ='} 2023-10-17T19:42:23.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099d45756a0152d7b-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.865 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.865 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.865 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.865 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.866 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.866 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k= 2023-10-17T19:42:23.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:kReMfXn4BDwtcOIO1IecaVxYh2k=', 'Content-Length': '0'} 2023-10-17T19:42:23.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015680c21ee493f6f-00652eddde-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:23.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:23.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:23.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55 2023-10-17T19:42:23.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:23.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.871 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:23.871 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:23.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:23.872 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:17:50 GMT 2023-10-17T19:42:23.872 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55 2023-10-17T19:42:23.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.873 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Vc41eCl6JLWmJaP83dHo2rWSav8= 2023-10-17T19:42:23.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:17:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vc41eCl6JLWmJaP83dHo2rWSav8='} 2023-10-17T19:42:23.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000418c68aecc28d7c5-00652eddde-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-64 2023-10-17T19:42:23.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:50.825+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:42:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:17:55.827+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:42:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:00.828+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:42:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:05.828+0000 7fa1f118f780 1 waiting to reach incremental sync.. 2023-10-17T19:42:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:10.829+0000 7fa1f118f780 1 bucket sync caught up with source: 2023-10-17T19:42:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , 00000000001.185.6, 00000000002.91.6, 00000000002.114.6] 2023-10-17T19:42:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.185.6, 00000000002.91.6, 00000000002.114.6] 2023-10-17T19:42:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:10.829+0000 7fa1f118f780 0 bucket checkpoint complete 2023-10-17T19:42:23.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hbmuog-64 2023-10-17T19:42:23.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:10.949+0000 7f0db6a7d780 1 waiting to reach incremental sync.. 2023-10-17T19:42:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:15.951+0000 7f0db6a7d780 1 bucket sync caught up with source: 2023-10-17T19:42:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: local status: [, , , , , , , , 00000000001.484.6, 00000000002.158.6, 00000000002.398.6] 2023-10-17T19:42:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: remote markers: [, , , , , , , , 00000000001.484.6, 00000000002.158.6, 00000000002.398.6] 2023-10-17T19:42:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:15.951+0000 7f0db6a7d780 0 bucket checkpoint complete 2023-10-17T19:42:23.881 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hbmuog-64 zones={a2, a1} 2023-10-17T19:42:23.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:23.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.886 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.886 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.886 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.887 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:15 GMT 2023-10-17T19:42:23.887 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.888 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4= 2023-10-17T19:42:23.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4=', 'Content-Length': '0'} 2023-10-17T19:42:23.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb418fcc4d8c55f1-00652eddf7-1d4d-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:23.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.892 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.892 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.892 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.892 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:15 GMT 2023-10-17T19:42:23.893 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.893 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4= 2023-10-17T19:42:23.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MBMA+uxu4yyy+e4TZ1bSqaYDA+4=', 'Content-Length': '0'} 2023-10-17T19:42:23.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b9efd1337762e0f-00652eddf7-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-17T19:42:23.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.899 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.899 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.900 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.900 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:15 GMT 2023-10-17T19:42:23.900 INFO:tasks.rgw_multisite_tests:/hbmuog-64/?versions 2023-10-17T19:42:23.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.901 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:CjyZJ+6T9mg+paddRGzexX+NJuk= 2023-10-17T19:42:23.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:15 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:CjyZJ+6T9mg+paddRGzexX+NJuk=', 'Content-Length': '0'} 2023-10-17T19:42:23.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e643ee8952f624-00652eddf7-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.902 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:23.903 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:23.903 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:23.903 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:23.903 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:23.904 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-17T19:42:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.908 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.909 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.909 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.909 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.910 INFO:tasks.rgw_multisite_tests:/hbmuog-64/?versions 2023-10-17T19:42:23.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.910 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:42:23.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:42:23.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa481eb620b7abd-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-17T19:42:23.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-17T19:42:23.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-17T19:42:23.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-17T19:42:23.913 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-17T19:42:23.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.917 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.918 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.918 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.918 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.918 INFO:tasks.rgw_multisite_tests:/hbmuog-64/?versions 2023-10-17T19:42:23.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.919 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:42:23.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:42:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4cab6b3de5668e-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?versions&max-keys=999 2023-10-17T19:42:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.924 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.925 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.925 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.925 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.926 INFO:tasks.rgw_multisite_tests:/hbmuog-64/?versions 2023-10-17T19:42:23.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.926 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI= 2023-10-17T19:42:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lnX1sWlGRUyqQz7a270B96dc2lI=', 'Content-Length': '0'} 2023-10-17T19:42:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5756d3ecb16b47-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64999falseobj11nulltrue2023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.932 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.932 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.932 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.932 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.933 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.933 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:42:23.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:42:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5c9a7f53d438c3-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.938 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.938 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.939 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.939 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.939 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.940 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:42:23.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:42:23.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afe955e940727cd-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.945 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.945 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.946 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.946 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.946 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.947 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA= 2023-10-17T19:42:23.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA=', 'Content-Length': '0'} 2023-10-17T19:42:23.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe71834a2302ea2a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.952 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.953 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.953 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.953 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.954 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.954 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA= 2023-10-17T19:42:23.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GqoxXjt1RPBIEmzirLUU99LHSfA=', 'Content-Length': '0'} 2023-10-17T19:42:23.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003133f98bd1b97c3c-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.955 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:23.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.960 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.960 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.960 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.961 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.961 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.962 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:42:23.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:42:23.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b9bda415bf781a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.967 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.967 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.967 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.967 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.968 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11?versionId=null 2023-10-17T19:42:23.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.968 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE= 2023-10-17T19:42:23.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:z24KInBl22mMYprupoCJq/UegKE=', 'Content-Length': '0'} 2023-10-17T19:42:23.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435ef6b811b4c4e3-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.974 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.974 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.974 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.975 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.975 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.976 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:23.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:23.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb78ef927390fb84-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:23.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:23.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:42:23.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.981 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:23.982 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.982 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.982 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.983 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:23.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.983 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:23.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:23.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8146735e6b9e5bd-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:23.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11 2023-10-17T19:42:23.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.988 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.988 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.988 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.988 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.989 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11 2023-10-17T19:42:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.989 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8= 2023-10-17T19:42:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8=', 'Content-Length': '0'} 2023-10-17T19:42:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002575848d76cd631f-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:23.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11 2023-10-17T19:42:23.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:23.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:23.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:23.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:23.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:23.994 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:23.994 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.995 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:23.995 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:23.995 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11 2023-10-17T19:42:23.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:23.996 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8= 2023-10-17T19:42:23.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7U0zVOB4hrZs6V8yCG0vh0Kpbb8=', 'Content-Length': '0'} 2023-10-17T19:42:23.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b23f7bea02c553aa-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:23.997 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-17T19:42:23.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:23.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:23.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:23.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11 2023-10-17T19:42:23.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:23.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:23.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:23.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.000 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.001 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.001 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.001 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.002 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11 2023-10-17T19:42:24.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.002 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI= 2023-10-17T19:42:24.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI=', 'Content-Length': '0'} 2023-10-17T19:42:24.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010246b53cd73547d-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj11 2023-10-17T19:42:24.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj11 2023-10-17T19:42:24.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj11 2023-10-17T19:42:24.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.006 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.007 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.007 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.007 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.007 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj11 2023-10-17T19:42:24.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.008 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI= 2023-10-17T19:42:24.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6kaEXySCQGVX1oylgpJ2uYDCDAI=', 'Content-Length': '0'} 2023-10-17T19:42:24.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000dfe4a4d2b88798-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.013 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.014 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.014 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.014 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.015 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.015 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc7975c3426d7e1-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj11 2023-10-17T19:42:24.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.021 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.021 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.022 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.022 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.022 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.023 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbcb36e6c5d9e82f-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj111000falseobj112023-10-17T19:17:50.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:24.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.028 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.029 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.029 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.029 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.029 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.030 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:42:24.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:42:24.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3410c9ffa4eb14-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.035 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.036 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.036 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.036 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.036 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.037 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:42:24.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:42:24.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aeea560c79e8971-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.042 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.042 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.043 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.043 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.043 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.044 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ= 2023-10-17T19:42:24.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ=', 'Content-Length': '0'} 2023-10-17T19:42:24.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d16304f6337994-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.049 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.050 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.050 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.050 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.051 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.051 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ= 2023-10-17T19:42:24.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:IZyPXJMZb9V5Bap/zM0/HV2q1xQ=', 'Content-Length': '0'} 2023-10-17T19:42:24.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bf1ab5f1ea3c3c-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.052 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:24.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.056 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.057 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.057 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.057 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.057 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.058 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:42:24.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:42:24.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc15af1596dffe8a-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.063 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.063 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.064 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.064 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.064 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22?versionId=null 2023-10-17T19:42:24.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.065 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80= 2023-10-17T19:42:24.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7/jhjev6q2Z7w8GjSxjffzy+h80=', 'Content-Length': '0'} 2023-10-17T19:42:24.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5a8e7f9d1c2591-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.070 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.070 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.070 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.071 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.071 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.072 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003932d7868cfcde58-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.077 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.077 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.078 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.078 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.078 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.079 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a57b651dfa6f5f6-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22 2023-10-17T19:42:24.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.084 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.084 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.084 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.085 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.085 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22 2023-10-17T19:42:24.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.086 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U= 2023-10-17T19:42:24.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U=', 'Content-Length': '0'} 2023-10-17T19:42:24.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074189c29a7081fd2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22 2023-10-17T19:42:24.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.090 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.090 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.091 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.091 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.091 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22 2023-10-17T19:42:24.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.092 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U= 2023-10-17T19:42:24.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uIwR3IemEnrRK9P0prRbhFyMc7U=', 'Content-Length': '0'} 2023-10-17T19:42:24.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50ccfb9e40300aa-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.093 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-17T19:42:24.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22 2023-10-17T19:42:24.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.096 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.097 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.097 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.097 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.098 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22 2023-10-17T19:42:24.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.098 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4= 2023-10-17T19:42:24.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4=', 'Content-Length': '0'} 2023-10-17T19:42:24.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103034f2d907f97f-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj22 2023-10-17T19:42:24.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj22 2023-10-17T19:42:24.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj22 2023-10-17T19:42:24.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.102 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.103 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.103 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.103 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.104 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj22 2023-10-17T19:42:24.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.104 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4= 2023-10-17T19:42:24.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:QAQq6vBnmy62+j0N9ujymc9Ndg4=', 'Content-Length': '0'} 2023-10-17T19:42:24.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa35a354a64e4b8-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.109 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.109 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.109 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.110 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.110 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.111 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a45b89e25a18a73-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj22 2023-10-17T19:42:24.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.116 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.116 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.117 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.117 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.117 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.118 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf2de6d7c080b1f-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj221000falseobj222023-10-17T19:17:50.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.119 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:24.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.123 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.124 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.124 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.124 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.125 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.125 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:42:24.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:42:24.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86deefcf1164cae-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.130 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.131 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.131 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.131 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.132 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.132 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:42:24.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:42:24.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ccd11cbc39c596-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.137 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.138 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.138 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.138 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.139 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.139 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg= 2023-10-17T19:42:24.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg=', 'Content-Length': '0'} 2023-10-17T19:42:24.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad9816d501c3868-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.145 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.145 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.145 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.146 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.146 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.147 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg= 2023-10-17T19:42:24.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:mcgF4uVzyxeAPiJCWGmdV2GmFjg=', 'Content-Length': '0'} 2023-10-17T19:42:24.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab8d859902515e6-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:24.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.152 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.152 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.153 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.153 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.153 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.154 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:42:24.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:42:24.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a546f42116aa83-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.159 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.159 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.159 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.159 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.160 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33?versionId=null 2023-10-17T19:42:24.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.160 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw= 2023-10-17T19:42:24.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:lbdzzg4eKGi6z4DoO+o7IIMx0sw=', 'Content-Length': '0'} 2023-10-17T19:42:24.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db5fb901655b756-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.165 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.166 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.166 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.166 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.167 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.167 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfbe72aa21bddac7-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.173 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.173 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.174 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.174 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.175 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad111c8dc8bbb81-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33 2023-10-17T19:42:24.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.179 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.180 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.180 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.181 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33 2023-10-17T19:42:24.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.181 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A= 2023-10-17T19:42:24.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A=', 'Content-Length': '0'} 2023-10-17T19:42:24.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ce630e1e5c8bf4-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33 2023-10-17T19:42:24.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.185 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.186 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.186 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.186 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.187 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33 2023-10-17T19:42:24.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.187 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A= 2023-10-17T19:42:24.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:HbpM2ejfo0g7B5SH+ytP46GTW7A=', 'Content-Length': '0'} 2023-10-17T19:42:24.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e73c0b1e110f4ffe-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.188 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-17T19:42:24.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33 2023-10-17T19:42:24.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.192 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.193 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.193 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.193 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.193 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33 2023-10-17T19:42:24.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.194 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o= 2023-10-17T19:42:24.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o=', 'Content-Length': '0'} 2023-10-17T19:42:24.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca68f9b03a44007-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj33 2023-10-17T19:42:24.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj33 2023-10-17T19:42:24.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj33 2023-10-17T19:42:24.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.198 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.199 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.199 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.199 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.200 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj33 2023-10-17T19:42:24.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.200 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o= 2023-10-17T19:42:24.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:FHtNpLhtCuYgVSakyE00Gr9Nx8o=', 'Content-Length': '0'} 2023-10-17T19:42:24.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006465aae08bd5f59e-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.205 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.205 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.206 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.206 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.206 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.207 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c6e150b6c984cd1-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj33 2023-10-17T19:42:24.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.212 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.213 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.213 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.213 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.213 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.214 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55ee0617a4187e9-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj331000falseobj332023-10-17T19:17:50.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.215 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:24.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.220 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.221 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.221 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.222 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:42:24.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:42:24.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624b8a0d917c7d78-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.227 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.227 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.227 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.228 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.228 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.228 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:42:24.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:42:24.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e0fb8d59c2204e1-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.233 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.234 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.234 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.234 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.235 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.235 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w= 2023-10-17T19:42:24.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w=', 'Content-Length': '0'} 2023-10-17T19:42:24.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0295afd75abe64-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.240 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.241 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.241 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.242 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.242 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w= 2023-10-17T19:42:24.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JoV4zmsqJqhX1YiczqpLf81Hm3w=', 'Content-Length': '0'} 2023-10-17T19:42:24.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89391769e1b600a-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.244 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:24.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.249 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.249 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.249 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.250 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.250 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.251 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:42:24.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:42:24.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddbf17dee2a185b-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.256 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.256 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.256 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.257 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.257 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44?versionId=null 2023-10-17T19:42:24.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.258 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc= 2023-10-17T19:42:24.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Yglcc2yytX8bkmkbKKEA5frDRGc=', 'Content-Length': '0'} 2023-10-17T19:42:24.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324e0dbffb99f065-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.262 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.263 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.263 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.263 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.264 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.264 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c296033b03284b2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.270 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.270 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.271 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.271 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.271 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.272 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000318887b12ee78244-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44 2023-10-17T19:42:24.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.277 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.277 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.278 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.278 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.278 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44 2023-10-17T19:42:24.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.279 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg= 2023-10-17T19:42:24.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg=', 'Content-Length': '0'} 2023-10-17T19:42:24.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004654b8392ec3ba36-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44 2023-10-17T19:42:24.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.283 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.284 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.284 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.284 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.285 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44 2023-10-17T19:42:24.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.285 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg= 2023-10-17T19:42:24.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/vQP1txc0VE5DDbcDI/sxZQTffg=', 'Content-Length': '0'} 2023-10-17T19:42:24.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da175fb100bcac5b-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.286 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-17T19:42:24.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44 2023-10-17T19:42:24.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.290 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.290 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.290 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.291 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.291 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44 2023-10-17T19:42:24.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.292 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk= 2023-10-17T19:42:24.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk=', 'Content-Length': '0'} 2023-10-17T19:42:24.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb4b73143fd03a2-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj44 2023-10-17T19:42:24.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj44 2023-10-17T19:42:24.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj44 2023-10-17T19:42:24.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.296 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.297 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.297 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.297 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.297 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj44 2023-10-17T19:42:24.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.298 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk= 2023-10-17T19:42:24.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts92mAaiTkXdDEB4lIXS7zIxjrk=', 'Content-Length': '0'} 2023-10-17T19:42:24.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22e7b99cab35311-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.303 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.303 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.304 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.304 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.304 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.305 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6e42e0f81a41f7-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj44 2023-10-17T19:42:24.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.310 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.311 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.311 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.311 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.311 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.312 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec0ca246b2e809d-00652eddf8-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj441000falseobj442023-10-17T19:17:50.672Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:24.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.318 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.318 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.318 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.319 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.319 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.319 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:42:24.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:42:24.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001036bbd031ee0c67-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.325 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.325 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.325 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.325 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.326 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.327 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:42:24.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:42:24.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9353e1c484b06e-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.331 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.332 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.332 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.332 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.333 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.333 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU= 2023-10-17T19:42:24.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU=', 'Content-Length': '0'} 2023-10-17T19:42:24.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056f3badd5b14bc50-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.339 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.339 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.340 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.340 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.341 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU= 2023-10-17T19:42:24.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:vDQHv943QeG7lw2HNN+xB0M5RcU=', 'Content-Length': '0'} 2023-10-17T19:42:24.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4e555838520a5a-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.342 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:24.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.348 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.349 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.349 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.349 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.350 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.351 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:42:24.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:42:24.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e18ff04ddb9968e-00652eddf8-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.356 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.356 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.356 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.357 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.357 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55?versionId=null 2023-10-17T19:42:24.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.358 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k= 2023-10-17T19:42:24.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Pvl8uvUk8W1iV3OqGgTGPz6199k=', 'Content-Length': '0'} 2023-10-17T19:42:24.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b6418d943b16d1-00652eddf8-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.363 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.363 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.363 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.364 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.364 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.365 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331bd9d58685efef-00652eddf8-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.371 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.371 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.371 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.371 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:16 GMT 2023-10-17T19:42:24.372 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.372 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4= 2023-10-17T19:42:24.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:16 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:giKsiMmspwK57VAv4nzc/H+BGG4=', 'Content-Length': '0'} 2023-10-17T19:42:24.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4af6bf93f055ab-00652eddf9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55 2023-10-17T19:42:24.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.377 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.378 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.378 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.379 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.379 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55 2023-10-17T19:42:24.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.380 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00= 2023-10-17T19:42:24.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00=', 'Content-Length': '0'} 2023-10-17T19:42:24.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d23111fbde76752-00652eddf9-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55 2023-10-17T19:42:24.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.384 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.385 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.385 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.385 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.386 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55 2023-10-17T19:42:24.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.386 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00= 2023-10-17T19:42:24.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6haOv+z/j8cAMeFwqF3Qv3o4M00=', 'Content-Length': '0'} 2023-10-17T19:42:24.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae09af494381c5ce-00652eddf9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-17T19:42:24.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55 2023-10-17T19:42:24.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.391 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.391 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.392 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.392 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.392 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55 2023-10-17T19:42:24.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.393 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw= 2023-10-17T19:42:24.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw=', 'Content-Length': '0'} 2023-10-17T19:42:24.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-64:4c84373e-d451-49bb-a8c9-c37d5e5e2160.5678.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c645ca3d06c7b3e-00652eddf9-1d4d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj55 2023-10-17T19:42:24.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj55 2023-10-17T19:42:24.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj55 2023-10-17T19:42:24.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.397 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.398 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.398 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.398 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.399 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj55 2023-10-17T19:42:24.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.399 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw= 2023-10-17T19:42:24.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ncipH99lrlNPo60EgFxHEz0XZdw=', 'Content-Length': '0'} 2023-10-17T19:42:24.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:17:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4278dbfb0fbb404-00652eddf9-162c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:24.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.405 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.405 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.405 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.406 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.406 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.407 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A= 2023-10-17T19:42:24.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A=', 'Content-Length': '0'} 2023-10-17T19:42:24.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1f20117a6aa330-00652eddf9-1d4d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:24.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/?prefix=obj55 2023-10-17T19:42:24.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.412 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:24.413 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.413 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.413 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:17 GMT 2023-10-17T19:42:24.414 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.414 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A= 2023-10-17T19:42:24.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:17 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:g3Kfgj+nH0BEuldyZTIGkezhB2A=', 'Content-Length': '0'} 2023-10-17T19:42:24.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedfba94a92a05a4-00652eddf9-162c-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:18:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hbmuog-64obj551000falseobj552023-10-17T19:17:50.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-17T19:42:24.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hbmuog-64 zones={a2, a1} 2023-10-17T19:42:24.416 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-10-17T19:42:24.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-10-17T19:42:24.416 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T19:18:17.095+0000 7f47e63fd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 169349) UID: 0 2023-10-17T19:42:24.417 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi049.stdout: INFO: 2023-10-17T19:18:17.095+0000 7f47ea6667c0 -1 shutting down 2023-10-17T19:42:24.417 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:42:24.417 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~6s 2023-10-17T19:42:24.418 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-10-17T19:42:24.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 5 2023-10-17T19:42:24.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.424 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.424 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.425 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.425 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.425 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.426 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef392578e4b5877e-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj17 2023-10-17T19:42:24.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj17 2023-10-17T19:42:24.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj17 2023-10-17T19:42:24.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.430 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.430 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.431 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.431 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.431 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj17 2023-10-17T19:42:24.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.432 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tcZk/uqAQctBvZllr6aJbiCwnzw= 2023-10-17T19:42:24.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tcZk/uqAQctBvZllr6aJbiCwnzw='} 2023-10-17T19:42:24.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab32273926943e44-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.436 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.437 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.437 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.437 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.438 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.438 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da735ff52f77ce07-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj34 2023-10-17T19:42:24.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj34 2023-10-17T19:42:24.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj34 2023-10-17T19:42:24.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.443 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.443 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.443 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.444 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.444 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj34 2023-10-17T19:42:24.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.445 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dAUjsJGEf33SA/BLVbWeWvIxFBs= 2023-10-17T19:42:24.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dAUjsJGEf33SA/BLVbWeWvIxFBs='} 2023-10-17T19:42:24.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074d5714dfeaf6aa1-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.449 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.449 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.450 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.450 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.450 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.451 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4953475cd0ff0a-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj51 2023-10-17T19:42:24.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj51 2023-10-17T19:42:24.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj51 2023-10-17T19:42:24.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.455 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.455 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.456 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.456 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.456 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj51 2023-10-17T19:42:24.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.457 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:BA+gHP4oZ2Cm/hqnRCffRaUf+HA= 2023-10-17T19:42:24.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:BA+gHP4oZ2Cm/hqnRCffRaUf+HA='} 2023-10-17T19:42:24.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096d2bfd271169098-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.461 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.462 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.462 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.462 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.463 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.463 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a1b2bce33471aaf-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj68 2023-10-17T19:42:24.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj68 2023-10-17T19:42:24.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj68 2023-10-17T19:42:24.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.468 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.468 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.468 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.469 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.469 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj68 2023-10-17T19:42:24.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.470 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4://TUWdmMa3DOqKBw2WWbMG4w5kI= 2023-10-17T19:42:24.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4://TUWdmMa3DOqKBw2WWbMG4w5kI='} 2023-10-17T19:42:24.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098e0192b5daaf16d-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.474 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.475 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.475 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.475 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.476 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.476 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b71ae44fb0b02ed-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj85 2023-10-17T19:42:24.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj85 2023-10-17T19:42:24.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj85 2023-10-17T19:42:24.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.481 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.481 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.481 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.482 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.482 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj85 2023-10-17T19:42:24.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.482 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:uaKguJVSi2lOtIAx3cRlUbOe00E= 2023-10-17T19:42:24.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:uaKguJVSi2lOtIAx3cRlUbOe00E='} 2023-10-17T19:42:24.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6ef587d5b7a959c-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.483 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 10 2023-10-17T19:42:24.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.490 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.490 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.490 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.491 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.491 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.492 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001747b5686c8f433c-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj19 2023-10-17T19:42:24.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj19 2023-10-17T19:42:24.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj19 2023-10-17T19:42:24.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.496 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.496 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.497 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.497 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.497 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj19 2023-10-17T19:42:24.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.498 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:F6H1i2Z6COrPSeWoCnhQB1kHniI= 2023-10-17T19:42:24.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F6H1i2Z6COrPSeWoCnhQB1kHniI='} 2023-10-17T19:42:24.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013fb6f1f708b93c0-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.502 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.502 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.503 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.503 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.503 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.504 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48c03ac565d1bbf-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj38 2023-10-17T19:42:24.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj38 2023-10-17T19:42:24.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj38 2023-10-17T19:42:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.508 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.508 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.508 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.509 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.509 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj38 2023-10-17T19:42:24.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.509 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:RGn7C6DjI+6jMZ4Dzlpe32vheVk= 2023-10-17T19:42:24.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:RGn7C6DjI+6jMZ4Dzlpe32vheVk='} 2023-10-17T19:42:24.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000300d50e734404c5b-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.513 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.514 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.514 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.514 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.515 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.515 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b0d60f2aa7cc9d4-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj57 2023-10-17T19:42:24.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj57 2023-10-17T19:42:24.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj57 2023-10-17T19:42:24.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.520 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.520 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.520 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.521 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj57 2023-10-17T19:42:24.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.521 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/JJRtjpJU9cjpXDF7AqiT48/15Q= 2023-10-17T19:42:24.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/JJRtjpJU9cjpXDF7AqiT48/15Q='} 2023-10-17T19:42:24.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f7684c0de32dd5d-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.526 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.526 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.526 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.526 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.527 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.527 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db814fdd1f42e02c-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj76 2023-10-17T19:42:24.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj76 2023-10-17T19:42:24.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj76 2023-10-17T19:42:24.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.532 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.532 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.533 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.533 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj76 2023-10-17T19:42:24.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.534 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+ZIp9xzy60td4YbUCxuLC7WGwk8= 2023-10-17T19:42:24.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+ZIp9xzy60td4YbUCxuLC7WGwk8='} 2023-10-17T19:42:24.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037ada24ef240143e-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.539 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.539 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.539 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.539 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.540 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.540 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE= 2023-10-17T19:42:24.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+7SGIEY20T/PehhtzO4vDUccFLE=', 'Content-Length': '0'} 2023-10-17T19:42:24.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000788cee9af6d80db3-00652eddff-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj95 2023-10-17T19:42:24.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj95 2023-10-17T19:42:24.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj95 2023-10-17T19:42:24.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.545 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.545 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.546 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:23 GMT 2023-10-17T19:42:24.546 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj95 2023-10-17T19:42:24.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.547 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:k0CzdsHpn0H5PDEfWEcUaf/9jig= 2023-10-17T19:42:24.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:23 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:k0CzdsHpn0H5PDEfWEcUaf/9jig='} 2023-10-17T19:42:24.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bf3aee9e6f2fe05-00652eddff-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 15 2023-10-17T19:42:24.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.554 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.554 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.554 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.555 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.555 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.556 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065725bfbd821f97f-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj23 2023-10-17T19:42:24.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj23 2023-10-17T19:42:24.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj23 2023-10-17T19:42:24.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.560 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.561 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.561 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.561 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.562 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj23 2023-10-17T19:42:24.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.562 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:/PzHQM7QtGNgvlUpmtV57Svqwy4= 2023-10-17T19:42:24.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:/PzHQM7QtGNgvlUpmtV57Svqwy4='} 2023-10-17T19:42:24.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fbeda7376f5fac0-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.566 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.567 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.567 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.567 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.568 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.568 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7201acfbd03adb0-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj46 2023-10-17T19:42:24.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj46 2023-10-17T19:42:24.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj46 2023-10-17T19:42:24.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.573 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.573 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.573 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.574 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.574 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj46 2023-10-17T19:42:24.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.574 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:trTcHmdE+wC9CrgWPLWD4X9lijQ= 2023-10-17T19:42:24.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:trTcHmdE+wC9CrgWPLWD4X9lijQ='} 2023-10-17T19:42:24.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000232a9887f65e56ae-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.579 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.579 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.579 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.580 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.580 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.581 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c09658949aecfc8-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj69 2023-10-17T19:42:24.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj69 2023-10-17T19:42:24.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj69 2023-10-17T19:42:24.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.585 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.585 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.586 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.586 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.586 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj69 2023-10-17T19:42:24.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.587 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:3MUselRa1VMCs3nvh3/SntbtOG4= 2023-10-17T19:42:24.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:3MUselRa1VMCs3nvh3/SntbtOG4='} 2023-10-17T19:42:24.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047d35870e95ab3ef-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.592 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.592 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.592 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.593 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.593 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.594 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092633e8119ae75fc-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj92 2023-10-17T19:42:24.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj92 2023-10-17T19:42:24.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj92 2023-10-17T19:42:24.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.598 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.599 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.599 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.599 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.600 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj92 2023-10-17T19:42:24.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.600 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nL97bv00LTOD4/FA5cuLPQd0jo0= 2023-10-17T19:42:24.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nL97bv00LTOD4/FA5cuLPQd0jo0='} 2023-10-17T19:42:24.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd0adc21b3d346b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.605 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.605 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.606 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.606 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.606 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.607 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073ac3ac611de0e20-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj115 2023-10-17T19:42:24.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj115 2023-10-17T19:42:24.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj115 2023-10-17T19:42:24.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.611 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.612 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.612 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.612 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.613 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj115 2023-10-17T19:42:24.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.613 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8D0mFah4IhNX5Y/0n7XbHm8zwns= 2023-10-17T19:42:24.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8D0mFah4IhNX5Y/0n7XbHm8zwns='} 2023-10-17T19:42:24.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b0eabf03439d05d-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 20 2023-10-17T19:42:24.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.620 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.620 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.621 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.621 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.621 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.622 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3f49b219019b18c-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj29 2023-10-17T19:42:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj29 2023-10-17T19:42:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj29 2023-10-17T19:42:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.627 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.627 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.628 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.628 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.628 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj29 2023-10-17T19:42:24.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.629 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:oGc8v5zgpmjokQSyD21ziIsVF+U= 2023-10-17T19:42:24.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:oGc8v5zgpmjokQSyD21ziIsVF+U='} 2023-10-17T19:42:24.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da9a2ef3169b9d9b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.633 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.634 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.634 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.634 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.635 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.635 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d1a906081918a15-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj58 2023-10-17T19:42:24.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj58 2023-10-17T19:42:24.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj58 2023-10-17T19:42:24.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.640 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.641 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.641 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.641 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.642 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj58 2023-10-17T19:42:24.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.642 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nksdvK9gkLVH1fStdNxFYOLJxe8= 2023-10-17T19:42:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nksdvK9gkLVH1fStdNxFYOLJxe8='} 2023-10-17T19:42:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008457d7e534e0ea7c-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.647 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.647 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.647 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.648 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.648 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.649 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da38a72011f1d79-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj87 2023-10-17T19:42:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj87 2023-10-17T19:42:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj87 2023-10-17T19:42:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.653 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.653 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.654 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.654 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.654 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj87 2023-10-17T19:42:24.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.655 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:awzZz2q8JtetkzQvAntvLYYgCIE= 2023-10-17T19:42:24.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:awzZz2q8JtetkzQvAntvLYYgCIE='} 2023-10-17T19:42:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8be247333a1752b-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.659 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.659 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.660 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.660 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.660 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.661 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002796066bb6a78979-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj116 2023-10-17T19:42:24.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj116 2023-10-17T19:42:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj116 2023-10-17T19:42:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.665 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.666 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.666 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.666 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.667 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj116 2023-10-17T19:42:24.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.667 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:qX/9z97wmDQSZv8A1OwIONPViWE= 2023-10-17T19:42:24.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:qX/9z97wmDQSZv8A1OwIONPViWE='} 2023-10-17T19:42:24.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcff96efac21e48d-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.671 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.672 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.672 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.672 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.673 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.673 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82eea0bdf57294b-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj145 2023-10-17T19:42:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj145 2023-10-17T19:42:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj145 2023-10-17T19:42:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.678 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.678 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.679 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.679 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.679 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj145 2023-10-17T19:42:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.680 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:MxeJFVjoQe6vcjKMxIHbqZgZkj8= 2023-10-17T19:42:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:MxeJFVjoQe6vcjKMxIHbqZgZkj8='} 2023-10-17T19:42:24.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d775050ca96c8a40-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 25 2023-10-17T19:42:24.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.687 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.687 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.688 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.688 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.688 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.689 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d971562159d979b6-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj31 2023-10-17T19:42:24.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj31 2023-10-17T19:42:24.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj31 2023-10-17T19:42:24.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.694 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.694 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.694 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.695 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.695 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj31 2023-10-17T19:42:24.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.695 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XB/5MQIo9tf1m3fSD8+HYPoDTGA= 2023-10-17T19:42:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XB/5MQIo9tf1m3fSD8+HYPoDTGA='} 2023-10-17T19:42:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf6823d2be727794-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.700 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.701 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.701 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.702 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004532a25169b3abfc-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj62 2023-10-17T19:42:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj62 2023-10-17T19:42:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj62 2023-10-17T19:42:24.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.706 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.706 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.707 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.707 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.707 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj62 2023-10-17T19:42:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.708 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Hq0ZK/6JmjCopHLT4MJL51TJEd4= 2023-10-17T19:42:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Hq0ZK/6JmjCopHLT4MJL51TJEd4='} 2023-10-17T19:42:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee2f8eda2fdc9f77-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.712 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.712 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.713 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.713 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.713 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.714 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f7f8e1dc0a550f-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj93 2023-10-17T19:42:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj93 2023-10-17T19:42:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj93 2023-10-17T19:42:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.719 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.719 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.719 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.719 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.720 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj93 2023-10-17T19:42:24.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.720 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Awu4vSm7EZxYNByeg7It4z6/64g= 2023-10-17T19:42:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Awu4vSm7EZxYNByeg7It4z6/64g='} 2023-10-17T19:42:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bca5208f078ac88-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.725 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.725 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.726 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.726 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.726 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.727 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bcd53269ae86309-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj124 2023-10-17T19:42:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj124 2023-10-17T19:42:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj124 2023-10-17T19:42:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.732 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.732 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.732 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.733 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.733 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj124 2023-10-17T19:42:24.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.734 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ewLBn3DyQ33Oxztw/u/RRVgPw00= 2023-10-17T19:42:24.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ewLBn3DyQ33Oxztw/u/RRVgPw00='} 2023-10-17T19:42:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073e4f840ab3a8343-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.738 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.739 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.739 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.739 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.740 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.740 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw= 2023-10-17T19:42:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:VpnO4NYPbOVHyxJRFaJZGNA3cBw=', 'Content-Length': '0'} 2023-10-17T19:42:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000939d52acacca8e51-00652ede00-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj155 2023-10-17T19:42:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj155 2023-10-17T19:42:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj155 2023-10-17T19:42:24.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.745 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.746 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.746 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.746 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:24 GMT 2023-10-17T19:42:24.747 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj155 2023-10-17T19:42:24.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.747 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7AOylJ5J+/l2vtdD+DeAelunfvM= 2023-10-17T19:42:24.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:24 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7AOylJ5J+/l2vtdD+DeAelunfvM='} 2023-10-17T19:42:24.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007407a18bf1e115ba-00652ede00-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hbmuog-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hbmuog-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: tenant: 2023-10-17T19:42:24.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: bucket name: hbmuog-64 2023-10-17T19:42:24.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: total entries: 30 2023-10-17T19:42:24.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.754 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.755 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.755 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.755 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.756 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.756 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:42:24.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:42:24.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b51fe35fac9cd57-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj37 2023-10-17T19:42:24.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj37 2023-10-17T19:42:24.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj37 2023-10-17T19:42:24.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.761 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.761 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.762 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.762 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj37 2023-10-17T19:42:24.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.762 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:KOAg+3ai0Hk/xzSgMcE9edG2ELg= 2023-10-17T19:42:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:KOAg+3ai0Hk/xzSgMcE9edG2ELg='} 2023-10-17T19:42:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000691507f2a32717a0-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.766 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.767 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.767 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.767 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.768 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.768 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:42:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:42:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aa6f161e1c8b261-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj74 2023-10-17T19:42:24.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj74 2023-10-17T19:42:24.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj74 2023-10-17T19:42:24.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.773 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.774 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.774 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.774 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.775 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj74 2023-10-17T19:42:24.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.775 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:F29kESgwKlQcLx1P/NwOC/RWclI= 2023-10-17T19:42:24.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:F29kESgwKlQcLx1P/NwOC/RWclI='} 2023-10-17T19:42:24.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddaea423727d10a3-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.780 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.780 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.781 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.781 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.781 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.782 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:42:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:42:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa9499a2799663a3-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj111 2023-10-17T19:42:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj111 2023-10-17T19:42:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj111 2023-10-17T19:42:24.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.786 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.786 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.787 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.787 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.787 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj111 2023-10-17T19:42:24.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.788 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XNEm7hKq6AdsqYnCbxnze0PIsgs= 2023-10-17T19:42:24.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XNEm7hKq6AdsqYnCbxnze0PIsgs='} 2023-10-17T19:42:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fb95e6d06e8bcf0-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.792 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.792 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.792 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.793 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.793 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.793 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:42:24.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:42:24.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad0af2c09f683df-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj148 2023-10-17T19:42:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj148 2023-10-17T19:42:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj148 2023-10-17T19:42:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.798 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.798 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.799 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj148 2023-10-17T19:42:24.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.799 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:JgMp2F0bWUm/SFH0qJjhJ6J1peI= 2023-10-17T19:42:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:JgMp2F0bWUm/SFH0qJjhJ6J1peI='} 2023-10-17T19:42:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000369992c85e85183-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/ 2023-10-17T19:42:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/ 2023-10-17T19:42:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/ 2023-10-17T19:42:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.804 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:24.804 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.804 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.804 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.805 INFO:tasks.rgw_multisite_tests:/hbmuog-64/ 2023-10-17T19:42:24.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.805 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM= 2023-10-17T19:42:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sUEbL6m9rjB0ISfS4ziZ4E92VlM=', 'Content-Length': '0'} 2023-10-17T19:42:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c722424017d1433f-00652ede01-162c-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-64/obj185 2023-10-17T19:42:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-64/obj185 2023-10-17T19:42:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-64/obj185 2023-10-17T19:42:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-17T19:42:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.810 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.811 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-17T19:42:24.811 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:24.811 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:18:25 GMT 2023-10-17T19:42:24.812 INFO:tasks.rgw_multisite_tests:/hbmuog-64/obj185 2023-10-17T19:42:24.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.812 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bWHJLL9BKITdRzMtOhn1+Z7iljM= 2023-10-17T19:42:24.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:18:25 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bWHJLL9BKITdRzMtOhn1+Z7iljM='} 2023-10-17T19:42:24.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5f3f4d80ed2a9f9-00652ede01-162c-a1'), ('Date', 'Tue, 17 Oct 2023 19:18:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:25.344+0000 7f8abe7a8780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-17T19:42:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:25.345+0000 7f8abe7a8780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-17T19:42:24.815 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-10-17T19:42:24.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-17T19:42:24.816 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-10-17T19:42:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:42:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-17T19:42:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-10-17T19:42:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi049.front.sepia.ceph.com port 8001: Connection refused 2023-10-17T19:42:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > curl http://smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-17T19:42:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-17T19:42:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-17T19:42:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: anonymous 2023-10-17T19:42:24.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hbmuog-64 2023-10-17T19:42:24.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:24.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hbmuog-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-17T19:42:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hbmuog-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-17T19:42:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:26.557+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.822 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-10-17T19:42:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:31.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:36.560+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:41.558+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:46.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:51.559+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:18:56.560+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:01.561+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:06.562+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:11.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:16.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:21.563+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:26.564+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:31.564+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:36.565+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:41.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:46.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:51.568+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:19:56.567+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 0.. 2023-10-17T19:42:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:01.568+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:06.569+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:11.570+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:16.571+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:21.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:26.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:31.572+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:36.573+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:41.573+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:46.574+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:51.575+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:20:56.576+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:01.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:06.576+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:11.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:16.577+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:21.578+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:26.579+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:31.580+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:36.581+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:41.580+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:46.581+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:51.582+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:21:56.582+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:01.583+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:06.584+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:11.585+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:16.586+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:21.585+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:26.586+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:31.587+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:36.587+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:41.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:46.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:51.590+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:22:56.589+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:01.590+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:06.591+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:11.591+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:16.592+0000 7f79c97e6780 1 waiting to reach latest gen 6, on 1.. 2023-10-17T19:42:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:21.593+0000 7f79c97e6780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-17T19:42:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:21.593+0000 7f79c97e6780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:23:21.593+0000 7f79c97e6780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-17T19:42:24.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-17T19:42:24.843 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:24.844 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.844 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:24.844 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2023-10-17T19:42:24.845 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:24.846 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:24.846 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:24.846 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:24.847 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2548, in test_sync_different_buckets 2023-10-17T19:42:24.847 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2023-10-17T19:42:24.847 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-10-17T19:42:24.848 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-10-17T19:42:24.848 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-17T19:42:24.848 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-17T19:42:24.848 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-17T19:42:24.849 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:24.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571089.530063_1066.1", 2023-10-17T19:42:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:29.530063Z" 2023-10-17T19:42:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:24.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:24.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:24.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:24.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:29.834+0000 7fe41df9b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:29.835+0000 7fe41df9b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697570965.601721_1052.1", 2023-10-17T19:42:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.601721Z", 2023-10-17T19:42:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:24.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:29Z" 2023-10-17T19:42:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:24.880 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:24.880 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:24.881 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:24.881 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:24.881 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.882 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.882 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:24.883 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.883 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:24.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1697570965.601721_1052.1", 2023-10-17T19:42:24.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.885 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:24.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.601721Z", 2023-10-17T19:42:24.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.886 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.886 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.887 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.887 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:24.887 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:24.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.890 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.890 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.890 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.890 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:24.891 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.891 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:24.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.892 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:24.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.893 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.893 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.894 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.894 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:24.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:24.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.897 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.897 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.897 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:24.897 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:24.898 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:24.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:24.899 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.899 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:29Z" 2023-10-17T19:42:24.899 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:24.899 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697570965.601721_1052.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.601721Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:24.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697570965.601721_1052.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571089.530063_1066.1 target=1_1697570965.601721_1052.1 2023-10-17T19:42:24.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:34.966+0000 7f4f2324f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:34.967+0000 7f4f2324f780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:24.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:24.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:24.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:24.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:24.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571089.530063_1066.1", 2023-10-17T19:42:24.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:24.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:29.530063Z", 2023-10-17T19:42:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:24.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:24.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:24.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:24.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:34Z" 2023-10-17T19:42:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:24.924 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:24.924 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:24.924 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:24.924 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:24.925 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.925 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.926 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:24.926 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.926 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:24.927 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571089.530063_1066.1", 2023-10-17T19:42:24.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.928 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:24.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:29.530063Z", 2023-10-17T19:42:24.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.929 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.930 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.930 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.930 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:24.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.932 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.933 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:24.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.933 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.934 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.934 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.934 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:24.935 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.936 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:24.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.937 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:24.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.938 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.938 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.938 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:24.939 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:24.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:24.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:24.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.941 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.942 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:24.942 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:24.942 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.943 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:24.943 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:24.943 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:24.944 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.944 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:34Z" 2023-10-17T19:42:24.944 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:24.945 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571089.530063_1066.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:29.530063Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:24.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571089.530063_1066.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:24.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[]} 2023-10-17T19:42:24.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hbmuog-70 2023-10-17T19:42:24.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/ 2023-10-17T19:42:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/ 2023-10-17T19:42:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/ 2023-10-17T19:42:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:24.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.952 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.953 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.953 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.953 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:35 GMT 2023-10-17T19:42:24.953 INFO:tasks.rgw_multisite_tests:/hbmuog-70/ 2023-10-17T19:42:24.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.954 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:aWKS2MzDbTSzS8Xyx8sHlgV4LIk= 2023-10-17T19:42:24.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:aWKS2MzDbTSzS8Xyx8sHlgV4LIk=', 'Content-Length': '0'} 2023-10-17T19:42:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000095e7bf9a6c0bd777-00652ee117-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hbmuog-71 2023-10-17T19:42:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-71/ 2023-10-17T19:42:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-71/ 2023-10-17T19:42:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-71/ 2023-10-17T19:42:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:24.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:24.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:24.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:24.958 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:24.959 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.959 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:24.959 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:35 GMT 2023-10-17T19:42:24.960 INFO:tasks.rgw_multisite_tests:/hbmuog-71/ 2023-10-17T19:42:24.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:24.960 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PgMxva4TIYONlT9cjAlq0tP3rsA= 2023-10-17T19:42:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:35 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PgMxva4TIYONlT9cjAlq0tP3rsA=', 'Content-Length': '0'} 2023-10-17T19:42:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000021b275ed8fef4503-00652ee117-22c3-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:24.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:42:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:24.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:24.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:24.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:24.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:24.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:24.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:24.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:24.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:24.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:24.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:35.344+0000 7f98e631c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:35.345+0000 7f98e631c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:24.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:24.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:24.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:24.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:24.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:24.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:24.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:24.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:24.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:35Z" 2023-10-17T19:42:24.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:24.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:24.994 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:24.995 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:24.995 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:24.995 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:24.996 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:24.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:24.996 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:24.997 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:24.997 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:24.997 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:24.998 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:24.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:24.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:24.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:24.999 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:24.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:25.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.000 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.001 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.001 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.001 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.001 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:25.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:25.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.004 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.004 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.004 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.004 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.005 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.005 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:25.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.006 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.007 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:25.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.007 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.007 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.008 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.008 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.008 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:25.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:25.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.011 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.011 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.011 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.011 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.012 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.012 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:25.012 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:25.013 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.013 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:35Z" 2023-10-17T19:42:25.013 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.013 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:25.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:25.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:25.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:25.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:42:25.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:42:25.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-70', '--dest-bucket=hbmuog-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-70', '--dest-bucket=hbmuog-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-70 --dest-bucket=hbmuog-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:25.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:25.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:25.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 18, 2023-10-17T19:42:25.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:42:25.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": [], 2023-10-17T19:42:25.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_map": { 2023-10-17T19:42:25.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:25.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zonegroups": [ 2023-10-17T19:42:25.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "a", 2023-10-17T19:42:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a", 2023-10-17T19:42:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": true, 2023-10-17T19:42:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a1", 2023-10-17T19:42:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:25.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:25.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:25.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:25.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:25.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:25.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:25.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:25.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:25.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:25.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:25.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a2", 2023-10-17T19:42:25.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:42:25.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [ 2023-10-17T19:42:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-group", 2023-10-17T19:42:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "data_flow": { 2023-10-17T19:42:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "symmetrical": [ 2023-10-17T19:42:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-flow1", 2023-10-17T19:42:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pipes": [ 2023-10-17T19:42:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-pipe", 2023-10-17T19:42:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-70", 2023-10-17T19:42:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:42:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": { 2023-10-17T19:42:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-71", 2023-10-17T19:42:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "params": { 2023-10-17T19:42:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "filter": { 2023-10-17T19:42:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [] 2023-10-17T19:42:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": {}, 2023-10-17T19:42:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "priority": 0, 2023-10-17T19:42:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "mode": "system", 2023-10-17T19:42:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user": "" 2023-10-17T19:42:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "enabled" 2023-10-17T19:42:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "b", 2023-10-17T19:42:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b", 2023-10-17T19:42:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": false, 2023-10-17T19:42:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b2", 2023-10-17T19:42:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:42:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b1", 2023-10-17T19:42:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [] 2023-10-17T19:42:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "short_zone_ids": [ 2023-10-17T19:42:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3843991460 2023-10-17T19:42:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1846350003 2023-10-17T19:42:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3451171648 2023-10-17T19:42:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1989104912 2023-10-17T19:42:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zonegroup": "a", 2023-10-17T19:42:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_config": { 2023-10-17T19:42:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_quota": { 2023-10-17T19:42:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_quota": { 2023-10-17T19:42:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_ratelimit": { 2023-10-17T19:42:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_ratelimit": { 2023-10-17T19:42:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "anonymous_ratelimit": { 2023-10-17T19:42:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_name": "test-realm", 2023-10-17T19:42:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:25.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/ 2023-10-17T19:42:25.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/ 2023-10-17T19:42:25.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/ 2023-10-17T19:42:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:25.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:25.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:25.108 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:25.108 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.109 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.109 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:36 GMT 2023-10-17T19:42:25.109 INFO:tasks.rgw_multisite_tests:/hbmuog-70/ 2023-10-17T19:42:25.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:25.110 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA= 2023-10-17T19:42:25.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA=', 'Content-Length': '0'} 2023-10-17T19:42:25.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a8e4616bafec702-00652ee118-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:25.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/obj1 2023-10-17T19:42:25.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/obj1 2023-10-17T19:42:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/obj1 2023-10-17T19:42:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:25.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:25.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:25.114 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:25.115 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:25.115 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:25.115 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:36 GMT 2023-10-17T19:42:25.115 INFO:tasks.rgw_multisite_tests:/hbmuog-70/obj1 2023-10-17T19:42:25.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:25.116 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:w7e3hTc98W+YT2PD32+zm4hE5Wc= 2023-10-17T19:42:25.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w7e3hTc98W+YT2PD32+zm4hE5Wc='} 2023-10-17T19:42:25.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ccfa923b795c69d-00652ee118-244a-a1'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:25.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/ 2023-10-17T19:42:25.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/ 2023-10-17T19:42:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/ 2023-10-17T19:42:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:25.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:25.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:25.120 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:25.121 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.121 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.121 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:36 GMT 2023-10-17T19:42:25.122 INFO:tasks.rgw_multisite_tests:/hbmuog-70/ 2023-10-17T19:42:25.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:25.122 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA= 2023-10-17T19:42:25.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TXX0nWYFFc9MvGTQ6q9Nt1HgzhA=', 'Content-Length': '0'} 2023-10-17T19:42:25.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbe42ec713e86a16-00652ee118-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:25.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/obj2 2023-10-17T19:42:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/obj2 2023-10-17T19:42:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/obj2 2023-10-17T19:42:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:25.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:25.127 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:25.127 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:25.128 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:25.128 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:31:36 GMT 2023-10-17T19:42:25.128 INFO:tasks.rgw_multisite_tests:/hbmuog-70/obj2 2023-10-17T19:42:25.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:25.129 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Lh9tcKEZHehNqpmxnxth4FFgQO0= 2023-10-17T19:42:25.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:31:36 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Lh9tcKEZHehNqpmxnxth4FFgQO0='} 2023-10-17T19:42:25.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f2eaa6d64b5baf9-00652ee118-244a-a1'), ('Date', 'Tue, 17 Oct 2023 19:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:25.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:42:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:25.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:25.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:36.635+0000 7fb4954fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:31:36.636+0000 7fb4954fa780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:42:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:25.161 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.161 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.162 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.162 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.162 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:25.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.163 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.163 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.163 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.164 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.165 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:25.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.165 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:25.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:25.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.167 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.167 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.167 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.168 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.168 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.169 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:25.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.169 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:25.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.170 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.171 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.171 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.171 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.172 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.172 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:25.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.173 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:25.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.174 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.174 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.175 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.175 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.175 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.176 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:25.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:25.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.177 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.177 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.177 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:25.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:25.178 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.178 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.178 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.179 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.179 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.179 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:25.179 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:25.180 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.180 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:42:25.180 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.181 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:25.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:25.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:25.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:25.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:25.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.191 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:42:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.211 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.211 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.211 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.212 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.212 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.212 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.213 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.213 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.213 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.214 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.214 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.215 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.216 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.216 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.217 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.217 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.219 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.219 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.219 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.220 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.220 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.220 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.221 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.221 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.222 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.223 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.223 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.224 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.224 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.224 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.225 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.226 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.226 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.227 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.227 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.227 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.228 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.228 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.228 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.228 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:36Z" 2023-10-17T19:42:25.229 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.229 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:42Z" 2023-10-17T19:42:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.251 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.251 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.251 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.252 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.252 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.252 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.253 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.253 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.253 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.254 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.254 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.254 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.256 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.256 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.256 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.257 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.257 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.258 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.258 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.259 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.260 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.260 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.261 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.261 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.261 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.261 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.262 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.263 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.263 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.264 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.264 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.265 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.265 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.266 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.267 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.267 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.267 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.267 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.268 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.268 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.268 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.269 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.269 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:42Z" 2023-10-17T19:42:25.269 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.270 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:47Z" 2023-10-17T19:42:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.293 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.293 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.294 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.294 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.294 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.295 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.295 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.295 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.296 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.296 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.296 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.298 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.298 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.299 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.299 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.299 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.300 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.300 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.301 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.302 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.302 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.302 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.303 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.303 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.303 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.304 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.305 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.305 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.305 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.306 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.306 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.306 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.306 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.307 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.308 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.308 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.309 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.309 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.309 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.309 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.310 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.310 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.310 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:47Z" 2023-10-17T19:42:25.311 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.311 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:52Z" 2023-10-17T19:42:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.333 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.333 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.334 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.334 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.334 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.335 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.335 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.335 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.335 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.336 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.336 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.336 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.337 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.338 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.338 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.338 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.340 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.341 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.341 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.341 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.342 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.342 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.342 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.343 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.344 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.344 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.344 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.345 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.345 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.345 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.346 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.346 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.347 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.348 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.348 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.348 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.349 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.349 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.349 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.350 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.350 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:52Z" 2023-10-17T19:42:25.350 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.351 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:31:57Z" 2023-10-17T19:42:25.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.372 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.373 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.373 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.373 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.374 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.374 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.374 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.375 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.375 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.375 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.376 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.376 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.378 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.378 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.378 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.379 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.379 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.379 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.381 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.381 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.382 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.382 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.382 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.383 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.383 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.384 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.385 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.385 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.385 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.386 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.386 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.387 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.387 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.387 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.388 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.389 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.389 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.389 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.389 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.390 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.390 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.390 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.391 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:31:57Z" 2023-10-17T19:42:25.391 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.391 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:02Z" 2023-10-17T19:42:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.413 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.413 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.413 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.414 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.414 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.414 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.415 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.415 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.415 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.415 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.416 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.416 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.417 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.418 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.418 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.418 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.419 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.419 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.421 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.421 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.421 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.422 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.422 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.422 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.423 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.423 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.424 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.424 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.425 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.425 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.425 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.426 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.426 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.426 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.427 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.428 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.428 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.428 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.429 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.429 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.429 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.430 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.430 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:02Z" 2023-10-17T19:42:25.430 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.431 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:07Z" 2023-10-17T19:42:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.453 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.453 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.453 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.454 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.454 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.454 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.455 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.455 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.455 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.455 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.456 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.456 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.457 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.458 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.458 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.458 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.461 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.461 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.461 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.461 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.462 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.463 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.464 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.464 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.465 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.465 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.465 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.465 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.466 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.467 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.468 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.468 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.468 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.468 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.469 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.469 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.469 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.470 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.470 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:07Z" 2023-10-17T19:42:25.470 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.471 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:12Z" 2023-10-17T19:42:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.496 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.496 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.497 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.497 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.498 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.498 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.498 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.499 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.499 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.502 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.502 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.502 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.503 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.503 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.503 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.504 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.505 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.506 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.506 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.506 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.506 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.507 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.507 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.508 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.509 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.509 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.509 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.510 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.510 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.511 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.511 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.512 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.512 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.513 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.513 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.513 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.514 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.514 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.514 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.515 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:12Z" 2023-10-17T19:42:25.515 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.515 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:17Z" 2023-10-17T19:42:25.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.537 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.538 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.538 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.538 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.539 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.539 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.539 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.540 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.540 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.540 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.541 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.541 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.543 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.543 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.543 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.544 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.544 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.545 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.546 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.546 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.547 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.547 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.547 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.548 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.548 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.548 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.549 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.550 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.550 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.550 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.551 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.551 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.551 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.552 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.553 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.553 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.553 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.554 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.554 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.554 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.554 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.555 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.555 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:17Z" 2023-10-17T19:42:25.555 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:23Z" 2023-10-17T19:42:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.578 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.579 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.579 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.579 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.579 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.580 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.580 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.580 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.581 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.581 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.582 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.583 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.583 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.584 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.584 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.584 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.585 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.585 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.587 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.587 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.587 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.588 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.588 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.588 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.589 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.589 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.590 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.591 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.591 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.592 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.592 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.592 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.593 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.593 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.594 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.595 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.595 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.595 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.596 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.596 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.596 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.596 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.597 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:23Z" 2023-10-17T19:42:25.597 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.597 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:28Z" 2023-10-17T19:42:25.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.620 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.621 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.621 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.621 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.622 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.622 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.622 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.623 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.624 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.624 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.625 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.625 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.625 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.626 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.626 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.627 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.628 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.628 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.628 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.629 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.629 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.630 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.631 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.631 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.631 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.631 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.632 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.632 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.632 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.633 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.634 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.634 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.635 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.635 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.635 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.636 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.636 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.636 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.637 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:28Z" 2023-10-17T19:42:25.637 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.637 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:33Z" 2023-10-17T19:42:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.660 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.660 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.660 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.661 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.661 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.661 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.662 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.662 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.662 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.663 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.663 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.663 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.665 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.665 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.665 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.666 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.667 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.668 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.669 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.669 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.669 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.669 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.670 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.670 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.671 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.672 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.672 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.672 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.673 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.673 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.674 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.674 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.675 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.676 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.676 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.676 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.677 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.677 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.677 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.677 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.678 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:33Z" 2023-10-17T19:42:25.678 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.678 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:38Z" 2023-10-17T19:42:25.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.701 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.701 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.702 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.702 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.702 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.703 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.703 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.703 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.704 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.704 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.705 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.706 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.706 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.707 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.707 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.708 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.708 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.709 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.710 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.710 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.710 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.711 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.711 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.712 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.713 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.713 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.713 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.714 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.714 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.714 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.714 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.715 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.716 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.716 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.717 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.717 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.717 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.717 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.718 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.718 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.718 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:38Z" 2023-10-17T19:42:25.719 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.719 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:43Z" 2023-10-17T19:42:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.741 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.741 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.742 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.742 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.742 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.743 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.743 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.743 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.744 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.744 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.746 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.746 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.746 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.747 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.747 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.747 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.749 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.749 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.749 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.751 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.752 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.753 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.753 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.753 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.754 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.754 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.754 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.755 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.756 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.756 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.756 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.757 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.757 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.757 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.758 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.758 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.758 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:43Z" 2023-10-17T19:42:25.759 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.759 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:48Z" 2023-10-17T19:42:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.781 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.782 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.782 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.782 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.783 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.783 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.784 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.784 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.785 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.787 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.787 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.787 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.788 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.788 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.789 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.791 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.791 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.791 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.792 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.793 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.794 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.795 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.795 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.796 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.796 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.797 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.798 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.798 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.799 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.799 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.799 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.800 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.800 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.800 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.801 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:48Z" 2023-10-17T19:42:25.801 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.801 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:53Z" 2023-10-17T19:42:25.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.824 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.824 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.824 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.825 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.825 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.825 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.826 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.826 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.826 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.827 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.827 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.827 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.829 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.830 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.830 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.830 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.831 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.833 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.833 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.833 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.834 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.835 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.835 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.836 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.837 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.837 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.837 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.837 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.838 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.838 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.839 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.840 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.840 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.840 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.840 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.841 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.841 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.841 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.842 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.842 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:53Z" 2023-10-17T19:42:25.842 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.843 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:32:59Z" 2023-10-17T19:42:25.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.865 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.865 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.866 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.866 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.866 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.867 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.867 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.867 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.868 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.868 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.868 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.869 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.870 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.871 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.871 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.871 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.871 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.872 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.872 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.874 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.874 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.874 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.875 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.875 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.876 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.876 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.877 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.878 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.878 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.878 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.879 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.879 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.879 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.880 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.881 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.881 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.881 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.882 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.882 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.882 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.883 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.883 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.883 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:32:59Z" 2023-10-17T19:42:25.884 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.884 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:04Z" 2023-10-17T19:42:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.907 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.907 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.908 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.908 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.908 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.909 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.909 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.909 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.912 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.912 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.912 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.912 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.913 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.913 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.915 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.915 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.916 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.916 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.916 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.917 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.917 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.918 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.919 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.919 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.919 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.920 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.921 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.921 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.922 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.922 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.923 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.923 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.923 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.924 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.924 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.924 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.924 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.925 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:04Z" 2023-10-17T19:42:25.925 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.925 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:09Z" 2023-10-17T19:42:25.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.947 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.948 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.948 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.948 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.949 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.949 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.950 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.950 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.951 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.952 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.953 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.953 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.953 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.954 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.954 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.956 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.956 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.956 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:25.957 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.957 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.957 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.958 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:25.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.959 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.959 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.960 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.960 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:25.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.961 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.961 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.961 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:25.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.962 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.963 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.963 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:25.963 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.964 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:25.964 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:25.964 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:25.964 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.965 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:09Z" 2023-10-17T19:42:25.965 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:25.965 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:25.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:25.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:25.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:25.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:25.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:25.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:14Z" 2023-10-17T19:42:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:25.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:25.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:25.990 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:25.990 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:25.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:25.991 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:25.992 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.992 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:25.992 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.993 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:25.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.994 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.994 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:25.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:25.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:25.997 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:25.997 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:25.997 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:25.997 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:25.998 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:25.998 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:25.998 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:25.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:25.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:25.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.000 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.001 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.001 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.001 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.002 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.002 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.003 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.003 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.004 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.004 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.005 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.005 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.005 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.006 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.007 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.007 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.007 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.008 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.008 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.008 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.008 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.009 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.009 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.009 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:14Z" 2023-10-17T19:42:26.010 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.010 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:26.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:26.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:26.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:19Z" 2023-10-17T19:42:26.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.033 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.033 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.033 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.034 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.034 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.034 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.035 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.035 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.035 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.036 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.036 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.038 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.038 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.038 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.039 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.039 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.040 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.041 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.041 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.042 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.042 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.042 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.043 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.043 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.044 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.045 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.045 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.045 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.046 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.046 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.046 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.047 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.047 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.048 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.049 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.049 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.049 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.050 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.050 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.050 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.051 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.051 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:19Z" 2023-10-17T19:42:26.051 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.052 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:26.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:26.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:26.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:24Z" 2023-10-17T19:42:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.074 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.074 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.074 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.075 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.075 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.075 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.075 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.076 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.076 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.076 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.077 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.077 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.078 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.078 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.078 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.079 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.079 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.080 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.080 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.080 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.082 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.082 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.082 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.083 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.083 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.083 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.084 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.084 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.085 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.085 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.086 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.086 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.087 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.087 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.087 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.088 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.089 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.089 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.089 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.090 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.090 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.090 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.091 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.091 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:24Z" 2023-10-17T19:42:26.091 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.092 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:26.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:26.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:26.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.094 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:29Z" 2023-10-17T19:42:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.114 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.115 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.116 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.116 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.117 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.118 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.119 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.120 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.120 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.120 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.121 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.121 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.121 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.123 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.123 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.124 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.124 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.124 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.125 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.125 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.126 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.126 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.127 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.127 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.128 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.128 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.128 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.129 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.130 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.130 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.130 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.131 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.131 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.131 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.132 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.132 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:29Z" 2023-10-17T19:42:26.132 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.133 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:26.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:26.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:26.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:34Z" 2023-10-17T19:42:26.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.155 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.155 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.156 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.156 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.156 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.156 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.157 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.157 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.158 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.158 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.160 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.160 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.160 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.161 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040842", 2023-10-17T19:42:26.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.163 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.291154Z" 2023-10-17T19:42:26.163 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.164 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.164 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.164 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.165 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.165 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.165 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.166 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.167 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.167 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.168 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.168 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.168 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.169 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.169 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046716", 2023-10-17T19:42:26.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.169 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:29:25.758413Z" 2023-10-17T19:42:26.171 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.171 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.171 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.171 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.172 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.172 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.173 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.173 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:34Z" 2023-10-17T19:42:26.173 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.174 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.291154Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046716', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:29:25.758413Z'}}] 2023-10-17T19:42:26.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040842', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046716'} 2023-10-17T19:42:26.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000040982 target=00000000000000000000:00000000000000040842, shard=3 master=00000000000000000000:00000000000000046856 target=00000000000000000000:00000000000000046716 2023-10-17T19:42:26.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.196 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.196 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.196 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.197 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.197 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.197 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.198 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.198 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.198 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.199 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.199 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.200 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.201 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.201 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.202 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.202 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.202 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.203 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.203 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.204 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.205 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.205 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.205 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.206 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.206 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.206 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.207 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.207 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.208 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.208 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.208 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.209 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.209 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.209 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.210 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.210 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.211 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.212 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.212 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.212 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.212 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.213 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.213 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.213 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.213 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.214 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.214 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.214 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.216 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:26.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/ 2023-10-17T19:42:26.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/ 2023-10-17T19:42:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/ 2023-10-17T19:42:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.220 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.220 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.221 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.221 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.221 INFO:tasks.rgw_multisite_tests:/hbmuog-70/ 2023-10-17T19:42:26.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.222 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:NaUmoEY3n75Er7cJBil4YFezxMU= 2023-10-17T19:42:26.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:NaUmoEY3n75Er7cJBil4YFezxMU=', 'Content-Length': '0'} 2023-10-17T19:42:26.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000281b88e0af011ed4-00652ee194-2444-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/obj1 2023-10-17T19:42:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/obj1 2023-10-17T19:42:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/obj1 2023-10-17T19:42:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.226 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.226 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.227 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.227 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.227 INFO:tasks.rgw_multisite_tests:/hbmuog-70/obj1 2023-10-17T19:42:26.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.228 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:pFM25cdRtoDz5hEQDVkUw0CIwbU= 2023-10-17T19:42:26.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pFM25cdRtoDz5hEQDVkUw0CIwbU=', 'Content-Length': '0'} 2023-10-17T19:42:26.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006d75e4c8782873fa-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-70/obj2 2023-10-17T19:42:26.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-70/obj2 2023-10-17T19:42:26.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-70/obj2 2023-10-17T19:42:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.232 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.232 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.233 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.233 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.233 INFO:tasks.rgw_multisite_tests:/hbmuog-70/obj2 2023-10-17T19:42:26.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.234 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ltV+3VEjEOHGxE4WyKEQObzsbLs= 2023-10-17T19:42:26.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ltV+3VEjEOHGxE4WyKEQObzsbLs=', 'Content-Length': '0'} 2023-10-17T19:42:26.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001c6ce76ef1e911e4-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-71/ 2023-10-17T19:42:26.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-71/ 2023-10-17T19:42:26.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-71/ 2023-10-17T19:42:26.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.238 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.239 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.239 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.239 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.240 INFO:tasks.rgw_multisite_tests:/hbmuog-71/ 2023-10-17T19:42:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.240 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ts7HkmSqkGypTWU0I/bnzs6h3To= 2023-10-17T19:42:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ts7HkmSqkGypTWU0I/bnzs6h3To=', 'Content-Length': '0'} 2023-10-17T19:42:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3430794fe4424b9-00652ee194-2444-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-71/obj1 2023-10-17T19:42:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-71/obj1 2023-10-17T19:42:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-71/obj1 2023-10-17T19:42:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.245 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.245 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.245 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.245 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.246 INFO:tasks.rgw_multisite_tests:/hbmuog-71/obj1 2023-10-17T19:42:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.246 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AbxSEp/FtzkFK6nAaqlV39c1hJY= 2023-10-17T19:42:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AbxSEp/FtzkFK6nAaqlV39c1hJY=', 'Content-Length': '0'} 2023-10-17T19:42:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fac36d797d86a4f9-00652ee194-2444-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.247 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:26.248 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.248 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-17T19:42:26.249 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-10-17T19:42:26.249 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:26.251 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-17T19:42:26.251 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-17T19:42:26.251 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-17T19:42:26.252 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2745, in test_sync_multiple_buckets_to_single 2023-10-17T19:42:26.252 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesD, content) 2023-10-17T19:42:26.252 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-10-17T19:42:26.253 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-10-17T19:42:26.253 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-17T19:42:26.253 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-17T19:42:26.254 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-17T19:42:26.254 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-17T19:42:26.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:26.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:26.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:31:35.147990Z" 2023-10-17T19:42:26.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:26.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:26.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:40.339+0000 7fb6530f8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:40.340+0000 7fb6530f8780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:26.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:26.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:26.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:26.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:26.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:26.285 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.286 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.286 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.286 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.287 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.288 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.288 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.288 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.289 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.289 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571095.147990_1074.1", 2023-10-17T19:42:26.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.290 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:26.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:35.147990Z", 2023-10-17T19:42:26.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.291 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.291 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.292 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.293 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.295 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.295 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.295 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.296 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.296 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.297 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:26.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.298 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:26.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.298 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.299 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.299 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.299 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.300 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.302 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.302 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.303 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.303 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.303 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.304 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:26.304 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:26.304 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.305 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.305 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.305 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571095.147990_1074.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:31:35.147990Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:26.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571095.147990_1074.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:26.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.307 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:26.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hbmuog-72 2023-10-17T19:42:26.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:26.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.313 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.313 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.314 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.314 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.314 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:26.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.315 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zjYPRiv3a1qq9D495ekAPt4ok2E= 2023-10-17T19:42:26.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zjYPRiv3a1qq9D495ekAPt4ok2E=', 'Content-Length': '0'} 2023-10-17T19:42:26.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005b7937fd78b24c33-00652ee194-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hbmuog-73 2023-10-17T19:42:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.319 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.320 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.320 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.320 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:40 GMT 2023-10-17T19:42:26.321 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:26.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.321 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:4l/29SmfyYiK4Y2Um93KqYXIjtY= 2023-10-17T19:42:26.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:40 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:4l/29SmfyYiK4Y2Um93KqYXIjtY=', 'Content-Length': '0'} 2023-10-17T19:42:26.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a5f45ef0853b3a6f-00652ee194-244a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:26.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:33:40.511436Z" 2023-10-17T19:42:26.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:26.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:26.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:26.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:26.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.332 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:40.709+0000 7f0d6358a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:40.710+0000 7f0d6358a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:26.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:26.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.356 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.356 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.357 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.357 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.358 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.358 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.358 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.359 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.359 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.360 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:26.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:26.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.361 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.362 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.362 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.362 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.362 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.365 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.365 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.365 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.366 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.367 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:26.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.368 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:26.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.368 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.369 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.369 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.370 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.371 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.372 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.372 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.372 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.373 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.373 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:26.374 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:26.374 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.374 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:40Z" 2023-10-17T19:42:26.375 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:26.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:26.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:26.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-72', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-72', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-72 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-72 --dest-bucket=hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:26.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-73', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hbmuog-73', '--dest-bucket=hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hbmuog-73 --source-zones=a1 --dest-zones=a2 --source-bucket=hbmuog-73 --dest-bucket=hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:26.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:26.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 19, 2023-10-17T19:42:26.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:42:26.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": [], 2023-10-17T19:42:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_map": { 2023-10-17T19:42:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zonegroups": [ 2023-10-17T19:42:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "a", 2023-10-17T19:42:26.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a", 2023-10-17T19:42:26.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": true, 2023-10-17T19:42:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:26.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:26.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:26.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a1", 2023-10-17T19:42:26.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:26.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:26.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:26.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:26.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:26.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:26.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:26.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:26.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:26.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a2", 2023-10-17T19:42:26.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:42:26.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:26.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:26.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:26.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:26.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:26.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:26.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:26.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:26.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:26.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:26.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:26.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:26.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:26.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:26.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [ 2023-10-17T19:42:26.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-group", 2023-10-17T19:42:26.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "data_flow": { 2023-10-17T19:42:26.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "symmetrical": [ 2023-10-17T19:42:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-flow1", 2023-10-17T19:42:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:26.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "directional": [ 2023-10-17T19:42:26.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest_zone": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:26.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pipes": [ 2023-10-17T19:42:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-pipe", 2023-10-17T19:42:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-70", 2023-10-17T19:42:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:42:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": { 2023-10-17T19:42:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-71", 2023-10-17T19:42:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "params": { 2023-10-17T19:42:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "filter": { 2023-10-17T19:42:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [] 2023-10-17T19:42:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": {}, 2023-10-17T19:42:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "priority": 0, 2023-10-17T19:42:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "mode": "system", 2023-10-17T19:42:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user": "" 2023-10-17T19:42:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-pipe-hbmuog-72", 2023-10-17T19:42:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-72", 2023-10-17T19:42:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:42:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": { 2023-10-17T19:42:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-73", 2023-10-17T19:42:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:26.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "params": { 2023-10-17T19:42:26.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "filter": { 2023-10-17T19:42:26.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [] 2023-10-17T19:42:26.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": {}, 2023-10-17T19:42:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "priority": 0, 2023-10-17T19:42:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "mode": "system", 2023-10-17T19:42:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user": "" 2023-10-17T19:42:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-pipe-hbmuog-73", 2023-10-17T19:42:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-73", 2023-10-17T19:42:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160" 2023-10-17T19:42:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": { 2023-10-17T19:42:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-73", 2023-10-17T19:42:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "params": { 2023-10-17T19:42:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "filter": { 2023-10-17T19:42:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [] 2023-10-17T19:42:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": {}, 2023-10-17T19:42:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "priority": 0, 2023-10-17T19:42:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "mode": "system", 2023-10-17T19:42:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user": "" 2023-10-17T19:42:26.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "enabled" 2023-10-17T19:42:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:26.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "b", 2023-10-17T19:42:26.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b", 2023-10-17T19:42:26.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:26.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": false, 2023-10-17T19:42:26.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:26.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b2", 2023-10-17T19:42:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:42:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:26.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b1", 2023-10-17T19:42:26.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:26.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:26.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:26.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:26.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:26.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:26.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:26.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [] 2023-10-17T19:42:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:26.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:26.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "short_zone_ids": [ 2023-10-17T19:42:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3843991460 2023-10-17T19:42:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1846350003 2023-10-17T19:42:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3451171648 2023-10-17T19:42:26.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1989104912 2023-10-17T19:42:26.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zonegroup": "a", 2023-10-17T19:42:26.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:26.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_config": { 2023-10-17T19:42:26.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_quota": { 2023-10-17T19:42:26.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:26.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:26.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:26.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:26.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:26.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_quota": { 2023-10-17T19:42:26.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:26.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:26.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:26.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:26.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:26.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_ratelimit": { 2023-10-17T19:42:26.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:26.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:26.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:26.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:26.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:26.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_ratelimit": { 2023-10-17T19:42:26.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:26.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:26.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:26.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:26.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:26.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "anonymous_ratelimit": { 2023-10-17T19:42:26.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:26.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:26.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:26.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:26.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:26.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:26.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_name": "test-realm", 2023-10-17T19:42:26.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:26.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:26.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.492 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.492 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.492 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.492 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.493 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.493 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c= 2023-10-17T19:42:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c=', 'Content-Length': '0'} 2023-10-17T19:42:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a868a8e63cb4523-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj1 2023-10-17T19:42:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj1 2023-10-17T19:42:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj1 2023-10-17T19:42:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.498 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.498 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:26.498 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:26.499 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.499 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj1 2023-10-17T19:42:26.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.500 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:GFBleF8ayJmh+S6iKG/6cj3KeKg= 2023-10-17T19:42:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:GFBleF8ayJmh+S6iKG/6cj3KeKg='} 2023-10-17T19:42:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006011d930674e02a8-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.504 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.505 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.505 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.505 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.506 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.506 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c= 2023-10-17T19:42:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:nvjUA8MAp1tNeiIx8cu8msb9Z3c=', 'Content-Length': '0'} 2023-10-17T19:42:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c40eb8be3868a5-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj2 2023-10-17T19:42:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj2 2023-10-17T19:42:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj2 2023-10-17T19:42:26.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:26.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.511 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.511 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:26.512 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:26.512 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.512 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj2 2023-10-17T19:42:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.513 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rdM2ZQ1ZQRJK+6PaCON/KnF5v74= 2023-10-17T19:42:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rdM2ZQ1ZQRJK+6PaCON/KnF5v74='} 2023-10-17T19:42:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000447b56d5e917a70d-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.517 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.518 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.518 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.518 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.519 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:26.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.519 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0= 2023-10-17T19:42:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0=', 'Content-Length': '0'} 2023-10-17T19:42:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000588c87b30211342a-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj3 2023-10-17T19:42:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj3 2023-10-17T19:42:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj3 2023-10-17T19:42:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:26.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.524 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.524 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:26.525 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:26.525 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.525 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj3 2023-10-17T19:42:26.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.526 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:6ErFdoxa/unfXlpBdnhXkgg6i+k= 2023-10-17T19:42:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:6ErFdoxa/unfXlpBdnhXkgg6i+k='} 2023-10-17T19:42:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba59ac70c12c7800-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.530 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:26.530 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.530 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.530 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.531 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:26.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.531 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0= 2023-10-17T19:42:26.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:PvPA6c8qH56XnNaElCSYbzzAvI0=', 'Content-Length': '0'} 2023-10-17T19:42:26.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fb0c0fc7bdb5d6b-00652ee195-2588-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj4 2023-10-17T19:42:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj4 2023-10-17T19:42:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj4 2023-10-17T19:42:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:26.536 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:26.536 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:26.537 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:26.537 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:33:41 GMT 2023-10-17T19:42:26.537 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj4 2023-10-17T19:42:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:26.538 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:pugnD5NfWfVGUqiRNX4+bsOxngs= 2023-10-17T19:42:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:33:41 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:pugnD5NfWfVGUqiRNX4+bsOxngs='} 2023-10-17T19:42:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002f931956067d8c4-00652ee195-2588-a1'), ('Date', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:26.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:26.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:26.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:33:40.511436Z" 2023-10-17T19:42:26.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:26.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:26.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:26.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:26.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:42.124+0000 7fb4f1d0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:33:42.126+0000 7fb4f1d0b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:26.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:26.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:42:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:26.569 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.570 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.570 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.570 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.571 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:26.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.571 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.572 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.572 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.572 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:26.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.574 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:26.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:26.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.575 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.575 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.576 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.576 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.576 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:26.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:26.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.579 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.579 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.579 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.580 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.581 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:26.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.582 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:26.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.583 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.583 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.583 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.584 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:26.584 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:26.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:26.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:26.586 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.587 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.587 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.587 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.588 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:26.588 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:26.588 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.589 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:42:26.589 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.589 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:26.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:26.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:26.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:26.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:26.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:26.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:26.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:42:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.620 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.621 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.621 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.621 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.622 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.622 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.622 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.623 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.624 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.624 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.625 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.625 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.626 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.626 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.627 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.628 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.628 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.628 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.629 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.629 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.629 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.630 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.630 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.631 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.632 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.632 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.632 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.632 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.633 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.633 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.634 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.635 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.635 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.635 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.636 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.636 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.636 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.637 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.637 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:42Z" 2023-10-17T19:42:26.637 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.638 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:47Z" 2023-10-17T19:42:26.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.660 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.661 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.661 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.661 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.662 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.662 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.663 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.663 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.663 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.664 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.664 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.665 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.666 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.666 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.667 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.667 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.669 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.669 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.670 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.670 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.671 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.671 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.672 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.672 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.673 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.673 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.674 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.674 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.674 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.675 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.676 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.677 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.677 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.677 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.678 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.678 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.678 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.678 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.679 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:47Z" 2023-10-17T19:42:26.679 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.679 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:52Z" 2023-10-17T19:42:26.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.702 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.703 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.703 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.703 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.704 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.704 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.704 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.705 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.705 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.705 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.706 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.706 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.707 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.708 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.708 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.708 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.709 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.709 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.711 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.711 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.712 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.712 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.713 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.713 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.714 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.715 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.715 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.715 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.716 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.716 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.716 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.717 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.718 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.718 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.719 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.719 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.719 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.719 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.720 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.720 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.720 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:52Z" 2023-10-17T19:42:26.721 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.721 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:33:57Z" 2023-10-17T19:42:26.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.742 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.743 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.743 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.743 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.743 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.744 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.745 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.745 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.745 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.746 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.746 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.748 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.748 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.748 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.751 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.751 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.752 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.752 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.752 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.753 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.753 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.754 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.754 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.755 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.755 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.755 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.756 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.756 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.756 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.757 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.758 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.758 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.758 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.758 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.759 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.759 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.759 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.760 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.760 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:33:57Z" 2023-10-17T19:42:26.760 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.760 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:02Z" 2023-10-17T19:42:26.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.783 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.784 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.784 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.784 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.784 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.785 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.785 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.786 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.786 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.788 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.788 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.788 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.789 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.789 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.789 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.791 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.791 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.792 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.792 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.794 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.795 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.795 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.796 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.796 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.797 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.797 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.797 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.798 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.799 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.799 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.799 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.800 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.800 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.800 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.801 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:02Z" 2023-10-17T19:42:26.801 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.801 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:08Z" 2023-10-17T19:42:26.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.826 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.826 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.826 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.827 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.827 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.828 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.828 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.828 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.829 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.829 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.830 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.831 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.832 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.832 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.832 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.833 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.833 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.835 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.835 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.835 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.836 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.836 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.836 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.837 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.837 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.838 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.838 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.839 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.839 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.840 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.840 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.840 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.841 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.842 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.842 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.842 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.843 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.843 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.843 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.843 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.844 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.844 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.844 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:08Z" 2023-10-17T19:42:26.845 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.845 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:13Z" 2023-10-17T19:42:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.866 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.866 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.867 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.867 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.867 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.868 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.868 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.868 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.868 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.869 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.869 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.871 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.871 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.872 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.872 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.875 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.875 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.876 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.876 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.877 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.877 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.877 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.878 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.879 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.879 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.879 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.880 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.880 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.880 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.881 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.882 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.882 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.882 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.882 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.883 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.883 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.883 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.884 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.884 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:13Z" 2023-10-17T19:42:26.884 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.885 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:18Z" 2023-10-17T19:42:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.907 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.908 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.908 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.908 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.908 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.909 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.909 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.909 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.912 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.912 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.913 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.913 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.914 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.914 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.916 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.916 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.916 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.916 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.917 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.917 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.918 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.919 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.919 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.919 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.920 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.920 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.921 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.922 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.922 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.922 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.922 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.923 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.923 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.923 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.924 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.924 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:18Z" 2023-10-17T19:42:26.924 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.924 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:23Z" 2023-10-17T19:42:26.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.946 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.946 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.947 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.947 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.948 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.948 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.948 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.948 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.949 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.949 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.951 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.951 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.951 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.952 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.952 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.953 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.954 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.954 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.955 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.955 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.956 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.956 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.957 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:26.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.958 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.958 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.958 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.958 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:26.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.959 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.960 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.960 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:26.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.961 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.961 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.962 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:26.962 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:26.963 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:26.963 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:26.964 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.964 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:23Z" 2023-10-17T19:42:26.964 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:26.965 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:26.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:26.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:26.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:26.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:26.966 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:26.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:26.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:26.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:26.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:26.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:26.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:26.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:26.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:26.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:28Z" 2023-10-17T19:42:26.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:26.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:26.988 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:26.988 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:26.988 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:26.989 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:26.989 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:26.989 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.989 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:26.990 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.990 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:26.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.991 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.991 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:26.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:26.993 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.993 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.993 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.994 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:26.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.994 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.995 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:26.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:26.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:26.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:26.996 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:26.996 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:26.997 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:26.997 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:26.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:26.998 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:26.998 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:26.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:26.999 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.002 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.002 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.002 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.003 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.003 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.003 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.004 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.004 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.005 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.005 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.006 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.006 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.006 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.007 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.007 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.008 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.008 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:28Z" 2023-10-17T19:42:27.008 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.008 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:33Z" 2023-10-17T19:42:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.032 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.032 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.032 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.033 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.033 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.033 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.034 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.034 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.034 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.035 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.035 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.037 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.037 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.037 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.038 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.038 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.039 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.040 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.040 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.041 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.041 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.041 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.042 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.042 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.043 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.044 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.044 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.044 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.045 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.045 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.045 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.046 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.046 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.047 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.048 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.048 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.048 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.048 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.049 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.049 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.049 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.049 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:33Z" 2023-10-17T19:42:27.050 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.050 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:38Z" 2023-10-17T19:42:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.072 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.073 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.074 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.074 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.074 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.075 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.075 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.076 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.076 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.078 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.078 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.079 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.079 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.080 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.080 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.081 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.081 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.082 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.082 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.082 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.083 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.083 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.083 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.084 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.085 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.085 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.085 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.086 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.086 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.087 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.088 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.088 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.089 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.089 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.089 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.090 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.090 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.090 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.091 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:38Z" 2023-10-17T19:42:27.091 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.091 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:43Z" 2023-10-17T19:42:27.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.114 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.114 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.115 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.115 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.116 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.116 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.116 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.117 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.117 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.118 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.119 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.119 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.120 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.120 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.122 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.122 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.122 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.123 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.123 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.123 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.124 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.124 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.125 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.126 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.126 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.126 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.127 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.127 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.128 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.129 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.129 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.130 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.130 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.130 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.131 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.131 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.131 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.132 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:43Z" 2023-10-17T19:42:27.132 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.132 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:49Z" 2023-10-17T19:42:27.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.153 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.154 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.154 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.154 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.155 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.155 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.155 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.156 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.156 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.157 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.157 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.158 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.159 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.159 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.159 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.160 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.160 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.160 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.161 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.162 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.162 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.162 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.163 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.163 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.163 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.164 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.165 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.165 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.165 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.165 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.166 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.166 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.166 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.167 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.168 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.168 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.168 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.169 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.169 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.169 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.170 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.170 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.170 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.170 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:49Z" 2023-10-17T19:42:27.171 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.171 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:54Z" 2023-10-17T19:42:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.193 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.193 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.194 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.194 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.194 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.195 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.195 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.195 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.196 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.196 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.197 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.198 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.199 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.199 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.199 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.200 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.200 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.202 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.202 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.203 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.203 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.203 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.204 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.204 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.205 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.205 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.206 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.206 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.206 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.207 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.207 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.207 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.207 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.208 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.209 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.209 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.210 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.210 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.210 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.211 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.211 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.211 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.212 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:54Z" 2023-10-17T19:42:27.212 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.212 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:34:59Z" 2023-10-17T19:42:27.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.235 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.236 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.236 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.236 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.237 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.237 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.238 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.238 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.239 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.239 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.241 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.241 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.241 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.241 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.242 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.242 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.242 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.243 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.244 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.244 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.244 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.245 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.245 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.245 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.245 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.246 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.246 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.247 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.248 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.248 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.248 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.249 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.249 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.249 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.250 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.251 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.251 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.251 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.252 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.252 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.252 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.252 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.253 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.253 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:34:59Z" 2023-10-17T19:42:27.253 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.254 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:04Z" 2023-10-17T19:42:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.275 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.276 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.276 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.276 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.277 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.277 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.277 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.278 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.278 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.278 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.279 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.279 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.280 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.281 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.281 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.282 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.282 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.284 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.284 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.284 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.285 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.285 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.285 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.285 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.286 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.287 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.288 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.288 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.289 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.289 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.290 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.291 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.291 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.291 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.291 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.292 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.292 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.292 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.293 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:04Z" 2023-10-17T19:42:27.293 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.293 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:09Z" 2023-10-17T19:42:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.316 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.317 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.317 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.318 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.318 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.319 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.319 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.319 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.320 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.320 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.322 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.322 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.322 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.323 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.323 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.323 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.324 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.325 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.325 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.326 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.326 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.326 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.327 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.327 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.328 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.328 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.329 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.329 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.329 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.330 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.330 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.330 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.331 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.332 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.332 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.332 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.332 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.333 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.333 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.333 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.333 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:09Z" 2023-10-17T19:42:27.334 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.334 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:14Z" 2023-10-17T19:42:27.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.356 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.357 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.357 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.357 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.358 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.358 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.358 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.359 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.359 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.359 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.360 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.361 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.361 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.362 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.362 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.362 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.363 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.363 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.364 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.365 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.365 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.365 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.367 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.368 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.368 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.368 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.370 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.371 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.371 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.372 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.372 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.372 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.373 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.373 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.374 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.374 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:14Z" 2023-10-17T19:42:27.374 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:19Z" 2023-10-17T19:42:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.398 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.398 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.398 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.399 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.399 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.399 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.400 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.400 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.400 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.401 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.401 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.401 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.403 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.403 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.403 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.404 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.406 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.407 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.407 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.407 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.408 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.408 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.408 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.409 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.410 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.410 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.411 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.411 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.411 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.412 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.412 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.412 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.413 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.414 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.414 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.415 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.415 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.415 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.416 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.416 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.416 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.417 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:19Z" 2023-10-17T19:42:27.417 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.417 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:25Z" 2023-10-17T19:42:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.440 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.441 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.441 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.441 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.442 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.442 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.443 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.443 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.445 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.445 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.445 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.445 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.446 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.446 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.446 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.448 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.448 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.449 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.449 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.450 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.450 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.451 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.452 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.452 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.452 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.453 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.453 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.453 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.454 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.454 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.455 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.455 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.456 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.456 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.456 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.456 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.457 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.457 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.457 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:25Z" 2023-10-17T19:42:27.458 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.458 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:30Z" 2023-10-17T19:42:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.479 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.479 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.479 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.479 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.480 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.480 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.480 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.481 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.481 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.481 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.481 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.482 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.482 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.483 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.484 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.484 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.484 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.484 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.485 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.485 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.486 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.487 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.487 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.487 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.488 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.488 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.488 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.489 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.490 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.490 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.490 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.490 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.491 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.491 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.491 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.492 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.492 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.492 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.493 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.493 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.493 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.493 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.494 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.494 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.494 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.495 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:30Z" 2023-10-17T19:42:27.495 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.495 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:35Z" 2023-10-17T19:42:27.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.518 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.519 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.519 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.519 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.520 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.520 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.520 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.520 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.521 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.521 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.521 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.522 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.523 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.523 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.524 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.524 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.524 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.525 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.525 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.526 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.527 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.527 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.527 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.528 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.528 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.528 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.529 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.530 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.530 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.531 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.531 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.531 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.531 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.532 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.532 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.533 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.533 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.533 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.534 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.534 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.534 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.535 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.535 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.535 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:35Z" 2023-10-17T19:42:27.535 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.536 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:40Z" 2023-10-17T19:42:27.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.558 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.558 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.559 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.559 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.559 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.560 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.560 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.560 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.561 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.561 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.561 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.562 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.563 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.563 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.563 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.564 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.564 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.564 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.565 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.565 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040982", 2023-10-17T19:42:27.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.272525Z" 2023-10-17T19:42:27.566 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.567 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.567 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.567 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.568 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.568 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.568 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028515", 2023-10-17T19:42:27.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.569 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:18:37.300757Z" 2023-10-17T19:42:27.570 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.570 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.570 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.571 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.571 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.571 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.571 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046856", 2023-10-17T19:42:27.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.572 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:31:36.237304Z" 2023-10-17T19:42:27.573 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.573 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.573 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.574 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.574 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.574 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.575 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.575 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.575 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:40Z" 2023-10-17T19:42:27.575 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.576 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040982', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.272525Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028515', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:18:37.300757Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046856', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:31:36.237304Z'}}] 2023-10-17T19:42:27.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000040982', 2: '00000000000000000000:00000000000000028515', 3: '00000000000000000000:00000000000000046856'} 2023-10-17T19:42:27.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041122 target=00000000000000000000:00000000000000040982, shard=2 master=00000000000000000000:00000000000000028656 target=00000000000000000000:00000000000000028515, shard=3 master=00000000000000000000:00000000000000047136 target=00000000000000000000:00000000000000046856 2023-10-17T19:42:27.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.577 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:27.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:27.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:27.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:27.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:27.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:45Z" 2023-10-17T19:42:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:27.597 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.597 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.598 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.598 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.598 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:27.598 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.599 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.600 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.600 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.601 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:27.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:27.603 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.603 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.603 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.604 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.604 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.604 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.605 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:27.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:27.606 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.607 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.607 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.607 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.608 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.608 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:27.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.609 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:27.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:27.610 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.610 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.611 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.611 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.612 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:27.612 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:27.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.613 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:27.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:27.613 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.614 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.614 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.614 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.615 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.615 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:27.615 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:27.615 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.616 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:45Z" 2023-10-17T19:42:27.616 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.616 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:42:27.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:27.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:27.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:27.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:27.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:27.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:27.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.622 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.622 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.623 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.623 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.623 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.624 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ip6nsxiiJDLCGbuUHA5SJ0ZzK2g= 2023-10-17T19:42:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ip6nsxiiJDLCGbuUHA5SJ0ZzK2g=', 'Content-Length': '0'} 2023-10-17T19:42:27.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074d14d95df35ef6d-00652ee211-2584-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj1 2023-10-17T19:42:27.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj1 2023-10-17T19:42:27.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj1 2023-10-17T19:42:27.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.628 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.629 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.629 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.629 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.630 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj1 2023-10-17T19:42:27.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.630 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jIjVm0hSvIehTVnXQIO/Urc9R/w= 2023-10-17T19:42:27.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jIjVm0hSvIehTVnXQIO/Urc9R/w=', 'Content-Length': '0'} 2023-10-17T19:42:27.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001e4099544d22f09b-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj2 2023-10-17T19:42:27.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj2 2023-10-17T19:42:27.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj2 2023-10-17T19:42:27.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.635 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.635 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.635 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.636 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj2 2023-10-17T19:42:27.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.636 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:tTcgFpxliZWHOWCvHCbj4Xkl2vY= 2023-10-17T19:42:27.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:tTcgFpxliZWHOWCvHCbj4Xkl2vY=', 'Content-Length': '0'} 2023-10-17T19:42:27.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000db4d77fd08ce9791-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj3 2023-10-17T19:42:27.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj3 2023-10-17T19:42:27.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj3 2023-10-17T19:42:27.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.641 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.641 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.641 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.642 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.642 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj3 2023-10-17T19:42:27.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.643 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:58NyIyKPQj8bC3TEwGXEdS86nhQ= 2023-10-17T19:42:27.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:58NyIyKPQj8bC3TEwGXEdS86nhQ=', 'Content-Length': '0'} 2023-10-17T19:42:27.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000089d2d711bb027878-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj4 2023-10-17T19:42:27.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj4 2023-10-17T19:42:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj4 2023-10-17T19:42:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.647 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.648 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.648 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.648 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.649 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj4 2023-10-17T19:42:27.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.649 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:bBSGuPf8+IczGZyg83BJpnN7RmY= 2023-10-17T19:42:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:bBSGuPf8+IczGZyg83BJpnN7RmY=', 'Content-Length': '0'} 2023-10-17T19:42:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006e6fa38c0cf7a3fd-00652ee211-2584-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:27.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:27.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.654 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.655 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.655 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.655 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.656 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:27.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.656 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Vl5/cdJYoKxC/bABPYFwYzPC7wA= 2023-10-17T19:42:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Vl5/cdJYoKxC/bABPYFwYzPC7wA=', 'Content-Length': '0'} 2023-10-17T19:42:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ed1bdb17cb36d8a-00652ee211-2584-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj1 2023-10-17T19:42:27.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj1 2023-10-17T19:42:27.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj1 2023-10-17T19:42:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.662 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.662 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.662 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.663 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj1 2023-10-17T19:42:27.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.663 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:+yRyzan9v08UXwnRaAcNzOs2BLU= 2023-10-17T19:42:27.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:+yRyzan9v08UXwnRaAcNzOs2BLU=', 'Content-Length': '0'} 2023-10-17T19:42:27.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d728c3808dd6b1d4-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj1 2023-10-17T19:42:27.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj1 2023-10-17T19:42:27.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:27.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj1 2023-10-17T19:42:27.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.667 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:27.668 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.668 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.668 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.668 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj1 2023-10-17T19:42:27.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.669 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:w5FCAF4HmaijGJ+ile4fnLdFbVk= 2023-10-17T19:42:27.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:w5FCAF4HmaijGJ+ile4fnLdFbVk=', 'Content-Length': '0'} 2023-10-17T19:42:27.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006427ceaa52446a1d-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj2 2023-10-17T19:42:27.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj2 2023-10-17T19:42:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj2 2023-10-17T19:42:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.673 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.674 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.674 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.674 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.675 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj2 2023-10-17T19:42:27.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.675 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:dt3SS4h+Uns9Lfoey4D7kNl2jsM= 2023-10-17T19:42:27.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:dt3SS4h+Uns9Lfoey4D7kNl2jsM=', 'Content-Length': '0'} 2023-10-17T19:42:27.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b66ae5256e2b35a4-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj2 2023-10-17T19:42:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj2 2023-10-17T19:42:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj2 2023-10-17T19:42:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.680 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:27.680 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.680 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.681 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.681 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj2 2023-10-17T19:42:27.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.681 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:x+I+NmnXE+W+iNLZ+M+S+NtpWdk= 2023-10-17T19:42:27.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:x+I+NmnXE+W+iNLZ+M+S+NtpWdk=', 'Content-Length': '0'} 2023-10-17T19:42:27.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-72:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e24da96fa82ac2fe-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj3 2023-10-17T19:42:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj3 2023-10-17T19:42:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj3 2023-10-17T19:42:27.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.686 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.686 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.687 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.687 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.687 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj3 2023-10-17T19:42:27.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.688 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XxrAEprI9GxuidcMkGsdQSycNXg= 2023-10-17T19:42:27.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XxrAEprI9GxuidcMkGsdQSycNXg=', 'Content-Length': '0'} 2023-10-17T19:42:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090786a7fe150a8bb-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj3 2023-10-17T19:42:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj3 2023-10-17T19:42:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj3 2023-10-17T19:42:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.692 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:27.693 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.693 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.693 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.694 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj3 2023-10-17T19:42:27.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.694 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:id591CZLAv/wGK8iaRIW7fizvR4= 2023-10-17T19:42:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:id591CZLAv/wGK8iaRIW7fizvR4=', 'Content-Length': '0'} 2023-10-17T19:42:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041b5197caa655d7b-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj4 2023-10-17T19:42:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj4 2023-10-17T19:42:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj4 2023-10-17T19:42:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.699 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:27.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.700 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.700 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.701 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj4 2023-10-17T19:42:27.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.701 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:zqag8nKSfFKKZ0sJRoDp1nQG66o= 2023-10-17T19:42:27.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:zqag8nKSfFKKZ0sJRoDp1nQG66o=', 'Content-Length': '0'} 2023-10-17T19:42:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de004abbd257428c-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj4 2023-10-17T19:42:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj4 2023-10-17T19:42:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-17T19:42:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj4 2023-10-17T19:42:27.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:27.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:27.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:27.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:27.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:27.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:27.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:27.706 INFO:tasks.rgw_multisite_tests:GET 2023-10-17T19:42:27.706 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.707 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.707 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:35:45 GMT 2023-10-17T19:42:27.707 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj4 2023-10-17T19:42:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:27.708 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:Ips92TLOxWlT8La1nITTs7R7Zi8= 2023-10-17T19:42:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:35:45 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:Ips92TLOxWlT8La1nITTs7R7Zi8=', 'Content-Length': '0'} 2023-10-17T19:42:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 17 Oct 2023 19:33:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4c84373e-d451-49bb-a8c9-c37d5e5e2160:hbmuog-73:4c84373e-d451-49bb-a8c9-c37d5e5e2160.9292.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047b7bf52420593dc-00652ee211-2584-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 17 Oct 2023 19:35:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:27.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"hbmuog-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:27.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hbmuog-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hbmuog-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:27.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:27.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1"]},"dest":{"bucket":"hbmuog-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:27.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-17T19:42:27.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:27.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "epoch": 20, 2023-10-17T19:42:27.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "predecessor_uuid": "291121e4-630c-452a-bf9e-7afaaae7cbe0", 2023-10-17T19:42:27.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": [], 2023-10-17T19:42:27.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_map": { 2023-10-17T19:42:27.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zonegroups": [ 2023-10-17T19:42:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "a", 2023-10-17T19:42:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a", 2023-10-17T19:42:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": true, 2023-10-17T19:42:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a1", 2023-10-17T19:42:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8000" 2023-10-17T19:42:27.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:27.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:27.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:27.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:27.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:27.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:27.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:27.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:27.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:27.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "a2", 2023-10-17T19:42:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi049.front.sepia.ceph.com:8001" 2023-10-17T19:42:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:27.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:27.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:27.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:27.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [ 2023-10-17T19:42:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-group", 2023-10-17T19:42:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "data_flow": { 2023-10-17T19:42:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "symmetrical": [ 2023-10-17T19:42:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-flow1", 2023-10-17T19:42:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pipes": [ 2023-10-17T19:42:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "sync-pipe", 2023-10-17T19:42:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:27.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-70", 2023-10-17T19:42:27.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:27.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:27.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": { 2023-10-17T19:42:27.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket": "hbmuog-71", 2023-10-17T19:42:27.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:27.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "8f644c5a-db9c-4565-b32b-e3a6790e3bcc" 2023-10-17T19:42:27.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "params": { 2023-10-17T19:42:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "source": { 2023-10-17T19:42:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "filter": { 2023-10-17T19:42:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [] 2023-10-17T19:42:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "dest": {}, 2023-10-17T19:42:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "priority": 0, 2023-10-17T19:42:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "mode": "system", 2023-10-17T19:42:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user": "" 2023-10-17T19:42:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "allowed" 2023-10-17T19:42:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "b", 2023-10-17T19:42:27.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b", 2023-10-17T19:42:27.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "api_name": "", 2023-10-17T19:42:27.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "is_master": false, 2023-10-17T19:42:27.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:27.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames": [], 2023-10-17T19:42:27.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "hostnames_s3website": [], 2023-10-17T19:42:27.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:27.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "zones": [ 2023-10-17T19:42:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b2", 2023-10-17T19:42:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8001" 2023-10-17T19:42:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:27.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "id": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:27.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "b1", 2023-10-17T19:42:27.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "endpoints": [ 2023-10-17T19:42:27.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "http://smithi071.front.sepia.ceph.com:8000" 2023-10-17T19:42:27.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_meta": false, 2023-10-17T19:42:27.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "log_data": true, 2023-10-17T19:42:27.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-17T19:42:27.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "read_only": false, 2023-10-17T19:42:27.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tier_type": "", 2023-10-17T19:42:27.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from_all": true, 2023-10-17T19:42:27.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_from": [], 2023-10-17T19:42:27.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "redirect_zone": "", 2023-10-17T19:42:27.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "supported_features": [ 2023-10-17T19:42:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "compress-encrypted", 2023-10-17T19:42:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "placement_targets": [ 2023-10-17T19:42:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "name": "default-placement", 2023-10-17T19:42:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "tags": [], 2023-10-17T19:42:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "storage_classes": [ 2023-10-17T19:42:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "STANDARD" 2023-10-17T19:42:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "default_placement": "default-placement", 2023-10-17T19:42:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:27.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_policy": { 2023-10-17T19:42:27.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "groups": [] 2023-10-17T19:42:27.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled_features": [ 2023-10-17T19:42:27.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "resharding" 2023-10-17T19:42:27.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ], 2023-10-17T19:42:27.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "short_zone_ids": [ 2023-10-17T19:42:27.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "049bc79f-3bb6-4b5b-bc52-f8b1db99140f", 2023-10-17T19:42:27.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3843991460 2023-10-17T19:42:27.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1846350003 2023-10-17T19:42:27.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "8f644c5a-db9c-4565-b32b-e3a6790e3bcc", 2023-10-17T19:42:27.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 3451171648 2023-10-17T19:42:27.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": "ad5bf4b6-5c62-4abf-bddb-0259cafe1aab", 2023-10-17T19:42:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": 1989104912 2023-10-17T19:42:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zonegroup": "a", 2023-10-17T19:42:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "master_zone": "4c84373e-d451-49bb-a8c9-c37d5e5e2160", 2023-10-17T19:42:27.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period_config": { 2023-10-17T19:42:27.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_quota": { 2023-10-17T19:42:27.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:27.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:27.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:27.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:27.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:27.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_quota": { 2023-10-17T19:42:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false, 2023-10-17T19:42:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "check_on_raw": false, 2023-10-17T19:42:27.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size": -1, 2023-10-17T19:42:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_size_kb": 0, 2023-10-17T19:42:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_objects": -1 2023-10-17T19:42:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "user_ratelimit": { 2023-10-17T19:42:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "bucket_ratelimit": { 2023-10-17T19:42:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "anonymous_ratelimit": { 2023-10-17T19:42:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_ops": 0, 2023-10-17T19:42:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_ops": 0, 2023-10-17T19:42:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_read_bytes": 0, 2023-10-17T19:42:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "max_write_bytes": 0, 2023-10-17T19:42:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "enabled": false 2023-10-17T19:42:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_id": "19b3e86c-cc39-4be9-99c4-971ddfc4a8c5", 2023-10-17T19:42:27.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_name": "test-realm", 2023-10-17T19:42:27.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:27.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-17T19:42:27.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-17T19:42:27.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-72 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-72 --source-bucket=hbmuog-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:42:27.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hbmuog-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hbmuog-72', '--source-bucket=hbmuog-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hbmuog-73 --source-zones=a1 --dest-zones=a2 --bucket=hbmuog-72 --source-bucket=hbmuog-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-17T19:42:27.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hbmuog-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hbmuog-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hbmuog-72","source":{"bucket":"hbmuog-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hbmuog-73","source":{"bucket":"hbmuog-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-17T19:42:27.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-17T19:42:27.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-17T19:42:27.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hbmuog-70","zones":["a1","a2"]},"dest":{"bucket":"hbmuog-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-17T19:42:27.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:35:47.356986Z" 2023-10-17T19:42:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:27.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:47.650+0000 7f946d6f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:47.651+0000 7f946d6f9780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:27.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:27.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:27.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:27.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:47Z" 2023-10-17T19:42:27.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:27.846 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.846 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.847 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.847 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.847 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.848 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.848 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.848 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.849 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.849 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.850 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.850 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:27.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.851 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.852 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.852 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.852 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.853 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.855 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.855 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.856 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.856 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:27.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.858 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:27.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.858 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.859 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.859 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.859 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.859 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.862 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.862 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.862 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.863 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.863 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.863 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:27.863 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:27.864 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.864 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:47Z" 2023-10-17T19:42:27.864 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.864 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:27.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:27.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:52.778+0000 7f042d799780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:52.779+0000 7f042d799780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:27.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:27.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:52Z" 2023-10-17T19:42:27.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:27.889 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.889 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.889 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.890 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.890 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.891 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.891 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.891 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.892 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.892 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.892 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.893 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.894 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:27.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.894 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.895 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.895 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.896 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.896 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.896 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.899 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.899 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.899 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.900 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.900 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.901 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:27.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.901 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.902 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:27.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.903 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.903 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.903 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.904 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.904 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.907 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.907 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.907 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.908 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.908 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:27.908 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:27.909 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.909 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:52Z" 2023-10-17T19:42:27.909 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.909 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:27.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:27.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:57.912+0000 7f68ce7cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:35:57.913+0000 7f68ce7cc780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:35:57Z" 2023-10-17T19:42:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:27.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.934 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.934 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.934 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.935 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.935 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.936 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.936 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.936 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.937 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.938 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:27.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.939 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.940 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.940 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.940 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.941 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.941 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.943 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.943 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.944 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.944 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.945 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:27.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.946 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:27.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.947 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.947 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.948 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.948 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.948 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.951 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.951 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.951 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.952 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.952 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.952 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:27.953 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:27.953 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.953 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:35:57Z" 2023-10-17T19:42:27.954 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:27.954 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:27.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:27.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:27.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:27.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:03.043+0000 7fad12982780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:03.044+0000 7fad12982780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:27.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:27.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:27.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:27.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:03Z" 2023-10-17T19:42:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:27.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:27.980 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:27.980 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:27.980 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:27.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:27.981 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:27.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.982 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.982 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:27.983 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.983 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:27.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:27.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.985 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:27.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:27.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.986 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.986 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.987 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.987 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:27.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:27.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:27.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.990 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.991 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.991 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.991 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:27.992 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.992 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:27.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.994 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:27.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.994 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.994 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.995 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:27.995 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:27.995 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:27.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:27.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:27.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:27.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:27.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:27.997 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:27.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:27.998 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.998 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:27.998 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:27.999 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:27.999 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:27.999 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.000 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.000 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:03Z" 2023-10-17T19:42:28.001 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.001 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:08.171+0000 7f336948e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:08.172+0000 7f336948e780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:08Z" 2023-10-17T19:42:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.026 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.026 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.027 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.027 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.027 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.028 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.028 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.028 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.028 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.029 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.029 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.029 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.030 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.032 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.032 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.032 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.033 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.033 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.035 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.036 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.036 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.036 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.037 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.038 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.039 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.039 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.040 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.040 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.041 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.043 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.043 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.043 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.044 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.044 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.044 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.044 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.045 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.045 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:08Z" 2023-10-17T19:42:28.045 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.046 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:13.297+0000 7f75e86a6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:13.299+0000 7f75e86a6780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:13Z" 2023-10-17T19:42:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.070 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.070 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.070 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.071 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.072 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.072 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.073 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.075 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.076 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.076 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.077 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.077 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.080 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.081 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.081 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.081 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.082 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.083 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.083 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.084 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.084 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.085 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.085 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.085 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.085 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.088 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.088 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.088 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.089 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.089 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.089 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.090 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.090 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.090 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:13Z" 2023-10-17T19:42:28.090 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.091 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:18.430+0000 7f8271558780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:18.431+0000 7f8271558780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:18Z" 2023-10-17T19:42:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.116 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.116 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.117 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.117 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.117 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.118 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.118 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.119 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.120 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.121 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.121 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.121 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.122 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.122 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.123 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.124 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.124 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.125 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.125 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.125 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.125 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.126 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.126 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.127 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.128 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.128 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.129 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.129 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.131 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.132 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.132 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.132 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.133 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.133 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.133 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.133 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.134 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.134 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:18Z" 2023-10-17T19:42:28.134 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.135 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:23.558+0000 7f6f9bc5d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:23.559+0000 7f6f9bc5d780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:23Z" 2023-10-17T19:42:28.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.159 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.159 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.160 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.160 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.161 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.161 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.162 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.162 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.163 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.163 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.164 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.165 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.165 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.165 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.166 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.166 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.168 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.168 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.169 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.169 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.170 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.171 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.172 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.172 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.173 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.173 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.173 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.176 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.176 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.177 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.177 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.177 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.178 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.178 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.178 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.179 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:23Z" 2023-10-17T19:42:28.179 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.179 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.181 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:28.695+0000 7fee93cef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:28.696+0000 7fee93cef780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:28Z" 2023-10-17T19:42:28.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.204 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.204 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.204 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.205 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.205 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.206 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.206 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.206 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.207 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.207 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.208 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.209 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.210 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.210 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.210 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.213 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.213 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.214 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.214 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.214 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.216 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.217 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.217 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.218 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.219 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.220 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.221 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.221 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.221 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.222 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.222 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.222 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.223 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.223 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.223 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:28Z" 2023-10-17T19:42:28.223 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.224 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:33.825+0000 7f570b288780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:33.826+0000 7f570b288780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:33Z" 2023-10-17T19:42:28.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.248 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.248 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.248 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.248 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.249 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.249 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.250 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.250 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.250 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.252 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.253 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.254 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.254 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.254 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.255 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.255 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.257 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.258 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.258 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.258 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.259 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.260 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.261 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.261 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.262 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.262 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.265 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.265 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.266 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.266 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.266 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.267 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.267 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.267 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.268 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:33Z" 2023-10-17T19:42:28.268 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.268 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:38.955+0000 7fa517b62780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:38.956+0000 7fa517b62780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:38Z" 2023-10-17T19:42:28.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.293 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.293 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.294 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.294 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.294 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.295 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.295 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.296 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.296 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.298 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.299 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.299 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.300 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.300 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.300 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.303 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.304 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.304 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.305 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.305 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.306 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.306 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.307 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.308 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.308 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.309 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.309 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.309 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.312 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.312 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.313 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.313 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.313 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.314 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.314 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.314 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.315 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:38Z" 2023-10-17T19:42:28.315 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.315 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:44.084+0000 7f3f7fc63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:44.086+0000 7f3f7fc63780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:44Z" 2023-10-17T19:42:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.341 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.341 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.342 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.342 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.342 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.343 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.343 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.343 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.344 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.344 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.345 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.346 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.347 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.347 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.347 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.347 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.348 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.348 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.350 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.350 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.350 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.351 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.351 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.352 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.353 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.353 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.354 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.354 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.354 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.355 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.355 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.357 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.358 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.358 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.358 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.359 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.359 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.359 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.360 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.360 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:44Z" 2023-10-17T19:42:28.360 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.360 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:49.213+0000 7f404cd2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:49.214+0000 7f404cd2b780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:49Z" 2023-10-17T19:42:28.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.385 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.385 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.386 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.386 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.386 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.387 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.387 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.388 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.388 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.388 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.389 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.390 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.391 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.391 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.392 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.392 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.395 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.395 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.396 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.396 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.396 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.397 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.398 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.399 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.399 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.400 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.400 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.400 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.403 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.403 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.403 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.404 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.404 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.404 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.405 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.405 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.405 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:49Z" 2023-10-17T19:42:28.406 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.406 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:54.341+0000 7f3788d7a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:54.342+0000 7f3788d7a780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:54Z" 2023-10-17T19:42:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.439 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.440 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.441 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.441 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.442 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.442 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.443 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.444 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.445 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.445 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.446 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.446 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.449 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.449 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.450 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.450 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.450 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.451 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.452 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.453 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.453 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.453 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.454 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.457 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.457 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.458 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.458 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.458 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.459 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.459 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.459 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.460 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:54Z" 2023-10-17T19:42:28.460 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.460 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:59.467+0000 7fb807934780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:36:59.468+0000 7fb807934780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:36:59Z" 2023-10-17T19:42:28.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.484 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.485 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.485 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.485 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.486 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.486 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.486 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.487 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.488 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.490 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.490 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.490 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.491 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.491 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.494 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.495 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.495 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.495 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.496 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.497 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.498 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.498 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.499 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.499 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.502 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.502 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.503 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.503 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.503 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.503 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.504 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.504 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.504 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:36:59Z" 2023-10-17T19:42:28.504 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.505 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:04.598+0000 7f3c0ca16780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:04.599+0000 7f3c0ca16780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:04Z" 2023-10-17T19:42:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.532 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.532 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.532 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.533 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.534 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.534 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.535 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.535 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.536 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.537 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.538 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.538 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.538 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.539 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.542 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.542 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.543 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.544 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.545 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.545 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.547 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.548 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.548 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.549 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.549 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.549 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.550 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.551 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.552 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.552 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.552 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.553 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.553 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.554 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.554 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.555 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.555 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:04Z" 2023-10-17T19:42:28.556 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.556 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:09.731+0000 7fd481b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:09.732+0000 7fd481b78780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:09Z" 2023-10-17T19:42:28.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.582 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.583 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.583 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.583 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.584 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.584 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.585 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.585 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.585 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.586 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.587 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.588 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.590 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.591 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.591 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.592 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.592 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.596 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.596 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.597 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.597 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.598 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.599 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.601 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.602 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.602 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.603 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.604 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.606 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.606 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.607 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.607 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.607 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.608 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.608 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.608 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.608 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:09Z" 2023-10-17T19:42:28.609 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.609 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:14.857+0000 7fcbaac12780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:14.858+0000 7fcbaac12780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:14Z" 2023-10-17T19:42:28.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.635 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.635 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.636 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.636 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.636 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.637 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.638 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.638 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.639 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.640 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.642 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.644 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.645 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.645 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.646 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.646 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.646 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.647 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.647 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.647 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.649 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.650 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.650 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.651 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.651 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.652 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.652 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.654 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.655 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.656 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.656 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.657 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.657 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.657 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.660 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.661 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.661 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.661 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.662 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.662 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.662 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.663 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.663 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:14Z" 2023-10-17T19:42:28.663 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.664 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:19.987+0000 7fb1ef045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:19.988+0000 7fb1ef045780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:19Z" 2023-10-17T19:42:28.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.689 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.689 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.689 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.690 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.690 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.691 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.691 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.691 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.692 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.692 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.695 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.695 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.696 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.699 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.699 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.700 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.700 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.701 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.702 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.703 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.704 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.704 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.704 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.705 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.707 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.707 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.708 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.708 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.708 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.709 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.709 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.709 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.710 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:19Z" 2023-10-17T19:42:28.710 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.710 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:25.116+0000 7f7e0ee70780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:25.117+0000 7f7e0ee70780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:25Z" 2023-10-17T19:42:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.736 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.736 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.736 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.737 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.737 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.738 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.738 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.738 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.739 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.740 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.742 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.742 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.742 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.743 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.748 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.748 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.749 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.749 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.750 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.751 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.752 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.752 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.752 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.753 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.753 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.756 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.756 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.757 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.757 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.758 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.758 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.758 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.759 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.759 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:25Z" 2023-10-17T19:42:28.760 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.760 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:30.243+0000 7f00595a7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:30.244+0000 7f00595a7780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:30Z" 2023-10-17T19:42:28.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.791 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.792 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.792 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.793 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.793 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.793 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.794 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.795 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.797 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.797 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.797 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.798 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.800 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.801 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.801 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.801 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.802 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.802 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.803 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.804 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.804 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.805 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.805 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.808 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.808 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.808 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.808 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.809 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.810 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.810 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:30Z" 2023-10-17T19:42:28.810 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.811 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:35.378+0000 7ff6a2751780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:35.379+0000 7ff6a2751780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:35Z" 2023-10-17T19:42:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.835 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.835 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.835 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.835 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.836 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.836 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.837 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.837 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.837 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.838 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.839 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.840 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.840 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.841 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.841 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.841 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.844 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.844 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.844 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.846 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.847 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.848 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.848 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.848 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.848 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.849 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.851 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.852 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.852 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.852 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.852 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.853 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.853 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.853 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.854 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:35Z" 2023-10-17T19:42:28.854 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.854 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:40.506+0000 7fe827040780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:40.507+0000 7fe827040780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:40Z" 2023-10-17T19:42:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.884 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.884 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.885 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.885 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.885 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.886 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.887 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.887 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.888 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.889 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.892 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.892 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.893 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.894 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.898 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.899 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.899 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.900 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.900 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.901 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.901 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.902 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.903 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.904 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.904 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.904 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.907 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.908 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.908 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.908 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.909 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.909 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.909 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.909 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.910 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:40Z" 2023-10-17T19:42:28.910 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.911 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:45.635+0000 7f3705cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:45.636+0000 7f3705cff780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:45Z" 2023-10-17T19:42:28.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.935 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.936 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.936 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.936 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.937 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.937 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.938 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.938 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.938 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571220.511436_1095.1", 2023-10-17T19:42:28.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.939 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:40.511436Z", 2023-10-17T19:42:28.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.941 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.941 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.941 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.941 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.944 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.945 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.945 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.945 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.946 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.946 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.948 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.948 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.948 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.949 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.949 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.952 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.952 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.952 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:28.953 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.953 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:28.953 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:28.954 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:28.954 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.954 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:45Z" 2023-10-17T19:42:28.954 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:28.955 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:28.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571220.511436_1095.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:33:40.511436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:28.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571220.511436_1095.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:28.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697571347.356986_1134.1 target=1_1697571220.511436_1095.1 2023-10-17T19:42:28.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:28.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:50.765+0000 7f3df1b3c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:50.766+0000 7f3df1b3c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:28.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:42:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:50Z" 2023-10-17T19:42:28.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:28.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:28.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:28.979 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:28.980 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:28.980 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:28.980 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:28.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.981 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:28.982 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:28.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:28.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.984 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:28.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:42:28.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.985 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.985 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.986 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.986 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:28.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:28.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:28.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.989 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.990 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.991 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.991 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:28.992 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.993 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:28.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.994 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:28.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.995 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:28.995 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:28.996 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:28.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:28.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:28.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:28.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:28.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:28.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:28.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:28.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:28.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:28.999 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.000 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.000 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.000 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.001 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:29.001 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:29.001 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.002 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:50Z" 2023-10-17T19:42:29.002 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.003 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571347.356986_1134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:35:47.356986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:29.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:29.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:29.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:29.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:29.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:29.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:29.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-17T19:42:29.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.011 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.012 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.012 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.012 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.013 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:29.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.013 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4= 2023-10-17T19:42:29.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4=', 'Content-Length': '0'} 2023-10-17T19:42:29.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049e7a501e6670afb-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj5 2023-10-17T19:42:29.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj5 2023-10-17T19:42:29.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:29.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj5 2023-10-17T19:42:29.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:29.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.018 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:29.018 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:29.019 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:29.019 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.019 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj5 2023-10-17T19:42:29.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.020 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:7OqWnv54bEkp627888Zl/NwqrwU= 2023-10-17T19:42:29.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:7OqWnv54bEkp627888Zl/NwqrwU='} 2023-10-17T19:42:29.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdbb6cc640d9d4af-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:29.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:29.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:29.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.025 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.025 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.025 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.025 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.026 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:29.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.027 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4= 2023-10-17T19:42:29.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:TRdUSVpB8h6Ncdf4p0KuLpxTxD4=', 'Content-Length': '0'} 2023-10-17T19:42:29.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc58951d65ec951f-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj6 2023-10-17T19:42:29.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj6 2023-10-17T19:42:29.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:29.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj6 2023-10-17T19:42:29.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:29.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.031 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:29.032 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:29.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:29.032 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.033 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj6 2023-10-17T19:42:29.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.033 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8ML7fPhP+dinNpfymqRgUlwsqHM= 2023-10-17T19:42:29.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8ML7fPhP+dinNpfymqRgUlwsqHM='} 2023-10-17T19:42:29.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d732bc6b2b18e6cd-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:29.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:29.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:29.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.038 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.038 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.038 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.039 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.039 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:29.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.039 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus= 2023-10-17T19:42:29.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus=', 'Content-Length': '0'} 2023-10-17T19:42:29.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000473f9a6624662329-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj7 2023-10-17T19:42:29.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj7 2023-10-17T19:42:29.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:29.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj7 2023-10-17T19:42:29.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:29.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.044 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:29.044 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:29.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:29.045 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.045 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj7 2023-10-17T19:42:29.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.046 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:AhqfJx1kOSRz5E5/FCd+GWPw5jw= 2023-10-17T19:42:29.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:AhqfJx1kOSRz5E5/FCd+GWPw5jw='} 2023-10-17T19:42:29.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1c6a3acc9303eac-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:29.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:29.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:29.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.051 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.051 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.051 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.051 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.052 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:29.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.054 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus= 2023-10-17T19:42:29.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:XZ1vwXef6wB5379riM/YWQE0Jus=', 'Content-Length': '0'} 2023-10-17T19:42:29.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f35240d96d1b2d6-00652ee28e-26ab-a1'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj8 2023-10-17T19:42:29.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj8 2023-10-17T19:42:29.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-17T19:42:29.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj8 2023-10-17T19:42:29.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-17T19:42:29.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8000 2023-10-17T19:42:29.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-17T19:42:29.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.058 INFO:tasks.rgw_multisite_tests:PUT 2023-10-17T19:42:29.058 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-17T19:42:29.059 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-17T19:42:29.059 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:37:50 GMT 2023-10-17T19:42:29.059 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj8 2023-10-17T19:42:29.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.060 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:ehU4YxZhiyDwVu+Kpc6MQSUrLm8= 2023-10-17T19:42:29.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 17 Oct 2023 19:37:50 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:ehU4YxZhiyDwVu+Kpc6MQSUrLm8='} 2023-10-17T19:42:29.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005799405f3e3eda8e-00652ee28e-26ab-a1'), ('Date', 'Tue, 17 Oct 2023 19:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: No --period given, using current period=cfc9d53d-2765-4da9-b7a8-37407f517c55 2023-10-17T19:42:29.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:29.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:29.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:35:47.356986Z" 2023-10-17T19:42:29.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:29.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.363964Z" 2023-10-17T19:42:29.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:29.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "0.000000" 2023-10-17T19:42:29.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:29.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:03:33.343990Z" 2023-10-17T19:42:29.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:29.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-17T19:42:29.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:51.116+0000 7fc9c1f6c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:29.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: 2023-10-17T19:37:51.117+0000 7fc9c1f6c780 20 RGW-SYNC:meta: read sync status 2023-10-17T19:42:29.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:29.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:29.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:29.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:29.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 1, 2023-10-17T19:42:29.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:42:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "", 2023-10-17T19:42:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "0.000000", 2023-10-17T19:42:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "state": 1, 2023-10-17T19:42:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "realm_epoch": 2 2023-10-17T19:42:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 1, 2023-10-17T19:42:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 1 2023-10-17T19:42:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:42:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-17T19:42:29.094 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.095 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.095 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.095 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.096 INFO:tasks.rgw_multisite_tests: "period": "cfc9d53d-2765-4da9-b7a8-37407f517c55", 2023-10-17T19:42:29.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:29.096 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.096 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.097 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.097 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.097 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:29.098 INFO:tasks.rgw_multisite_tests: "marker": "1_1697571347.356986_1134.1", 2023-10-17T19:42:29.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.099 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-17T19:42:29.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:35:47.356986Z", 2023-10-17T19:42:29.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:29.100 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.100 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.100 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.101 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.101 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:29.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.363964_763.1", 2023-10-17T19:42:29.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.363964Z", 2023-10-17T19:42:29.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:29.104 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.104 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.104 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.105 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.105 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:29.106 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-17T19:42:29.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.107 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-17T19:42:29.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:29.107 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.108 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.108 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-17T19:42:29.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1697569413.343990_262.1", 2023-10-17T19:42:29.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:03:33.343990Z", 2023-10-17T19:42:29.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-17T19:42:29.111 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.111 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.112 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.112 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.112 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.113 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-17T19:42:29.113 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-17T19:42:29.113 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.114 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:42:29.114 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.114 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697571347.356986_1134.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-17T19:35:47.356986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697569413.363964_763.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.363964Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697569413.343990_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:03:33.343990Z', 'realm_epoch': 2}}] 2023-10-17T19:42:29.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:29.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697571347.356986_1134.1', 1: '1_1697569413.363964_763.1', 2: '', 3: '1_1697569413.343990_262.1'} 2023-10-17T19:42:29.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-17T19:42:29.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: [ 2023-10-17T19:42:29.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:42:29.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:42:29.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:29.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:29.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:42:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "last_update": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:42:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:29.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:29.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:29.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:42:29.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:29.144 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.145 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.145 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.145 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.146 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:29.146 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.146 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.147 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.147 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.147 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.148 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.148 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.150 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.150 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.150 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.151 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.151 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.153 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.153 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.153 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.154 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.154 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.155 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.155 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.156 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:29.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.157 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.157 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.157 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.158 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.158 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.158 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.159 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.159 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:29.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.160 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.160 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.161 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.161 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.161 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.162 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:29.162 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:29.162 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.163 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:51Z" 2023-10-17T19:42:29.163 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.163 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:42:29.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:29.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:42:29.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:29.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:29.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:37:56Z" 2023-10-17T19:42:29.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:29.187 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.187 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.187 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.188 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.188 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:29.188 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.189 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.189 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.189 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.190 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.190 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.190 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.192 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.192 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.192 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.193 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.193 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.193 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.194 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.196 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.196 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.196 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.197 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.197 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.198 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.198 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:29.199 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.199 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.199 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.199 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.200 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.200 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.201 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.201 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.201 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.202 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:29.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.203 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.203 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.204 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.204 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.204 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.205 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:29.205 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:29.205 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.206 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:37:56Z" 2023-10-17T19:42:29.206 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.206 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:42:29.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:29.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:42:29.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:29.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:29.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:29.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:29.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:29.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:38:01Z" 2023-10-17T19:42:29.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:29.229 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.229 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.229 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.229 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.230 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:29.230 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.230 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.231 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.231 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.231 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.232 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.232 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.233 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.233 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.234 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.234 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.234 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.235 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.235 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.237 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.237 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.237 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.238 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.238 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.239 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.239 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:29.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.240 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.240 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.241 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.241 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.241 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.242 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.242 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.243 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:29.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.244 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.244 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.244 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.244 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.245 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.245 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:29.245 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:29.246 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.246 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:38:01Z" 2023-10-17T19:42:29.246 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.247 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:42:29.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:29.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:42:29.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:38:06Z" 2023-10-17T19:42:29.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:29.271 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.271 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.271 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.272 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.272 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:29.272 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.272 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.273 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.273 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.273 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.274 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.274 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.276 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.276 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.276 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.277 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.277 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.278 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041122", 2023-10-17T19:42:29.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.702225Z" 2023-10-17T19:42:29.279 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.279 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.280 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.280 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.281 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028656", 2023-10-17T19:42:29.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.282 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:29.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.789887Z" 2023-10-17T19:42:29.283 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.283 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.283 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.284 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.284 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.284 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047136", 2023-10-17T19:42:29.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.285 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:29.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:33:41.762296Z" 2023-10-17T19:42:29.286 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.286 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.286 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.287 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.287 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:29.288 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:29.288 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.288 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:38:06Z" 2023-10-17T19:42:29.289 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.289 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041122', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.702225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028656', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.789887Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047136', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:33:41.762296Z'}}] 2023-10-17T19:42:29.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041122', 2: '00000000000000000000:00000000000000028656', 3: '00000000000000000000:00000000000000047136'} 2023-10-17T19:42:29.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000041262 target=00000000000000000000:00000000000000041122, shard=2 master=00000000000000000000:00000000000000028796 target=00000000000000000000:00000000000000028656, shard=3 master=00000000000000000000:00000000000000047416 target=00000000000000000000:00000000000000047136 2023-10-17T19:42:29.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-17T19:42:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-17T19:42:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-17T19:42:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-17T19:42:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "sync_status": { 2023-10-17T19:42:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "info": { 2023-10-17T19:42:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "sync", 2023-10-17T19:42:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "num_shards": 4, 2023-10-17T19:42:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "instance_id": 16154068986440829441 2023-10-17T19:42:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "markers": [ 2023-10-17T19:42:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 0, 2023-10-17T19:42:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 1, 2023-10-17T19:42:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:42:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 0, 2023-10-17T19:42:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:42:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 2, 2023-10-17T19:42:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 9, 2023-10-17T19:42:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: { 2023-10-17T19:42:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "key": 3, 2023-10-17T19:42:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "val": { 2023-10-17T19:42:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "status": "incremental-sync", 2023-10-17T19:42:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:42:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "next_step_marker": "", 2023-10-17T19:42:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total_entries": 55, 2023-10-17T19:42:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "pos": 0, 2023-10-17T19:42:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:42:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: ] 2023-10-17T19:42:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "full_sync": { 2023-10-17T19:42:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "total": 64, 2023-10-17T19:42:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "complete": 64 2023-10-17T19:42:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: }, 2023-10-17T19:42:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: "current_time": "2023-10-17T19:38:11Z" 2023-10-17T19:42:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi049.stdout: INFO: } 2023-10-17T19:42:29.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-17T19:42:29.311 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-17T19:42:29.311 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-17T19:42:29.312 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-17T19:42:29.312 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-17T19:42:29.312 INFO:tasks.rgw_multisite_tests: "instance_id": 16154068986440829441 2023-10-17T19:42:29.313 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.313 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-17T19:42:29.313 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.314 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-17T19:42:29.314 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.314 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.315 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000040216", 2023-10-17T19:42:29.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:27:51.381293Z" 2023-10-17T19:42:29.316 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.317 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.317 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.317 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-17T19:42:29.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.318 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.318 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000041262", 2023-10-17T19:42:29.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-17T19:42:29.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:37:50.844371Z" 2023-10-17T19:42:29.320 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.320 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.320 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.321 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-17T19:42:29.321 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.321 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.321 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028796", 2023-10-17T19:42:29.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.322 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-17T19:42:29.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:37:50.926594Z" 2023-10-17T19:42:29.323 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.323 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.323 INFO:tasks.rgw_multisite_tests: { 2023-10-17T19:42:29.324 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-17T19:42:29.324 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-17T19:42:29.324 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-17T19:42:29.325 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047416", 2023-10-17T19:42:29.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-17T19:42:29.325 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-17T19:42:29.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-17T19:42:29.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-17T19:37:50.864535Z" 2023-10-17T19:42:29.326 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.326 INFO:tasks.rgw_multisite_tests: } 2023-10-17T19:42:29.327 INFO:tasks.rgw_multisite_tests: ] 2023-10-17T19:42:29.327 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-17T19:42:29.328 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-17T19:42:29.328 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-17T19:42:29.328 INFO:tasks.rgw_multisite_tests: }, 2023-10-17T19:42:29.328 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-17T19:38:11Z" 2023-10-17T19:42:29.329 INFO:tasks.rgw_multisite_tests:} 2023-10-17T19:42:29.329 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:27:51.381293Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000041262', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.844371Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028796', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.926594Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047416', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-17T19:37:50.864535Z'}}] 2023-10-17T19:42:29.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000040216', 1: '00000000000000000000:00000000000000041262', 2: '00000000000000000000:00000000000000028796', 3: '00000000000000000000:00000000000000047416'} 2023-10-17T19:42:29.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-17T19:42:29.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/ 2023-10-17T19:42:29.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/ 2023-10-17T19:42:29.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/ 2023-10-17T19:42:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-17T19:42:29.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.334 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.335 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.335 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.335 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.336 INFO:tasks.rgw_multisite_tests:/hbmuog-73/ 2023-10-17T19:42:29.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.336 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:8ITrzw6hRCYX6qrZ3fgkI7v04hw= 2023-10-17T19:42:29.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:8ITrzw6hRCYX6qrZ3fgkI7v04hw=', 'Content-Length': '0'} 2023-10-17T19:42:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000730d59fc16e9b519-00652ee2a3-26af-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj5 2023-10-17T19:42:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj5 2023-10-17T19:42:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj5 2023-10-17T19:42:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.340 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.340 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.341 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.341 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.341 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj5 2023-10-17T19:42:29.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.342 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:rL75/CMAIClm4NCoIUQOlxUNRo0= 2023-10-17T19:42:29.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:rL75/CMAIClm4NCoIUQOlxUNRo0=', 'Content-Length': '0'} 2023-10-17T19:42:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000061c21bc59653c41b-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj6 2023-10-17T19:42:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj6 2023-10-17T19:42:29.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj6 2023-10-17T19:42:29.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.347 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.347 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.348 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.348 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj6 2023-10-17T19:42:29.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.349 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:O7EAb2VQdIC3hENsBrXIkW4U4ds= 2023-10-17T19:42:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:O7EAb2VQdIC3hENsBrXIkW4U4ds=', 'Content-Length': '0'} 2023-10-17T19:42:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000054f1f0f7dc3fed6a-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj7 2023-10-17T19:42:29.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj7 2023-10-17T19:42:29.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj7 2023-10-17T19:42:29.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.353 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.354 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.354 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.354 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.355 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj7 2023-10-17T19:42:29.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.355 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:W96P5RLmBjXt9E6RMdoILdE70zE= 2023-10-17T19:42:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:W96P5RLmBjXt9E6RMdoILdE70zE=', 'Content-Length': '0'} 2023-10-17T19:42:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009e65f13d73eca24b-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-73/obj8 2023-10-17T19:42:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-73/obj8 2023-10-17T19:42:29.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-73/obj8 2023-10-17T19:42:29.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.359 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.359 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.360 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.360 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.360 INFO:tasks.rgw_multisite_tests:/hbmuog-73/obj8 2023-10-17T19:42:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.361 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:jxH0uFqbn1s6RVb3JcqCNGRTa/k= 2023-10-17T19:42:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:jxH0uFqbn1s6RVb3JcqCNGRTa/k=', 'Content-Length': '0'} 2023-10-17T19:42:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c46153a391f231af-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/ 2023-10-17T19:42:29.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/ 2023-10-17T19:42:29.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/ 2023-10-17T19:42:29.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.365 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.365 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.365 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.366 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.366 INFO:tasks.rgw_multisite_tests:/hbmuog-72/ 2023-10-17T19:42:29.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.367 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:2Vw+LP9XXnf2/qkKXNmXHHWQDiQ= 2023-10-17T19:42:29.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:2Vw+LP9XXnf2/qkKXNmXHHWQDiQ=', 'Content-Length': '0'} 2023-10-17T19:42:29.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a0db4938b5f101a-00652ee2a3-26af-a2'), ('Content-Length', '0'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hbmuog-72/obj7 2023-10-17T19:42:29.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hbmuog-72/obj7 2023-10-17T19:42:29.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-17T19:42:29.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hbmuog-72/obj7 2023-10-17T19:42:29.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-17T19:42:29.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-17T19:42:29.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi049.front.sepia.ceph.com:8001 2023-10-17T19:42:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-17T19:42:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-17T19:42:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-17T19:42:29.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-17T19:42:29.371 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-17T19:42:29.371 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.371 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.372 INFO:tasks.rgw_multisite_tests:Tue, 17 Oct 2023 19:38:11 GMT 2023-10-17T19:42:29.372 INFO:tasks.rgw_multisite_tests:/hbmuog-72/obj7 2023-10-17T19:42:29.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-17T19:42:29.372 INFO:tasks.rgw_multisite_tests:AWS EERKR7CLBH40TV9B4PL4:sEhiNP/kGPp4rHQp2KNhw9hApJ4= 2023-10-17T19:42:29.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Tue, 17 Oct 2023 19:38:11 GMT', 'Authorization': 'AWS EERKR7CLBH40TV9B4PL4:sEhiNP/kGPp4rHQp2KNhw9hApJ4=', 'Content-Length': '0'} 2023-10-17T19:42:29.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000166e34046079af75-00652ee2a3-26af-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 17 Oct 2023 19:38:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-17T19:42:29.373 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-17T19:42:29.374 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.374 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-17T19:42:29.375 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 5283.707s 2023-10-17T19:42:29.375 INFO:tasks.rgw_multisite_tests: 2023-10-17T19:42:29.375 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=6) 2023-10-17T19:42:29.378 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-17T19:42:29.442 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=60920426ddef413bb577b22d1275130e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_cff47d96488123dae42a0c6258323ad6422fe0a6/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-17T19:42:29.443 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-10-17T19:42:29.453 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-10-17T19:42:29.496 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-17T19:42:29.529 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-17T19:42:29.530 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:29.542 INFO:tasks.rgw.c1.client.0.smithi049.stdout:2023-10-17T19:42:29.528+0000 7f95dbb38700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 121632) UID: 0 2023-10-17T19:42:29.543 INFO:tasks.rgw.c1.client.0.smithi049.stdout:2023-10-17T19:42:29.528+0000 7f95dfda17c0 -1 shutting down 2023-10-17T19:42:30.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-17T19:42:35.632 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-17T19:42:35.633 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-10-17T19:42:35.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/vault-root-token 2023-10-17T19:42:35.731 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-17T19:42:35.731 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:35.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-10-17T19:42:35.772 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:42:35.771+0000 7fc71c922700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 173240) UID: 0 2023-10-17T19:42:35.772 INFO:tasks.rgw.c1.client.1.smithi049.stdout:2023-10-17T19:42:35.771+0000 7fc720b8b7c0 -1 shutting down 2023-10-17T19:42:41.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-17T19:42:41.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-17T19:42:41.834 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-17T19:42:41.834 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-10-17T19:42:41.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/vault-root-token 2023-10-17T19:42:41.931 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-17T19:42:41.931 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:42.006 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T19:42:41.931+0000 7fc7ca42c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 121835) UID: 0 2023-10-17T19:42:42.006 INFO:tasks.rgw.c2.client.0.smithi071.stdout:2023-10-17T19:42:41.931+0000 7fc7ce6957c0 -1 shutting down 2023-10-17T19:42:46.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-17T19:42:47.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-17T19:42:48.033 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-17T19:42:48.034 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-10-17T19:42:48.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/vault-root-token 2023-10-17T19:42:48.328 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-17T19:42:48.328 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:48.329 INFO:tasks.rgw.c2.client.1.smithi071.stdout:2023-10-17T19:42:48.328+0000 7fb97fd63700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 123129) UID: 0 2023-10-17T19:42:48.330 INFO:tasks.rgw.c2.client.1.smithi071.stdout:2023-10-17T19:42:48.328+0000 7fb983fcc7c0 -1 shutting down 2023-10-17T19:42:52.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-17T19:42:52.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-17T19:42:54.431 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-17T19:42:54.432 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-10-17T19:42:54.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/vault-root-token 2023-10-17T19:42:54.527 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-17T19:42:54.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-10-17T19:42:55.301 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-17T19:42:55.302 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-17T19:42:55.303 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-10-17T19:42:55.303 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:55.384 INFO:tasks.ceph.c2.osd.0:Stopped 2023-10-17T19:42:55.385 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-10-17T19:42:55.385 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:55.449 INFO:tasks.ceph.c2.osd.1:Stopped 2023-10-17T19:42:55.450 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-10-17T19:42:55.450 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:55.525 INFO:tasks.ceph.c2.osd.2:Stopped 2023-10-17T19:42:55.525 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-17T19:42:55.526 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-10-17T19:42:55.526 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:55.550 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-10-17T19:42:55.551 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-17T19:42:55.552 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-10-17T19:42:55.552 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:55.608 INFO:tasks.ceph.c2.mon.a:Stopped 2023-10-17T19:42:55.609 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-17T19:42:55.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-17T19:42:55.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T19:42:55.750 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-10-17T19:42:56.205 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T19:42:56.206 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-10-17T19:42:56.335 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T19:42:56.335 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-10-17T19:42:56.447 INFO:tasks.ceph:Archiving mon data... 2023-10-17T19:42:56.449 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/data/c2.mon.a.tgz 2023-10-17T19:42:56.449 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-10-17T19:42:56.468 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.IbGX1SbizL 2023-10-17T19:42:56.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.IbGX1SbizL 2023-10-17T19:42:56.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.IbGX1SbizL 2023-10-17T19:42:56.735 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.IbGX1SbizL is 315KB 2023-10-17T19:42:56.829 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.IbGX1SbizL 2023-10-17T19:42:56.847 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-17T19:42:56.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T19:42:56.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T19:42:56.913 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-17T19:42:56.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-10-17T19:42:57.286 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-17T19:42:57.286 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-17T19:42:57.287 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-10-17T19:42:57.288 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:57.329 INFO:tasks.ceph.c1.osd.0:Stopped 2023-10-17T19:42:57.330 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-10-17T19:42:57.330 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:57.376 INFO:tasks.ceph.c1.osd.1:Stopped 2023-10-17T19:42:57.377 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-10-17T19:42:57.377 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:57.421 INFO:tasks.ceph.c1.osd.2:Stopped 2023-10-17T19:42:57.422 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-17T19:42:57.422 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-10-17T19:42:57.423 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:57.447 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-10-17T19:42:57.448 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-17T19:42:57.448 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-10-17T19:42:57.448 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T19:42:57.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~0s 2023-10-17T19:42:57.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2023-10-17T19:42:57.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2023-10-17T19:42:57.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2023-10-17T19:42:57.508 INFO:tasks.ceph.c1.mon.a:Stopped 2023-10-17T19:42:57.509 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-17T19:42:57.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-17T19:42:57.577 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T19:42:57.577 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-10-17T19:42:57.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-17T19:42:57.943 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T19:42:57.943 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-10-17T19:42:58.062 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T19:42:58.063 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-10-17T19:42:58.145 INFO:tasks.ceph:Archiving mon data... 2023-10-17T19:42:58.145 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/data/c1.mon.a.tgz 2023-10-17T19:42:58.146 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-17T19:42:58.164 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.4MV46WpfeZ 2023-10-17T19:42:58.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.4MV46WpfeZ 2023-10-17T19:42:58.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.4MV46WpfeZ 2023-10-17T19:42:58.423 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.4MV46WpfeZ is 355KB 2023-10-17T19:42:58.518 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.4MV46WpfeZ 2023-10-17T19:42:58.536 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-17T19:42:58.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T19:42:58.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T19:42:58.611 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-17T19:42:58.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-17T19:42:58.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-17T19:42:58.672 INFO:teuthology.orchestra.run.smithi049.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-17T19:42:58.682 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-17T19:42:58.689 INFO:tasks.ceph:Archiving crash dumps... 2023-10-17T19:42:58.691 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi049/crash 2023-10-17T19:42:58.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-17T19:42:58.746 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi071/crash 2023-10-17T19:42:58.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-17T19:42:58.778 INFO:tasks.ceph:Compressing logs... 2023-10-17T19:42:58.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T19:42:58.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T19:43:02.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 has been restored 2023-10-17T19:43:02.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 has been restored 2023-10-17T19:43:02.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 has been restored 2023-10-17T19:43:02.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a has been restored 2023-10-17T19:53:11.119 INFO:tasks.ceph:Archiving logs... 2023-10-17T19:53:11.120 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi049/log 2023-10-17T19:53:11.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-17T19:53:24.478 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi071/log 2023-10-17T19:53:24.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-17T19:54:32.764 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-17T19:54:32.776 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-17T19:54:32.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-17T19:54:32.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-17T19:54:32.961 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-17T19:54:32.971 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-17T19:54:32.971 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T19:54:32.974 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T19:54:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-10-17T19:54:32.995 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-10-17T19:54:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T19:54:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-17T19:54:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 38 +1590us[+1590us] +/- 91ms 2023-10-17T19:54:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 8 377 170 -927us[ -929us] +/- 52ms 2023-10-17T19:54:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -107us[ -107us] +/- 60ms 2023-10-17T19:54:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T19:54:33.034 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T19:54:33.034 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-10-17T19:54:33.035 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 42 +1418us[+1418us] +/- 91ms 2023-10-17T19:54:33.035 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 8 377 162 -1049us[-1100us] +/- 52ms 2023-10-17T19:54:33.035 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -300us[ -300us] +/- 60ms 2023-10-17T19:54:33.036 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T19:54:33.036 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-17T19:54:33.047 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-17T19:54:33.048 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-17T19:54:33.086 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-17T19:54:33.192 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-17T19:54:33.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T19:54:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T19:54:33.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T19:54:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T19:54:33.425 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-17T19:54:33.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-17T19:54:33.478 INFO:teuthology.task.internal:Duration was 6960.953231 seconds 2023-10-17T19:54:33.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-17T19:54:33.512 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-17T19:54:33.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-17T19:54:33.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-17T19:54:33.564 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T19:54:33.594 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T19:54:34.040 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-17T19:54:34.041 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-10-17T19:54:34.041 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-17T19:54:34.067 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2023-10-17T19:54:34.068 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-17T19:54:34.094 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-17T19:54:34.094 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T19:54:34.110 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T19:54:34.176 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-17T19:54:34.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-17T19:54:34.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-17T19:54:34.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-17T19:54:34.305 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-17T19:54:34.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-17T19:54:34.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-17T19:54:34.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-17T19:54:34.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-17T19:54:34.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-17T19:54:34.414 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-10-17T19:54:34.460 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2023-10-17T19:54:34.474 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-17T19:54:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T19:54:34.508 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-17T19:54:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T19:54:34.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-17T19:54:34.538 INFO:teuthology.task.internal:Transferring archived files... 2023-10-17T19:54:34.539 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi049 2023-10-17T19:54:34.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-17T19:54:34.589 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137/remote/smithi071 2023-10-17T19:54:34.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-17T19:54:34.627 INFO:teuthology.task.internal:Removing archive directory... 2023-10-17T19:54:34.627 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-17T19:54:34.630 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-17T19:54:34.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-17T19:54:34.694 INFO:teuthology.task.internal:Not uploading archives. 2023-10-17T19:54:34.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-17T19:54:34.720 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-17T19:54:34.720 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-17T19:54:34.723 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-17T19:54:34.739 INFO:teuthology.orchestra.run.smithi049.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 17 19:54 /home/ubuntu/cephtest 2023-10-17T19:54:34.740 INFO:teuthology.orchestra.run.smithi071.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 17 19:54 /home/ubuntu/cephtest 2023-10-17T19:54:34.742 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-17T19:54:34.755 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-17T19:54:34.821 INFO:teuthology.nuke:Checking targets against current locks 2023-10-17T19:54:34.846 DEBUG:teuthology.nuke:shortname: smithi049 2023-10-17T19:54:34.847 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T19:54:34.871 DEBUG:teuthology.nuke:shortname: smithi071 2023-10-17T19:54:34.871 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T19:54:34.875 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 17:47:27.961529', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKx21743cUlmo6TKUbvR5PAR3YJyl6RDmMH8t+R6LplmFaEgnLfqs3jO/mdESrHA1/7zybwQDizFBDdcbLTm/18='} 2023-10-17T19:54:34.895 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-16_21:55:28-rgw-wip-yuri-testing-2023-10-16-1247-reef-distro-default-smithi/7430137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-17 17:47:27.959582', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxePA3TlvYqb+4IRWm+eI1CkcB6eXCdh5zeTPXhaTJWuWW7972TipdWZLJP4OU5UiJN2/V8uTym1MqHpDcG0e4='} 2023-10-17T19:54:34.925 INFO:teuthology.orchestra.console.smithi049:Power off 2023-10-17T19:54:34.927 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-17T19:54:34.950 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2023-10-17T19:54:34.951 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:34.968 INFO:teuthology.orchestra.console.smithi071:Power off 2023-10-17T19:54:34.969 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-17T19:54:34.980 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-17T19:54:34.994 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2023-10-17T19:54:34.994 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:35.018 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-10-17T19:54:38.981 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:39.019 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:39.095 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-17T19:54:39.130 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-10-17T19:54:43.097 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:43.132 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T19:54:43.209 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2023-10-17T19:54:43.244 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2023-10-17T19:54:43.310 INFO:teuthology.orchestra.console.smithi049:Power off completed 2023-10-17T19:54:43.345 INFO:teuthology.orchestra.console.smithi071:Power off completed 2023-10-17T19:54:43.450 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 6960.953231096268 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=60920426ddef413bb577b22d1275130e status: fail success: false 2023-10-17T19:54:43.450 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T19:54:43.529 INFO:teuthology.run:FAIL